Horton

Thread

Commits

  1. Doc: warn against using parallel restore with --load-via-partition-root.

  1. pg_dump --load-via-partition-root vs. parallel restore

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-08-28T19:53:39Z

    Parallel pg_restore generally assumes that the archive file is telling it
the truth about data dependencies; in particular, that a TABLE DATA item
naming a particular target table is loading data into exactly that table.
--load-via-partition-root creates a significant probability that that
assumption is wrong, at least in scenarios where the data really does get
redirected into other partitions than the original one.  This can result
in inefficiencies (e.g., index rebuild started before a table's data is
really all loaded) or outright failures (foreign keys or RLS policies
applied before the data is all loaded).  I suspect that deadlock failures
during restore are also possible, since identify_locking_dependencies
is not going to be nearly close to the truth about which operations
might hold which locks.

This could possibly be fixed by changing around the dependencies shown
in the archive file so that POST_DATA objects that're nominally dependent
on any one of a partitioned table's members are shown as dependent on all
of them.  I'm not particularly eager to write that patch though.

For the moment I'm inclined to just document the problem, e.g. "It's
recommended that parallel restore not be used with archives generated
with this option."

			regards, tom lane

  2. Re: pg_dump --load-via-partition-root vs. parallel restore

    Robert Haas <robertmhaas@gmail.com> — 2018-09-02T22:31:01Z

    On Tue, Aug 28, 2018 at 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Parallel pg_restore generally assumes that the archive file is telling it
> the truth about data dependencies; in particular, that a TABLE DATA item
> naming a particular target table is loading data into exactly that table.
> --load-via-partition-root creates a significant probability that that
> assumption is wrong, at least in scenarios where the data really does get
> redirected into other partitions than the original one.  This can result
> in inefficiencies (e.g., index rebuild started before a table's data is
> really all loaded) or outright failures (foreign keys or RLS policies
> applied before the data is all loaded).  I suspect that deadlock failures
> during restore are also possible, since identify_locking_dependencies
> is not going to be nearly close to the truth about which operations
> might hold which locks.

Hmm.  I had the idea that this wasn't a problem because I thought we
had all of pg_dump strictly separated into pre-data, data, and
post-data; therefore, I thought it would be the case that none of that
other stuff would happen until all table data was loaded.  From what
you are saying here, I guess that's not the case?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  3. Re: pg_dump --load-via-partition-root vs. parallel restore

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-02T22:42:12Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Aug 28, 2018 at 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Parallel pg_restore generally assumes that the archive file is telling it
>> the truth about data dependencies; in particular, that a TABLE DATA item
>> naming a particular target table is loading data into exactly that table.
>> --load-via-partition-root creates a significant probability that that
>> assumption is wrong, at least in scenarios where the data really does get
>> redirected into other partitions than the original one.  This can result
>> in inefficiencies (e.g., index rebuild started before a table's data is
>> really all loaded) or outright failures (foreign keys or RLS policies
>> applied before the data is all loaded).  I suspect that deadlock failures
>> during restore are also possible, since identify_locking_dependencies
>> is not going to be nearly close to the truth about which operations
>> might hold which locks.

> Hmm.  I had the idea that this wasn't a problem because I thought we
> had all of pg_dump strictly separated into pre-data, data, and
> post-data; therefore, I thought it would be the case that none of that
> other stuff would happen until all table data was loaded.  From what
> you are saying here, I guess that's not the case?

We don't run restore operations in parallel during the pre-data phase,
mainly because we lack faith that the dependencies are really completely
represented for those objects (particularly in older archives).  Once we
get to parallel operation, though, everything is schedulable as soon as
its dependencies are satisfied; so it's possible for post-data objects
to run before supposedly-unrelated data objects are done.  There's no
explicit representation of the data/post-data boundary in the archive's
dependencies.

			regards, tom lane