Horton
Menu

Re: Fix pg_upgrade to preserve datdba

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Jan Wieck <jan@wi3ck.info>
Cc: Magnus Hagander <magnus@hagander.net>, Robins Tharakan <tharakan@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-03-21T19:56:14Z
Lists: pgsql-hackers 
Jan Wieck <jan@wi3ck.info> writes:
> So let's focus on the actual problem of running out of XIDs and memory 
> while doing the upgrade involving millions of small large objects.

Right.  So as far as --single-transaction vs. --create goes, that's
mostly a definitional problem.  As long as the contents of a DB are
restored in one transaction, it's not gonna matter if we eat one or
two more XIDs while creating the DB itself.  So we could either
relax pg_restore's complaint, or invent a different switch that's
named to acknowledge that it's not really only one transaction.

That still leaves us with the lots-o-locks problem.  However, once
we've crossed the Rubicon of "it's not really only one transaction",
you could imagine that the switch is "--fewer-transactions", and the
idea is for pg_restore to commit after every (say) 100000 operations.
That would both bound its lock requirements and greatly cut its XID
consumption.

The work you described sounded like it could fit into that paradigm,
with the additional ability to run some parallel restore tasks
that are each consuming a bounded number of locks.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Count individual SQL commands in pg_restore's --transaction-size mode.

  2. Reduce number of commands dumpTableSchema emits for binary upgrade.

  3. Invent --transaction-size option for pg_restore.

  4. Rearrange pg_dump's handling of large objects for better efficiency.

  5. Add temporal PRIMARY KEY and UNIQUE constraints

  6. Fix typo and case in messages