AW: Assorted improvements in pg_dump

Hans Buschmann <buschmann@nidsa.net>

From: Hans Buschmann <buschmann@nidsa.net>

To: Tom Lane <tgl@sss.pgh.pa.us>

Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>, "Andres Freund" <andres@anarazel.de>

Date: 2021-12-07T08:05:32Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

pg_dump: avoid unsafe function calls in getPolicies().
- b7333e826955 11.19 landed
- a5b26aaafe4f 13.10 landed
- 1ed6f1b9116c 12.14 landed
- 03ac48549438 14.7 landed
- 3e6e86abca01 15.0 landed
Postpone calls of unsafe server-side functions in pg_dump.
- e46e986baef0 13.10 landed
- b1f106420b1a 11.19 landed
- 55f30e6c7640 14.7 landed
- 344b7849200f 12.14 landed
- e3fcbbd623b9 15.0 landed
Account for TOAST data while scheduling parallel dumps.
- 65aaed22a849 15.0 landed
Use PREPARE/EXECUTE for repetitive per-object queries in pg_dump.
- be85727a3df7 15.0 landed
Avoid per-object queries in performance-critical paths in pg_dump.
- 9895961529ef 15.0 landed
Rethink pg_dump's handling of object ACLs.
- 0c9d84427f44 15.0 landed
Refactor pg_dump's tracking of object components to be dumped.
- 5209c0ba0bfd 15.0 landed
pg_dump: fix mis-dumping of non-global default privileges.
- 2acc84c6fd29 15.0 cited

Hello Tom,

from your mail from 25.10.2021:

>0005 implements your suggestion of accounting for TOAST data while
>scheduling parallel dumps.  I realized while looking at that that
>there's a pre-existing bug, which this'd exacerbate: on machines
>with 32-bit off_t, dataLength can overflow.  Admittedly such machines
>are just about extinct in the wild, but we do still trouble to support
>the case.  So 0005 also includes code to check for overflow and clamp
>the result to INT_MAX blocks.

>Maybe we should back-patch 0005.  OTOH, how likely is it that anyone
>is wrangling tables exceeding 16TB on a machine with 32-bit off_t?
>Or that poor parallel dump scheduling would be a real problem in
>such a case?

I noticed that you patched master with all the improvements in pg_dump.

Did you change your mind about backpatching patch 0005 to fix the toast size matter?

It would be rather helpfull for handling existent user data in active branches.


On the matter of 32bit versions I think they are used only in much more little instances.

BTW the 32 bit build of postgres on Windows does not work any more with more modern tool sets (tested with VS2019 and VS2022) albeit not excluded explicitly in the docs. But no one complained yet (for a long time now...).

Thanks

Hans Buschmann