Re: optimizing pg_upgrade's once-in-each-database steps
Jeff Davis <pgsql@j-davis.com>
From: Jeff Davis <pgsql@j-davis.com>
To: Nathan Bossart <nathandbossart@gmail.com>, pgsql-hackers@postgresql.org
Date: 2024-05-17T00:09:55Z
Lists: pgsql-hackers
On Thu, 2024-05-16 at 16:16 -0500, Nathan Bossart wrote: > I am posting this early to get thoughts on the general approach. If > we > proceeded with this strategy, I'd probably create some generic > tooling that > each relevant step would provide a set of callback functions. The documentation states: "pg_dump -j uses multiple database connections; it connects to the database once with the leader process and once again for each worker job." That might need to be adjusted. How much complexity do you avoid by using async instead of multiple processes? Also, did you consider connecting once to each database and running many queries? Most of those seem like just checks. Regards, Jeff Davis
Commits
-
Introduce framework for parallelizing various pg_upgrade tasks.
- 40e2e5e92b7d 18.0 landed
-
pg_upgrade: Parallelize WITH OIDS check.
- f93f5f7b984e 18.0 landed
-
pg_upgrade: Parallelize contrib/isn check.
- 9db3018cf83b 18.0 landed
-
pg_upgrade: Parallelize data type checks.
- bbf83cab98f2 18.0 landed
-
pg_upgrade: Parallelize encoding conversion check.
- c880cf258864 18.0 landed
-
pg_upgrade: Parallelize incompatible polymorphics check.
- cf2f82a37cc3 18.0 landed
-
pg_upgrade: Parallelize postfix operator check.
- c34eabfbbfd3 18.0 landed
-
pg_upgrade: Parallelize retrieving extension updates.
- 6ab8f27bc716 18.0 landed
-
pg_upgrade: Parallelize retrieving loadable libraries.
- 46cad8b31927 18.0 landed
-
pg_upgrade: Parallelize retrieving relation information.
- 6d3d2e8e541f 18.0 landed
-
pg_upgrade: Parallelize subscription check.
- 7baa36de58bd 18.0 landed
-
pg_upgrade: Move live_check variable to user_opts.
- 4b56bb4ab485 18.0 landed
-
pg_upgrade: run all data type checks per connection
- 347758b12063 17.0 cited