Horton

Re: optimizing pg_upgrade's once-in-each-database steps

Daniel Gustafsson <daniel@yesql.se>

View thread View on postgr.es
From: Daniel Gustafsson <daniel@yesql.se>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Ilya Gladyshev <ilya.v.gladyshev@gmail.com>, Corey Huinker <corey.huinker@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Jeff Davis <pgsql@j-davis.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2024-09-09T21:20:28Z
Lists: pgsql-hackers 
> On 9 Sep 2024, at 21:17, Nathan Bossart <nathandbossart@gmail.com> wrote:
> 
> On Thu, Sep 05, 2024 at 01:32:34PM +0200, Daniel Gustafsson wrote:
>> I've read and tested through the latest version of this patchset and I think
>> it's ready to go in.
> 
> Thanks for reviewing.  I'm aiming to commit it later this week.

+1.  Looking forward to seeing what all the pg_dump/pg_upgrade changes amount
to in speed improvement when combined.

>> The one concern I have is that tasks can exit(1) on libpq
>> errors tearing down perfectly functional connections without graceful shutdown.
>> Longer term I think it would make sense to add similar exit handler callbacks
>> to the ones in pg_dump for graceful cleanup of connections.  However, in order
>> to keep goalposts in clear view I don't think this patch need to have it, but
>> it would be good to consider once in.
> 
> This did cross my mind.  I haven't noticed any problems in my testing, and
> it looks like there are several existing places in pg_upgrade that call
> pg_fatal() with open connections, so I'm inclined to agree that this is a
> nice follow-up task that needn't hold up this patch set.

It could perhaps be a good introductory task for a new contributor who want a
fairly confined project to hack on.

--
Daniel Gustafsson

Commits

  1. Introduce framework for parallelizing various pg_upgrade tasks.

  2. pg_upgrade: Parallelize WITH OIDS check.

  3. pg_upgrade: Parallelize contrib/isn check.

  4. pg_upgrade: Parallelize data type checks.

  5. pg_upgrade: Parallelize encoding conversion check.

  6. pg_upgrade: Parallelize incompatible polymorphics check.

  7. pg_upgrade: Parallelize postfix operator check.

  8. pg_upgrade: Parallelize retrieving extension updates.

  9. pg_upgrade: Parallelize retrieving loadable libraries.

  10. pg_upgrade: Parallelize retrieving relation information.

  11. pg_upgrade: Parallelize subscription check.

  12. pg_upgrade: Move live_check variable to user_opts.

  13. pg_upgrade: run all data type checks per connection