Horton

Re: optimizing pg_upgrade's once-in-each-database steps

Nathan Bossart <nathandbossart@gmail.com>

View thread View on postgr.es
From: Nathan Bossart <nathandbossart@gmail.com>
To: Corey Huinker <corey.huinker@gmail.com>
Cc: Ilya Gladyshev <ilya.v.gladyshev@gmail.com>, Daniel Gustafsson <daniel@yesql.se>, Robert Haas <robertmhaas@gmail.com>, Jeff Davis <pgsql@j-davis.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2024-08-10T15:35:46Z
Lists: pgsql-hackers 
On Sat, Aug 10, 2024 at 10:17:27AM -0500, Nathan Bossart wrote:
> On Fri, Aug 09, 2024 at 04:06:16PM -0400, Corey Huinker wrote:
>>> Furthermore, most of the callbacks should do almost nothing for a given
>>> upgrade, and since pg_upgrade runs on the server, client/server round-trip
>>> time should be pretty low.
>> 
>> To my mind, that makes pipelining make more sense, you throw out N queries,
>> most of which are trivial, and by the time you cycle back around and start
>> digesting result sets via callbacks, more of the queries have finished
>> because they were waiting on the query ahead of them in the pipeline, not
>> waiting on a callback to finish consuming its assigned result set and then
>> launching the next task query.
> 
> My assumption is that the "waiting for a callback before launching the next
> query" time will typically be pretty short in practice.  I could try
> measuring it...

Another option might be to combine all the queries for a task into a single
string and then send that in one PQsendQuery() call.  That may be a simpler
way to eliminate the time between queries.

-- 
nathan

Commits

  1. Introduce framework for parallelizing various pg_upgrade tasks.

  2. pg_upgrade: Parallelize WITH OIDS check.

  3. pg_upgrade: Parallelize contrib/isn check.

  4. pg_upgrade: Parallelize data type checks.

  5. pg_upgrade: Parallelize encoding conversion check.

  6. pg_upgrade: Parallelize incompatible polymorphics check.

  7. pg_upgrade: Parallelize postfix operator check.

  8. pg_upgrade: Parallelize retrieving extension updates.

  9. pg_upgrade: Parallelize retrieving loadable libraries.

  10. pg_upgrade: Parallelize retrieving relation information.

  11. pg_upgrade: Parallelize subscription check.

  12. pg_upgrade: Move live_check variable to user_opts.

  13. pg_upgrade: run all data type checks per connection