Horton

Re: optimizing pg_upgrade's once-in-each-database steps

Daniel Gustafsson <daniel@yesql.se>

View thread View on postgr.es
From: Daniel Gustafsson <daniel@yesql.se>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Jeff Davis <pgsql@j-davis.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2024-07-18T07:57:23Z
Lists: pgsql-hackers 
> On 17 Jul 2024, at 23:32, Nathan Bossart <nathandbossart@gmail.com> wrote:
> On Wed, Jul 17, 2024 at 11:16:59PM +0200, Daniel Gustafsson wrote:

>> +static void
>> +dispatch_query(const ClusterInfo *cluster, AsyncSlot *slot,
>>    ....
>> +   pg_free(query);
>> +}
>> 
>> A minor point, perhaps fueled by me not having played around much with this
>> patchset.  It seems a bit odd that dispatch_query is responsible for freeing
>> the query from the get_query callback.  I would have expected the output from
>> AsyncTaskGetQueryCB to be stored in AsyncTask and released by async_task_free.
> 
> I don't see any problem with doing it the way you suggest.
> 
> Tangentially related, I waffled a bit on whether to require the query
> callbacks to put the result in allocated memory.  Some queries are the same
> no matter what, and some require customization at runtime.  As you can see,
> I ended up just requiring allocated memory.  That makes the code a tad
> simpler, and I doubt the extra work is noticeable.

I absolutely agree with this.

>> +   char       *query = pg_malloc(QUERY_ALLOC);
>> 
>> Should we convert this to a PQExpBuffer?
> 
> Seems like a good idea.  I think I was trying to change as few lines as
> possible for my proof-of-concept.  :)

Yeah, that's a good approach, I just noticed it while reading the hunks.  We
can do that separately from this patchset.

In order to trim down the size of the patchset I think going ahead with 0003
independently of this makes sense, it has value by itself.

--
Daniel Gustafsson

Commits

  1. Introduce framework for parallelizing various pg_upgrade tasks.

  2. pg_upgrade: Parallelize WITH OIDS check.

  3. pg_upgrade: Parallelize contrib/isn check.

  4. pg_upgrade: Parallelize data type checks.

  5. pg_upgrade: Parallelize encoding conversion check.

  6. pg_upgrade: Parallelize incompatible polymorphics check.

  7. pg_upgrade: Parallelize postfix operator check.

  8. pg_upgrade: Parallelize retrieving extension updates.

  9. pg_upgrade: Parallelize retrieving loadable libraries.

  10. pg_upgrade: Parallelize retrieving relation information.

  11. pg_upgrade: Parallelize subscription check.

  12. pg_upgrade: Move live_check variable to user_opts.

  13. pg_upgrade: run all data type checks per connection