Horton

Re: Inefficiency in parallel pg_restore with many tables

Nathan Bossart <nathandbossart@gmail.com>

View thread View on postgr.es
From: Nathan Bossart <nathandbossart@gmail.com>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Andrew Dunstan <andrew@dunslane.net>, pgsql-hackers@lists.postgresql.org
Date: 2023-07-20T19:06:44Z
Lists: pgsql-hackers

Attachments

Here is a work-in-progress patch set for converting ready_list to a
priority queue.  On my machine, Tom's 100k-table example [0] takes 11.5
minutes without these patches and 1.5 minutes with them.

One item that requires more thought is binaryheap's use of Datum.  AFAICT
the Datum definitions live in postgres.h and aren't available to frontend
code.  I think we'll either need to move the Datum definitions to c.h or to
adjust binaryheap to use "void *".

[0] https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Commits

  1. Remove open-coded binary heap in pg_dump_sort.c.

  2. Convert pg_restore's ready_list to a priority queue.

  3. Add function for removing arbitrary nodes in binaryheap.

  4. Make binaryheap available to frontend code.