Horton

Re: [PING] [PATCH v2] parallel pg_restore: avoid disk seeks when jumping short distance forward

Dimitrios Apostolou <jimis@gmx.net>

View thread View on postgr.es
From: Dimitrios Apostolou <jimis@gmx.net>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@lists.postgresql.org
Date: 2025-07-23T21:54:44Z
Lists: pgsql-hackers 
Hi Nathan, I've noticed you've set yourself as a reviewer of this patch 
in the commitfest. I appreciate it, but you might want to combine it 
with another simple patch [1] that speeds up the same part of 
pg_restore: the initial full scan on TOC-less archives.

[1] https://commitfest.postgresql.org/patch/5817/


On Saturday 2025-06-14 00:04, Nathan Bossart wrote:
>
> On Fri, Jun 13, 2025 at 01:00:26AM +0200, Dimitrios Apostolou wrote:
>> By the way, I might have set the threshold to 1MB in my program, but
>> lowering it won't show a difference in my test case, since the lseek()s I
>> was noticing before the patch were mostly 8-16KB forward. Not sure what is
>> the defining factor for that. Maybe the compression algorithm, or how wide
>> the table is?
>
> I may have missed it, but could you share what the strace looks like with
> the patch applied?

I hope you've seen my response here, with special focus on the small 
block size that both compressed and uncompressed custom format archives 
have.

I have been needing to pg_restore 10TB TOC-less dumps recently, and it's 
a pain to do the full scan, even with both of my patches applied.

Maybe the block size could be a command line option of pg_dump, so that 
one could set it to sizes like 100MB, which sounds like a normal block 
from the perspective of a 10TB gigantic dump.

Regards,
Dimitris

Commits

  1. Avoid short seeks in pg_restore.

  2. Don't rely on zlib's gzgetc() macro.

  3. Add more TAP test coverage for pg_dump.

  4. Split 002_pg_dump.pl into two test files.

  5. Align the data block sizes of pg_dump's various compression modes.

  6. Fix serious performance problems in LZ4Stream_read_internal.

  7. Fix poor buffering logic in pg_dump's lz4 and zstd compression code.

  8. Fix issue with reading zero bytes in Gzip_read.

  9. Restore test coverage of LZ4Stream_gets().