Horton

Re: [PATCH v1] parallel pg_restore: avoid disk seeks when jumping short distance forward

Dimitrios Apostolou <jimis@gmx.net>

View thread View on postgr.es
From: Dimitrios Apostolou <jimis@gmx.net>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>
Date: 2025-04-01T22:25:25Z
Lists: pgsql-hackers 
Thanks. This is the first value I tried and it works well. In the archive I have all blocks seem to be between 8 and 20KB so the jump forward before the change never even got close to 1MB. Could it be bigger in an uncompressed archive? Or in a future pg_dump that raises the block size? I don't really know, so it is difficult to test such scenario but it made sense to guard against these cases too.

I chose 1MB by basically doing a very crude calculation in my mind: when would it be worth seeking forward instead of reading? On very slow drives 60MB/s sequential and 60 IOPS for random reads is a possible speed. In that worst case it would be better to seek() forward for lengths of over 1MB. 

On 1 April 2025 22:04:00 CEST, Nathan Bossart <nathandbossart@gmail.com> wrote:
>On Tue, Apr 01, 2025 at 09:33:32PM +0200, Dimitrios Apostolou wrote:
>> It didn't break any test, but I also don't see any difference, the
>> performance boost is noticeable only when restoring a huge archive that is
>> missing offsets.
>
>This seems generally reasonable to me, but how did you decide on 1MB as the
>threshold?  Have you tested other values?  Could the best threshold vary
>based on the workload and hardware?
>

Commits

  1. Avoid short seeks in pg_restore.

  2. Don't rely on zlib's gzgetc() macro.

  3. Add more TAP test coverage for pg_dump.

  4. Split 002_pg_dump.pl into two test files.

  5. Align the data block sizes of pg_dump's various compression modes.

  6. Fix serious performance problems in LZ4Stream_read_internal.

  7. Fix poor buffering logic in pg_dump's lz4 and zstd compression code.

  8. Fix issue with reading zero bytes in Gzip_read.

  9. Restore test coverage of LZ4Stream_gets().