Horton

Re: [PING] [PATCH v2] parallel pg_restore: avoid disk seeks when jumping short distance forward

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Chao Li <li.evan.chao@gmail.com>
Cc: Dimitrios Apostolou <jimis@gmx.net>, Nathan Bossart <nathandbossart@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, pgsql-hackers@lists.postgresql.org
Date: 2025-10-15T19:21:50Z
Lists: pgsql-hackers

Attachments

I wrote:
> 0004 increases the row width in the existing test case that says
> it's trying to push more than DEFAULT_IO_BUFFER_SIZE through
> the compressors.  While I agree with the premise, this solution
> is hugely expensive: it adds about 12% to the already-long runtime
> of 002_pg_dump.pl.  I'd like to find a better way, but ran out of
> energy for today.  (I think the reason this costs so much is that
> it's effectively iterated hundreds of times because of
> 002_pg_dump.pl's more or less cross-product approach to testing
> everything.  Maybe we should pull it out of that structure?)

The attached patchset accomplishes that by splitting 002_pg_dump.pl
into two scripts, one that is just concerned with the compression
test cases and one that does everything else.  This might not be
the prettiest solution, since it duplicates a lot of perl code.
I thought about refactoring 002_pg_dump.pl so that it could handle
two separate sets of runs-plus-tests, but decided it was overly
complicated already.

Anyway, 0001 attached is the same as in v4, 0002 performs the
test split without intending to change coverage, and then 0003
adds the new test cases I wanted.  For me, this ends up with
just about the same runtime as before, or maybe a smidge less.
I'd hoped for possibly more savings than that, but I'm content
with it being a wash.

I think this is more or less committable, and then we could get
back to the original question of whether it's worth tweaking
pg_restore's seek-vs-scan behavior.

			regards, tom lane

Commits

  1. Avoid short seeks in pg_restore.

  2. Don't rely on zlib's gzgetc() macro.

  3. Add more TAP test coverage for pg_dump.

  4. Split 002_pg_dump.pl into two test files.

  5. Align the data block sizes of pg_dump's various compression modes.

  6. Fix serious performance problems in LZ4Stream_read_internal.

  7. Fix poor buffering logic in pg_dump's lz4 and zstd compression code.

  8. Fix issue with reading zero bytes in Gzip_read.

  9. Restore test coverage of LZ4Stream_gets().