Horton

Re: Introduce pg_receivewal gzip compression tests

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: gkokolatos@pm.me
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-07-13T08:14:05Z
Lists: pgsql-hackers 
On Tue, Jul 13, 2021 at 04:37:53PM +0900, Michael Paquier wrote:
> Hmm.  It looks like a waste in runtime once we mix LZ4 in that as that
> would mean 5 runs of pg_receivewal, but we really need only three of
> them with --endpos:
> - One with ZLIB compression.
> - One with LZ4 compression.
> - One without compression.
> 
> Do you think that we could take advantage of what is now the only run
> of pg_receivewal --endpos for that?  We could make the ZLIB checks run
> first, conditionally, and then let the last command with --endpos
> perform a full scan of the contents in $stream_dir with the .gz files
> already in place.  The addition of LZ4 would be an extra conditional
> block similar to what's introduced in ZLIB, running before the last
> command without compression.

Poking at this problem, I partially take this statement back as this
requires an initial run of pg_receivewal --endpos to ensure the
creation of one .gz and one .gz.partial.  So I guess that this should
be structured as:
1) Keep the existing pg_receivewal --endpos.
2) Add the ZLIB block, with one pg_receivewal --endpos.
3) Add at the end one extra pg_receivewal --endpos, outside of the ZLIB
block, which should check the creation of a .partial, non-compressed
segment.  This should mention that we place this command at the end of
the test for the start streaming point computation.

LZ4 tests would be then between 2) and 3), or 1) and 2).
--
Michael

Commits

  1. Re-enable TAP tests of pg_receivewal for ZLIB on Windows

  2. Fix some issues with WAL segment opening for pg_receivewal --compress

  3. Disable tests involving ZLIB on Windows for pg_receivewal

  4. Fix portability issue with gzip in TAP test of pg_receivewal

  5. Add TAP tests for ZLIB compression for pg_receivewal