Horton

Re: Introduce pg_receivewal gzip compression tests

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: gkokolatos@pm.me
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-07-13T07:37:53Z
Lists: pgsql-hackers 
On Tue, Jul 13, 2021 at 06:36:59AM +0000, gkokolatos@pm.me wrote:
> I am sorry this was not so clear. It is indeed running twice the binary
> with different flags. However the goal is not to check the flags, but
> to make certain that the partial file has now been completed. That is
> why there was code asserting that the previous FILENAME.gz.partial file
> after the second invocation is converted to FILENAME.gz.

The first run you are adding checks the same thing thanks to
pg_switch_wal(), where pg_receivewal completes the generation of
000000010000000000000002.gz and finishes with
000000010000000000000003.gz.partial.

> Additionally the second invocation of pg_receivewal is extending the
> coverage of FindStreamingStart().

Hmm.  It looks like a waste in runtime once we mix LZ4 in that as that
would mean 5 runs of pg_receivewal, but we really need only three of
them with --endpos:
- One with ZLIB compression.
- One with LZ4 compression.
- One without compression.

Do you think that we could take advantage of what is now the only run
of pg_receivewal --endpos for that?  We could make the ZLIB checks run
first, conditionally, and then let the last command with --endpos
perform a full scan of the contents in $stream_dir with the .gz files
already in place.  The addition of LZ4 would be an extra conditional
block similar to what's introduced in ZLIB, running before the last
command without compression.
--
Michael

Commits

  1. Re-enable TAP tests of pg_receivewal for ZLIB on Windows

  2. Fix some issues with WAL segment opening for pg_receivewal --compress

  3. Disable tests involving ZLIB on Windows for pg_receivewal

  4. Fix portability issue with gzip in TAP test of pg_receivewal

  5. Add TAP tests for ZLIB compression for pg_receivewal