Re: Introduce pg_receivewal gzip compression tests

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Tuesday, July 13th, 2021 at 09:37, Michael Paquier <michael@paquier.xyz> wrote:

> On Tue, Jul 13, 2021 at 06:36:59AM +0000, gkokolatos@pm.me wrote:
>
> > I am sorry this was not so clear. It is indeed running twice the binary
> > with different flags. However the goal is not to check the flags, but
> > to make certain that the partial file has now been completed. That is
> > why there was code asserting that the previous FILENAME.gz.partial file
> > after the second invocation is converted to FILENAME.gz.
>
> The first run you are adding checks the same thing thanks to
> pg_switch_wal(), where pg_receivewal completes the generation of
> 000000010000000000000002.gz and finishes with
> 000000010000000000000003.gz.partial.

This is correct. It is the 000000010000000000000003 awl that the rest
of the tests are targeting.

>
> > Additionally the second invocation of pg_receivewal is extending the
> > coverage of FindStreamingStart().
>
> Hmm. It looks like a waste in runtime once we mix LZ4 in that as that
> would mean 5 runs of pg_receivewal, but we really need only three of
> them with --endpos:
> -   One with ZLIB compression.
> -   One with LZ4 compression.
> -   One without compression.
>
>     Do you think that we could take advantage of what is now the only run
>     of pg_receivewal --endpos for that? We could make the ZLIB checks run
>     first, conditionally, and then let the last command with --endpos
>     perform a full scan of the contents in $stream_dir with the .gz files
>     already in place. The addition of LZ4 would be an extra conditional
>     block similar to what's introduced in ZLIB, running before the last
>     command without compression.

I will admit that for the current patch I am not taking lz4 into account as
at the moment I have little idea as to how the lz4 patch will advance with the
review rounds. I simply accepted that it will be rebased on top of the patch
in the current thread and probably need to modify the current then.

But I digress. I would like have some combination of .gz and .gz.parial but
I will not take too strong of a stance. I am happy to go with your suggestion.

Cheers,
//Georgios

>
>     --
>
>     Michael