Horton

Re: Add LZ4 compression in pg_dump

Tomas Vondra <tomas.vondra@enterprisedb.com>

View thread View on postgr.es
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: gkokolatos@pm.me, Justin Pryzby <pryzby@telsasoft.com>
Cc: Michael Paquier <michael@paquier.xyz>, shiy.fnst@fujitsu.com, pgsql-hackers@lists.postgresql.org, Rachel Heaton <rachelmheaton@gmail.com>
Date: 2023-03-01T16:14:08Z
Lists: pgsql-hackers 

On 2/27/23 15:56, gkokolatos@pm.me wrote:
> 
> 
> 
> 
> 
> ------- Original Message -------
> On Saturday, February 25th, 2023 at 3:05 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:
> 
> 
>>
>>
>> On Fri, Feb 24, 2023 at 11:02:14PM -0600, Justin Pryzby wrote:
>>
>>> I have some fixes (attached) and questions while polishing the patch for
>>> zstd compression. The fixes are small and could be integrated with the
>>> patch for zstd, but could be applied independently.
> 
> 
> Please find some comments on the rest of the fixes patch that Tomas has not
> commented on.
> 
>             can be compressed with the <application>gzip</application> or
> -           <application>lz4</application>tool.
> +           <application>lz4</application> tools.
> 
> +1
> 
>          The compression method can be set to <literal>gzip</literal> or
> -        <literal>lz4</literal> or <literal>none</literal> for no compression.
> +        <literal>lz4</literal>, or <literal>none</literal> for no compression.
> 
> I am not a native English speaker. Yet I think that if one adds commas
> in one of the options, then one should add commas to all the options.
> Namely, the aboveis missing a comma between gzip and lz4. However I
> think that not having any commas still works grammatically and
> syntactically.
> 

I pushed a fix with most of these wording changes. As for this comma, I
believe the correct style is

   a, b, or c

At least that's what the other places in the pg_dump.sgml file do.

> -               ($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
> -               ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4))
> +               (($pgdump_runs{$run}->{compile_option} eq 'gzip' && !$supports_gzip) ||
> +               ($pgdump_runs{$run}->{compile_option} eq 'lz4' && !$supports_lz4)))
> 

Pushed a fix for this too.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Advance input pointer when LZ4 compressing data

  2. Null-terminate the output buffer of LZ4Stream_gets

  3. Rework code defining default compression for dir/custom formats in pg_dump

  4. pg_dump: Use only LZ4 frame format for compression

  5. Minor comment improvements for compress_lz4

  6. Unify buffer sizes in pg_dump compression API

  7. Improve type handling in pg_dump's compress file API

  8. Improve wording in pg_dump compression docs

  9. Fix condition in pg_dump TAP test

  10. Add LZ4 compression to pg_dump

  11. Introduce a generic pg_dump compression API

  12. Prepare pg_dump internals for additional compression methods

  13. Fix behavior with pg_restore -l and compressed dumps

  14. Add 250c8ee07ed to git-blame-ignore-revs

  15. Provide test coverage in pg_dump for default behaviors with compression

  16. Switch pg_dump to use compression specifications

  17. Refactor code parsing compression option values (-Z/--compress)

  18. meson: Add some missing env settings for tests of pg_dump and pg_verifybackup

  19. Extend TAP tests of pg_dump to test for compression with gzip

  20. Clean up some dead code in pg_dump with tar format and gzip compression

  21. Add TAP test in pg_dump with --format=tar and --compress

  22. Replace BASE_BACKUP COMPRESSION_LEVEL option with COMPRESSION_DETAIL.

  23. Refactor the pg_dump zlib code from pg_backup_custom.c to a separate file,