Horton

Thread

Commits

  1. Fix memory leak in pg_restore with zstd-compressed data.

  1. Not able to restore database - error: could not decompress data: Allocation error : not enough memory

    Tomasz Szypowski <tomasz.szypowski@asseco.pl> — 2024-12-17T20:29:20Z

    Our database consists of 408032 blobs (pg_largeobject).
Backup was made using zstd and it weights 70GB.
While restoring the used memory increases constantly (from 5 MB to 5GB - i have got 8GB RAM, after reaching 5GB it uses 5GB for some time, maybe swaping) and then crashes with the error.
During this period it has restored about 48000 blobs.

pg_restore: restoring large object with OID 169685
pg_restore: restoring large object with OID 169686
pg_restore: restoring large object with OID 169687
pg_restore: restoring large object with OID 169688
pg_restore: restoring large object with OID 169689
pg_restore: restoring large object with OID 169690
pg_restore: restoring large object with OID 169691
pg_restore: restoring large object with OID 169692
pg_restore: restoring large object with OID 169693
pg_restore: restoring large object with OID 169694
pg_restore: restoring large object with OID 169695
pg_restore: restoring large object with OID 169696
pg_restore: restoring large object with OID 169697
pg_restore: restoring large object with OID 169698
pg_restore: restoring large object with OID 169699
pg_restore: error: could not decompress data: Allocation error : not enough memory

It was checked with the original pg_restore.exe and libzstd.dll as well as with compiled by mingw both 32 and 64 bits. Several libzstd.dll was used. Without success.

Dump: pg_dump.exe --format=custom --compress=zstd:4 --file=zstd.zstd  dbname=TEST
Restore: pg_restore -d TEST zstd.zstd

Best regards
Thomas

  2. Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-12-18T02:22:00Z

    Tomasz Szypowski <tomasz.szypowski@asseco.pl> writes:
> Our database consists of 408032 blobs (pg_largeobject).
> Backup was made using zstd and it weights 70GB.
> While restoring the used memory increases constantly (from 5 MB to 5GB - i have got 8GB RAM, after reaching 5GB it uses 5GB for some time, maybe swaping) and then crashes with the error.

Yeah, leak reproduced here.  Apparently it's specific to the zstd
code path, because I don't see it with the default compression
method.  Should be easy to fix (awaiting valgrind results),
but in the meantime just use default compression.

Thanks for the report!

			regards, tom lane

  3. RE: Not able to restore database - error: could not decompress data: Allocation error : not enough memory

    Tomasz Szypowski <tomasz.szypowski@asseco.pl> — 2024-12-18T07:12:06Z

    Thank you for the quick response. 
Do you think it will be fixed within few months? Next year we plan to upgrade our clients to 17 and zstd compression is about two times faster than gzip and produces about 20% less backup.

Regards
Thomas


-----Original Message-----
From: Tom Lane <tgl@sss.pgh.pa.us> 
Sent: Wednesday, December 18, 2024 3:22 AM
To: Tomasz Szypowski <tomasz.szypowski@asseco.pl>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory

[Nie otrzymujesz cz?sto wiadomo?ci e-mail z tgl@sss.pgh.pa.us. Dowiedz si?, dlaczego jest to wa?ne, na stronie https://aka.ms/LearnAboutSenderIdentification ]

Tomasz Szypowski <tomasz.szypowski@asseco.pl> writes:
> Our database consists of 408032 blobs (pg_largeobject).
> Backup was made using zstd and it weights 70GB.
> While restoring the used memory increases constantly (from 5 MB to 5GB - i have got 8GB RAM, after reaching 5GB it uses 5GB for some time, maybe swaping) and then crashes with the error.

Yeah, leak reproduced here.  Apparently it's specific to the zstd code path, because I don't see it with the default compression method.  Should be easy to fix (awaiting valgrind results), but in the meantime just use default compression.

Thanks for the report!

                        regards, tom lane

  4. Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-12-18T15:13:04Z

    Tomasz Szypowski <tomasz.szypowski@asseco.pl> writes:
> Do you think it will be fixed within few months? Next year we plan to upgrade our clients to 17 and zstd compression is about two times faster than gzip and produces about 20% less backup.

The fix will be in February's releases.

			regards, tom lane