Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Andrew Dunstan <andrew@dunslane.net>

View thread
From: Andrew Dunstan <andrew@dunslane.net>
To: Tom Lane <tgl@sss.pgh.pa.us>, Michael Paquier <michael@paquier.xyz>
Cc: Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-21T12:21:49Z
Lists: pgsql-hackers

Attachments

On 2026-03-21 Sa 2:34 AM, Tom Lane wrote:
> Michael Paquier<michael@paquier.xyz> writes:
>> On Fri, Mar 20, 2026 at 11:49:02PM -0400, Tom Lane wrote:
>>> Buildfarm members batta and hachi don't like this very much.
>> I did not look at what's happening on the host, but it seems like a
>> safe bet to assume that we are not seeing many failures in the
>> buildfarm because we don't have many animals that have the idea to add
>> --with-zstd to their build configuration, like these two ones.
> That may be part of the story, but only part.  I spent a good deal of
> time trying to reproduce batta & hachi's configurations locally, on
> several different platforms, but still couldn't duplicate what they
> are showing.
>
> 			



Yeah, I haven't been able to reproduce it either. But while 
investigating I found a couple of issues. We neglected to add one of the 
tests to meson.build, and we neglected to close some files, causing 
errors on windows.

I also noticed a possible bug in astreamer, where the decompressor 
finalize functions send bbs_buffer.maxlen bytes to the next streamer 
when flushing remaining data at end-of-stream. This seems wrong because 
the buffer may only be partially filled with valid decompressed data. 
Possible patch for that attached. (But I don't think it's related to 
these failures).


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.