Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Amul Sul <sulamul@gmail.com>
Cc: Andrew Dunstan <andrew@dunslane.net>, Michael Paquier <michael@paquier.xyz>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-21T19:31:10Z
Lists: pgsql-hackers

Attachments

I don't like the v3 patches too much: in particular, they do nothing
for the failure-to-finalize bug I identified yesterday.  v3-0003
is on the right track but seems overcomplicated.  Here is my own
set of proposed patches, which I think will fix what we are seeing
in the buildfarm:

v4-0001 is Andrew's fix for incorrect decompressor finalization.

v4-0002 fixes read_archive_file() to do the missing finalize step.

v4-0003 fixes init_archive_reader() to not depend on cur_file.
This is closely allied to v3-0003 but simpler.  I also added
some commentary to pg_waldump.h about what it's safe to do with
cur_file.

get_archive_wal_entry() violates that advice and is pretty much
utterly broken IMO, because it still believes that it can use cur_file
in an incorrect way.  However, the impact of that is that it may fail
to flush some hashtable entries out to temp files (in case a single
read_archive_file() step reads more than one WAL file, which is
entirely possible with compression).  That is a performance issue but
it's not causing our buildfarm problems, so I left it untouched here.
But I don't think any of the patches proposed so far fix it properly.
What it should do should look more like the revised version of
init_archive_reader's loop: call read_archive_file(), then scan the
hash table for WAL entries we need to flush to files, then finally
return the desired WAL entry if it's present, else loop around.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.