Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-22T17:29:44Z
Lists: pgsql-hackers

Attachments

Andrew Dunstan <andrew@dunslane.net> writes:
> OK, patch 5 of this set does that. I reworked your previous patches 2 and 3
> slightly - mostly additional comments, and fixing a bug in use
> of sizeof(XLogLongPageHeader). Patch 4 here tries to fix the wrong use of
> cur_file in get_archive_wal_entry()

A few nits:

0001: I think we need to rewrite the commit message now that we've
run the unknowns to ground.  Perhaps like

Send the correct amount of data to the next astreamer, not the
whole allocated buffer size.  This bug escaped detection because
in present uses the next astreamer is always a tar-file parser
which is insensitive to trailing garbage.  But that may not
always be true.

Also, I think we ought to back-patch 0001, just in case we have
misunderstood the scope of usage or somebody tries to make use of
back-branch code for a new purpose.  This is a bit tedious because
before f80b09bac/3c9056981 the code was somewhere else and used
different names.  But the bugs are clearly there, back to v15.

0002, 0003, 0005: LGTM.

0004: Don't like this at all.  It fails to respond to the point
I was trying to make: the code will only consider spilling hash
entries that are the cur_file at some read_archive_file boundary.
Also, it seems to be doubling down on the already over-complicated
and probably under-correct goal of spilling entries that are still
in progress of being read.  We can make this function far simpler
and more obviously correct if we just accept that we'll read a
WAL file completely before spilling it.  See my proposed
alternative to 0004, attached.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.