Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Amul Sul <sulamul@gmail.com>
Cc: Andrew Dunstan <andrew@dunslane.net>, Michael Paquier <michael@paquier.xyz>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-21T18:23:55Z
Lists: pgsql-hackers 
I have made some progress on the question of how to reproduce
these failures.  If I do this:

diff --git a/src/bin/pg_waldump/archive_waldump.c b/src/bin/pg_waldump/archive_waldump.c
index b078c2d6960..c389a227be5 100644
--- a/src/bin/pg_waldump/archive_waldump.c
+++ b/src/bin/pg_waldump/archive_waldump.c
@@ -178,7 +178,7 @@ init_archive_reader(XLogDumpPrivate *privateInfo,
     */
    while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
    {
-       if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
+       if (read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
            pg_fatal("could not find WAL in archive \"%s\"",
                     privateInfo->archive_name);

then I get the "could not find WAL in archive" failures in
pg_verifybackup's gzip and lz4 tests, but not zstd.  This happens
reproducibly even without any special hacks on XLOG_BLCKSZ or
wal_compression settings.  Of course, this is not exactly what's
happening on batta/hachi, because they fail on zstd and not the
other two.  But I think it confirms my theory that the problem
is essentially poor handling of EOF boundary conditions.

(Per discussion, there are other bugs here too; I don't mean
to minimize that aspect.)

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.