Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-21T03:49:02Z
Lists: pgsql-hackers 
Andrew Dunstan <andrew@dunslane.net> writes:
> Thanks, committed with very minor tweaks.

Buildfarm members batta and hachi don't like this very much.
They fail the pg_verifybackup tests like so:

# Running: pg_verifybackup --exit-on-error /home/admin/batta/buildroot/HEAD/pgsql.build/src/bin/pg_verifybackup/tmp_check/t_008_untar_primary_data/backup/server-backup
pg_waldump: error: could not find WAL in archive "base.tar.zst"
pg_verifybackup: error: WAL parsing failed for timeline 1

Only the zstd-compression case fails.  I've spent several hours trying
to reproduce this, without any luck, although I can get a similar
failure in only the gzip case if I build with --with-wal-blocksize=64.
I do not have an explanation for the seeming cross-platform
difference.  However after adding a lot of debug tracing, I believe
I see the bug, or at least a related bug.  This bit in
archive_waldump.c's init_archive_reader is where the error comes from:

    /*
     * Read until we have at least one full WAL page (XLOG_BLCKSZ bytes) from
     * the first WAL segment in the archive so we can extract the WAL segment
     * size from the long page header.
     */
    while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
    {
        if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
            pg_fatal("could not find WAL in archive \"%s\"",
                     privateInfo->archive_name);

        entry = privateInfo->cur_file;
    }

That looks plausible but is in fact utterly broken when there's not a
lot of WAL data in the archive, as there is not in this test case.
There are at least two problems:

1. read_archive_file reads some data from the source WAL archive and
shoves it into the astreamer decompression pipeline.  However, once it
runs out of source data, it just returns zero and we fail immediately.
This does not account for the possibility --- nay, certainty --- that
there is data queued inside the decompression pipeline.  So this
doesn't work if the data we need has been compressed into less than
XLOG_BLCKSZ worth of compressed data.  (I suppose that the seeming
cross-platform differences have to do with the effectiveness of the
compression algorithm, but I don't really understand why it'd not be
the same everywhere.)  We need to do astreamer_finalize once we run
out of source data.  I think the cleanest place to handle that would
be inside read_archive_file, but its return convention will need some
rework if we want to put it there (because rc == 0 shouldn't cause an
immediate failure if we were able to finalize some more data).  As an
ugly experiment I put an astreamer_finalize call into the rc == 0 path
of the above loop, but it still didn't work, because:

2. If the decompression pipeline reaches the end of the WAL file that
we want, the ASTREAMER_MEMBER_TRAILER case in
astreamer_waldump_content instantly resets privateInfo->cur_file to
NULL.  Then the loop in init_archive_reader cannot exit successfully,
and it will just read till the end of the archive and fail.

I see that of the three callers of read_archive_file, only
get_archive_wal_entry is aware of this possibility; but
init_archive_reader certainly needs to deal with it and I bet
read_archive_wal_page does too.  Moreover, get_archive_wal_entry's
solution looks to me like a fragile kluge that probably doesn't work
reliably either, the reason being that privateInfo->cur_file can
change multiple times during a single call to read_archive_file,
if the WAL data has been compressed sufficiently.  That whole API
seems to need some rethinking, not to mention better documentation
than the zero it has now.

While I'm bitching: this error message "could not find WAL in archive
\"%s\"" seems to me to be completely misleading and off-point.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.