Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-22T04:24:47Z
Lists: pgsql-hackers 
I wrote:
> Unsurprisingly, applying this change to unmodified master results
> in the pg_waldump and pg_verifybackup tests falling over.  More
> surprisingly, they still fall over after applying your fix to the
> decompressors, so there's some other source of garbage trailing
> data.  I haven't figured out what.

In the learn-something-new-every-day dept.: good ol' GNU tar itself
does that.  By default, it zero-pads its output to a multiple of 10kB
after it's written the required terminator.  Moreover, this behavior
is actually specified by POSIX:

  -x format
    Specify the output archive format. The pax utility shall support
    the following formats:
    ...
    ustar
      The tar interchange format; see the EXTENDED DESCRIPTION
      section. The default blocksize for this format for character
      special archive files shall be 10240. Implementations shall
      support all blocksize values less than or equal to 32256 that
      are multiples of 512.

So, astreamer_tar_parser_content's idea that it should disallow more
than 1024 bytes of trailer is completely wrong, which we would have
figured out long ago if the code attempting to enforce that weren't
completely broken.

You could argue that this means the tar files our existing utilities
create aren't POSIX-compliant.  I think it's all right though: we
can just say that we write these files with blocksize 1024 not
blocksize 10240, and tar-file readers are required to accept that
per the above spec text.

However, this discourages me from editorializing on the file trailer
emitted by whatever wrote the tar file we are reading.  I think
emitting it as-is is the most appropriate thing.  So we should just
get rid of astreamer_tar_parser_content's nonfunctional error check
and not change its behavior otherwise.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.