Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Thomas Munro <thomas.munro@gmail.com>

View thread
From: Thomas Munro <thomas.munro@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-02T22:41:33Z
Lists: pgsql-hackers 
On Fri, Apr 3, 2026 at 7:48 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> *This* tar defaults to:
> --format=posix -f- -b20 --quoting-style=escape --rmt-command=/usr/bin/rmt
> --rsh-command=/usr/bin/ssh

From some quick googling, OpenSUSE is the only distro I could find
that does this, but the fine manual says they plan to make pax the
default upstream so eventually it might be everywhere:

https://www.gnu.org/software/tar/manual/html_section/Formats.html

> Interestingly, pg_verifybackup's t/003_corruption.pl test also fails
> with the same issue, so apparently this platform is even more
> aggressive about sparse-ifying files than Thomas' FreeBSD box.

Looks like sparse files and BTRFS might be a red herring then, if it's
simply been told to put a pax header on every file?

I think my system might be defaulting to "restricted pax", meaning
that pax headers are added to individual files only if necessary,
path-too-long etc, magic sparse format triggered, etc.

https://man.freebsd.org/cgi/man.cgi?query=libarchive-formats

> I'm inclined to push the logic for selecting these tar options
> into some common subroutine in Test::Utils, rather than having
> two copies (and maybe more later).

How about using --format=ustar, instead of that sparse control stuff?
That solves the BSD tar + ZFS problem and the OpenSUSE GNU tar default
format problem in one fell swoop, and actually says what we want
explicitly.

Should we have a <note> near the --path/--wal-path documentation to
state explicitly that usta format is required, and that pax is not
[yet] supported, and that --format=ustar might be required on some
systems?

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.