Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Tomas Vondra <tomas@vondra.me>, Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-01T13:26:00Z
Lists: pgsql-hackers 
Hi,

On 2026-04-01 06:39:05 -0400, Andrew Dunstan wrote:
> 
> On 2026-03-31 Tu 10:05 PM, Thomas Munro wrote:
> > On Mon, Mar 30, 2026 at 11:23 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
> > > Thomas Munro<thomas.munro@gmail.com> writes:
> > > > Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
> > > > unlikely to hit this in the wild, and the symptom is a confusing error
> > > > in a maintenance tool, not corruption, so I don't think this is a big
> > > > deal.  I might still try teaching the astreamer code to understand PAX
> > > > 1.0 when it sees it in the next cycle though, for the benefit of
> > > > FreeBSD users.
> > > I agree that this isn't too critical if the effects are confined to
> > > pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
> > > astreamer_tar.c, but it's not clear to me if they'd ever be asked to
> > > parse files made by tar(1) and not by our own sparseness-ignorant
> > > tar-writing code.  If they can be, that'd be a higher-priority reason
> > > to fill in this gap.
> > I pushed the workaround for the test.
> 
> 
> It occurred to me this morning that we probably shouldn't run this test on
> Windows, and if we do we shouldn't be using /dev/null (the Windows
> equivalent of which is just "nul"). The simplest fix would just be to add a
> "!$windows_os" to the if test.

Why should we skip this test on windows?

I think we have historically been way too liberal about sprinkling
!$windows_os test disablements around. More than once there were actual bugs
that we just swept under the rug by disabling the tests that detected them.
Either we support windows or we don't.

Greetings,

Andres Freund

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.