Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-28T21:49:29Z
Lists: pgsql-hackers 
Thomas Munro <thomas.munro@gmail.com> writes:
> This fails 100% of the time on my machine, even after e9d72348 and ff84efe4, eg:

> # Running: pg_waldump --path /tmp/D8WG1Sv2HE/pg_wal.tar --start
> 0/017A2610 --end 0/02093848
> [09:43:29.288](0.148s) not ok 104 - runs with path option and start
> and end locations: exit code 0
> [09:43:29.289](0.001s) #   Failed test 'runs with path option and
> start and end locations: exit code 0'
> #   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
> line 402.
> [09:43:29.290](0.001s) not ok 105 - runs with path option and start
> and end locations: no stderr
> [09:43:29.291](0.001s) #   Failed test 'runs with path option and
> start and end locations: no stderr'
> #   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
> line 402.
> [09:43:29.291](0.000s) #          got: 'pg_waldump: error: could not
> find WAL "000000010000000000000002" in archive "pg_wal.tar"
> # '

Huh.  What is your platform exactly?  Maybe more to the point,
what is the tar program you're using?

> Seems like it already stepped over 000000010000000000000002 earlier?
> Could it be a table-of-contents order dependency bug or something like
> that?

If you look at the TAP script, you'll see that it tries to randomize
the order of the entries in the tar file (see sub generate_archive).
So if that's the problem, it shouldn't reproduce 100%, and also we
should be seeing lots of freckles on the buildfarm.  We're not, so
there must be something off-the-beaten-track about your test
environment.

I'm certainly more than prepared to believe there's still bugs
lurking here, but we need to figure out why you're seeing it.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.