Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Tomas Vondra <tomas@vondra.me>
Cc: Thomas Munro <thomas.munro@gmail.com>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-28T23:12:11Z
Lists: pgsql-hackers 
Tomas Vondra <tomas@vondra.me> writes:
> On 3/28/26 23:36, Tom Lane wrote:
>> It's apparently been there and been default since FreeBSD 13.1.
>> This leads one to wonder how come BF member dikkop is managing
>> to run this test successfully.  I speculate that it's using a
>> filesystem type that doesn't do sparse files (cc'ing Vondra
>> for confirmation on that).

> It's running on ufs. But I think the explanation is very simple. We had
> a short power outage on Thursday, and the FreeBSD machine failed to boot
> properly after the power was restored. IIUC this test is new, right?

Not that new, it dates to b15c15139, about a week ago.

I've reproduced Thomas' failure on a local FreeBSD 15.0 image
using zfs, and confirmed that this cowboy hack fixes it:

diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 8bb8fa225f6..2904f4ef118 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -346,7 +346,7 @@ sub generate_archive
        # move into the WAL directory before archiving files
        my $cwd = getcwd;
        chdir($directory) || die "chdir: $!";
-       command_ok([$tar, $compression_flags, $archive, @files]);
+       command_ok([$tar, $compression_flags, $archive, '--no-read-sparse', @files]);
        chdir($cwd) || die "chdir: $!";
 }

Of course we can't commit that, but maybe somebody wants to
product-ize it.  (Not me, at least not today.)

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.