Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Andrew Dunstan <andrew@dunslane.net>

View thread
From: Andrew Dunstan <andrew@dunslane.net>
To: Andres Freund <andres@anarazel.de>
Cc: Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Tomas Vondra <tomas@vondra.me>, Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-01T14:19:32Z
Lists: pgsql-hackers 
On 2026-04-01 We 9:26 AM, Andres Freund wrote:
> Hi,
>
> On 2026-04-01 06:39:05 -0400, Andrew Dunstan wrote:
>> On 2026-03-31 Tu 10:05 PM, Thomas Munro wrote:
>>> On Mon, Mar 30, 2026 at 11:23 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>>>> Thomas Munro<thomas.munro@gmail.com> writes:
>>>>> Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
>>>>> unlikely to hit this in the wild, and the symptom is a confusing error
>>>>> in a maintenance tool, not corruption, so I don't think this is a big
>>>>> deal.  I might still try teaching the astreamer code to understand PAX
>>>>> 1.0 when it sees it in the next cycle though, for the benefit of
>>>>> FreeBSD users.
>>>> I agree that this isn't too critical if the effects are confined to
>>>> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
>>>> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
>>>> parse files made by tar(1) and not by our own sparseness-ignorant
>>>> tar-writing code.  If they can be, that'd be a higher-priority reason
>>>> to fill in this gap.
>>> I pushed the workaround for the test.
>>
>> It occurred to me this morning that we probably shouldn't run this test on
>> Windows, and if we do we shouldn't be using /dev/null (the Windows
>> equivalent of which is just "nul"). The simplest fix would just be to add a
>> "!$windows_os" to the if test.
> Why should we skip this test on windows?
>
> I think we have historically been way too liberal about sprinkling
> !$windows_os test disablements around. More than once there were actual bugs
> that we just swept under the rug by disabling the tests that detected them.
> Either we support windows or we don't.
>

Maybe I misunderstood, but I didn't think this was going to be an issue 
on NTFS.

In general I agree with you, though. I try to avoid skipping things on 
Windows.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.