Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Andrew Dunstan <andrew@dunslane.net>

View thread
From: Andrew Dunstan <andrew@dunslane.net>
To: Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-02T11:36:25Z
Lists: pgsql-hackers 
On 2026-04-01 We 8:16 PM, Thomas Munro wrote:
> On Thu, Apr 2, 2026 at 7:25 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> Also, if we are admitting the possibility that what we are reading
>> was made by a platform-supplied tar and not our own code, I think
>> it verges on lunacy to behave as though unsupported typeflags are
>> regular files.
> Yeah, if this is the first time we parse files we didn't make then
> that makes total sense.  I was a bit unsure of that question when I
> suggested we reject pax only after we've failed to find a file, in
> case there are scenarios that work today with harmless ignorable pax
> headers that don't change the file name.
>
>> So I think we need something more or less like the attached.
> LGTM.  Tested with both tars here.  I updated that little test patch
> for this.  Not sure if you think it's worth a test though, now that
> it's so simple.
>
> @Andrew: I tried usingFile::Spec->devnull() this time.  Are you able
> to check if this works OK on Windows, applied on top of Tom's patch?
> AFAIK should be able to run this new test and pass, not skip it.  But
> it could be that the shell invocation needs tweaking.  It's hard to
> tell from CI.  (Huh, apparently Windows ships a copy of BSD tar as
> C:\Windows\System32\tar.exe these days.)


Yes, that appears to work. I would put a "2>&1" at the end - we don't 
care about the output, just whether or not it succeeds:


C:\Windows\system32>perl -MFile::Spec -e "print File::Spec->devnull();"
nul

C:\Windows\system32>tar --no-read-sparse -c - nul > nul 2>&1 && echo hello

C:\Windows\system32>tar --no-read-sparse -c - nul > nul 2>&1 || echo hello
hello


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.