Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tomas Vondra <tomas@vondra.me>

View thread
From: Tomas Vondra <tomas@vondra.me>
To: Tom Lane <tgl@sss.pgh.pa.us>, Thomas Munro <thomas.munro@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-02T17:13:08Z
Lists: pgsql-hackers 
On 4/2/26 18:29, Tom Lane wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
>> On Thu, Apr 2, 2026 at 2:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> So I think we need something like the attached, in addition
>>> to what I sent before.  This just makes astreamer_tar.c use
>>> the isValidTarHeader function that pg_dump already had.
> 
>> LGTM.
> 
> Pushed, thanks for reviewing!  In the event I decided to back-patch to
> v18, where these fixes could protect pg_verifybackup against tar files
> it can't handle.  In older branches the astreamer (nee bbstreamer)
> logic only exists inside pg_basebackup, and IIUC that's not really
> exposed to tar files that weren't made by our own code.  So I did
> not bother with back-patching further, though it'd be possible to
> do that if someone knows a scenario where it'd matter.
> 

It seems jay/hippopotamus failed on this, for some reason. Those two
animals are on the same host, just using different compilers. I did
update+reboot the machine this afternoon (before the test), but I don't
see why that would cause the failure.

Maybe there's something special about OpenSUSE?

-- 
Tomas Vondra

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.