Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Thomas Munro <thomas.munro@gmail.com>

View thread
From: Thomas Munro <thomas.munro@gmail.com>
To: Nazir Bilal Yavuz <byavuz81@gmail.com>
Cc: Sami Imseih <samimseih@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-04T01:07:04Z
Lists: pgsql-hackers 
On Sat, Apr 4, 2026 at 4:30 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> On Fri, 3 Apr 2026 at 17:59, Sami Imseih <samimseih@gmail.com> wrote:
> > > On Sat, Apr 4, 2026 at 12:38 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > > > I also tried Thomas'
> > > > "v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
> > > > didn't fix the problem on OpenBSD [4].
> > >
> > > Apparently it wants -F ustar, like this.  Funny that it passed on the
> > > build farm animals though.  Oh, it looks like they changed the default
> > > fairly recently.
> >
> > LGTM with just a correction of my earlier comment.
>
> Thanks for the patches! I confirm that both v3 and v4 fix the problem
> for OpenBSD CI.

Pushed, after testing on an OpenBSD VM and making some corrections:

* I'd screwed up the test command line in a way that worked by coincidence
** OpenBSD tar writes to a tape device by default, so use -f /dev/null
** I'd forgotten == 0, so the result was inverted, hiding that screwup
* -f /dev/null is a better form for all of them because the default
destination is a build option
* needed elsif instead of if, or BSD tar finished up getting both
--format=ustar and -F ustar
* ran perltidy, keeping only the hunks due to this patch

CI passes and shows "212 subtests passed" for all five Unixen +
Windows/mingw, but only "156 subtests passed" for Windows/MSVC.
.cirrus.tasks.yml appears to use the same $TAR for both, namely the
system tar, so I think we can say that *this* thing is working, but
something else might be wrong with our scripting glue somewhere?

The other OSes on our list are AIX and Solaris.  From a quick look at
their manuals, I don't foresee issues with pax or large UIDs.

Hopefully that covers everything!

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.