Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Nazir Bilal Yavuz <byavuz81@gmail.com>

View thread
From: Nazir Bilal Yavuz <byavuz81@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas@vondra.me>, Thomas Munro <thomas.munro@gmail.com>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-04-03T11:37:47Z
Lists: pgsql-hackers 
Hi,

On Thu, 2 Apr 2026 at 21:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>  FORMAT is one of the following:
>     gnu                      GNU tar 1.13.x format
>     oldgnu                   GNU format as per tar <= 1.12
>     pax                      POSIX 1003.1-2001 (pax) format
>     posix                    same as pax
>     ustar                    POSIX 1003.1-1988 (ustar) format
>     v7                       old V7 tar format
> ...
> *This* tar defaults to:
> --format=posix -f- -b20 --quoting-style=escape --rmt-command=/usr/bin/rmt
> --rsh-command=/usr/bin/ssh
>
> So there you have it: pax format by default.  This is unlike what
> I see on RHEL or Fedora:

It seems that the problem also applies to OpenBSD [1]:

-F format
    Specify the output archive format, with the default format being
pax. tar currently supports the following formats:

OpenBSD CI tasks started to fail [2] after bc30c704ad with the errors:

```
Listing only the last 100 lines from a long log.
#   at /home/postgres/postgres/src/bin/pg_waldump/t/001_basic.pl line 440.
#          got: 'pg_waldump: error: pax extensions to tar format are
not supported

#   Failed test 'corrupt backup fails verification: extra_file: matches'
#   at /home/postgres/postgres/src/bin/pg_verifybackup/t/003_corruption.pl
line 198.
#                   'pg_verifybackup: error: pax extensions to tar
format are not supported

Summary of Failures:
239/381 postgresql:pg_waldump / pg_waldump/001_basic
                 ERROR            18.58s   exit status 84
225/381 postgresql:pg_verifybackup / pg_verifybackup/003_corruption
                 ERROR            45.12s   exit status 8
```

I also tried Thomas'
"v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
didn't fix the problem on OpenBSD [4].

[1] https://man.openbsd.org/tar#F
[2] https://cirrus-ci.com/task/5439721360326656
[3] https://postgr.es/m/CA%2BhUKGLMkv_fnGXzVRO8qbx5uHs-qMn151GTJYCfn9w1ZamGNg%40mail.gmail.com
[4] https://cirrus-ci.com/task/5602126958690304

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.