Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Michael Paquier <michael@paquier.xyz>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Fujii Masao <masao.fujii@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-25T17:28:26Z
Lists: pgsql-hackers 
Hi,

On 2026-03-24 12:11:44 +0900, Michael Paquier wrote:
> On Sun, Mar 22, 2026 at 11:02:20PM -0400, Tom Lane wrote:
> > Proposed patch attached.  There might be an argument for using some
> > other size than 256K for the other two decompressors, but my
> > inclination is to try to make all three use roughly the same block
> > size.  (See also 66ec01dc4.)
>
> The buildfarm has switched mostly to green, except on this one:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42

I think there's a few more failues. Fairywren regularly fails, including in a
run from today.

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2026-03-25%2003%3A48%3A06

There also are a lot of CI failures. E.g.

https://cirrus-ci.com/task/6153854431002624
https://api.cirrus-ci.com/v1/artifact/task/6153854431002624/testrun/build/testrun/pg_waldump/001_basic/log/regress_log_001_basic
# Running: pg_waldump --path C:\msys64\tmp\tNfU5IfQ4a/pg_wal.tar.gz --start 0/01806F48 --end 0/03093BD8
[22:46:25.358](3.991s) not ok 160 - runs with path option and start and end locations: exit code 0
[22:46:25.363](0.005s) #   Failed test 'runs with path option and start and end locations: exit code 0'
#   at C:/cirrus/src/bin/pg_waldump/t/001_basic.pl line 399.
[22:46:25.364](0.001s) ok 161 - runs with path option and start and end locations: no stderr
[22:46:25.365](0.001s) not ok 162 - runs with path option and start and end locations: matches
[22:46:25.365](0.000s) #   Failed test 'runs with path option and start and end locations: matches'
#   at C:/cirrus/src/bin/pg_waldump/t/001_basic.pl line 399.
[22:46:25.366](0.000s) #                   ''
#     doesn't match '(?^:.)'


I was first suspecting that this is due to

commit 1c162c965a1
Author: Fujii Masao <fujii@postgresql.org>
Date:   2026-03-24 22:33:09 +0900

    Report detailed errors from XLogFindNextRecord() failures.

but there are afaict failures from before that:
https://cirrus-ci.com/task/5501609960013824
which is for 4019f725f5d, preceding 1c162c965a1
and
https://cirrus-ci.com/task/5317196043255808


It does feel however the failure frequency has increased substantially:
https://cirrus-ci.com/github/postgres/postgres/master

Greetings,

Andres Freund

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.