Re: pg_waldump: support decoding of WAL inside tarfile

amul sul <sulamul@gmail.com>

From: Amul Sul <sulamul@gmail.com>

To: Chao Li <li.evan.chao@gmail.com>

Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>, Robert Haas <robertmhaas@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2026-01-02T12:29:54Z

Lists: pgsql-hackers

Attachments

v10-0001-Refactor-pg_waldump-Move-some-declarations-to-ne.patch (application/octet-stream) patch v10-0001
v10-0002-Refactor-pg_waldump-Separate-logic-used-to-calcu.patch (application/octet-stream) patch v10-0002
v10-0003-Refactor-pg_waldump-Restructure-TAP-tests.patch (application/octet-stream) patch v10-0003
v10-0004-pg_waldump-Add-support-for-archived-WAL-decoding.patch (application/octet-stream) patch v10-0004
v10-0005-pg_waldump-Remove-the-restriction-on-the-order-o.patch (application/octet-stream) patch v10-0005
v10-0006-pg_verifybackup-Delay-default-WAL-directory-prep.patch (application/octet-stream) patch v10-0006
v10-0007-pg_verifybackup-Rename-the-wal-directory-switch-.patch (application/octet-stream) patch v10-0007
v10-0008-pg_verifybackup-enabled-WAL-parsing-for-tar-form.patch (application/octet-stream) patch v10-0008

Attached is the rebased version, includes some code comment improvements.

Regards,
Amul

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Use size_t instead of Size in pg_waldump
- 3f8913f683ba 19 (unreleased) landed
More tar portability adjustments.
- 5079e420b92d 18.4 landed
- bab656bb87b0 19 (unreleased) landed
Further harden tests that might use not-so-compatible tar versions.
- ebba64c08d96 19 (unreleased) landed
- c4b7be4ecb12 18.4 landed
Harden astreamer tar parsing logic against archives it can't handle.
- bc30c704add5 19 (unreleased) landed
- 698eae7db7ab 18.4 landed
Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.
- 852de579a6ee 19 (unreleased) landed
Remove a low-value, high-risk optimization in pg_waldump.
- e9d723487b26 19 (unreleased) landed
Fix misuse of simplehash.h hash operations in pg_waldump.
- ff84efe4fdc0 19 (unreleased) landed
Fix file descriptor leakages in pg_waldump.
- 03b1e30e7aa5 19 (unreleased) landed
Fix poorly-sized buffers in astreamer compression modules.
- 6e243d81c549 19 (unreleased) landed
Remove read_archive_file()'s "count" parameter.
- ca1f1ade3f64 19 (unreleased) landed
Report detailed errors from XLogFindNextRecord() failures.
- 1c162c965a11 19 (unreleased) cited
Fix assorted bugs in archive_waldump.c.
- 860359ea029f 19 (unreleased) landed
Remove nonfunctional tar file trailer size check.
- 5868372bbfb9 19 (unreleased) landed
Fix finalization of decompressor astreamers.
- 9aa1fcc54702 19 (unreleased) landed
- 80785a527b9c 16.14 landed
- 6ccfc4492212 17.10 landed
- 5f9642614275 18.4 landed
- 5540f9c430f6 15.18 landed
Move tar detection and compression logic to common.
- c8a350a43982 19 (unreleased) landed
pg_verifybackup: Enable WAL parsing for tar-format backups
- b3cf461b3cf9 19 (unreleased) landed
pg_waldump: Add support for reading WAL from tar archives
- b15c1513984e 19 (unreleased) landed
pg_waldump: Preparatory refactoring for tar archive WAL decoding.
- f8a0cd267170 19 (unreleased) landed
pg_waldump: Remove file-level global WalSegSz.
- 9a446d0256dc 19 (unreleased) landed
pg_verifybackup: Verify tar-format backups.
- 8dfd31290279 18.0 cited