pg_waldump: support decoding of WAL inside tarfile
amul sul <sulamul@gmail.com>
From: Amul Sul <sulamul@gmail.com>
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2025-08-07T14:17:08Z
Lists: pgsql-hackers
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Move tar detection and compression logic to common.
- c8a350a43982 19 (unreleased) landed
-
pg_verifybackup: Enable WAL parsing for tar-format backups
- b3cf461b3cf9 19 (unreleased) landed
-
pg_waldump: Add support for reading WAL from tar archives
- b15c1513984e 19 (unreleased) landed
-
pg_waldump: Preparatory refactoring for tar archive WAL decoding.
- f8a0cd267170 19 (unreleased) landed
-
pg_verifybackup: Verify tar-format backups.
- 8dfd31290279 18.0 cited
Attachments
- v1-0001-Refactor-pg_waldump-Move-some-declarations-to-new.patch (application/x-patch) patch v1-0001
- v1-0002-Refactor-pg_waldump-Separate-logic-used-to-calcul.patch (application/x-patch) patch v1-0002
- v1-0003-Refactor-pg_waldump-Restructure-TAP-tests.patch (application/x-patch) patch v1-0003
- v1-0004-pg_waldump-Rename-directory-creation-routine-for-.patch (application/x-patch) patch v1-0004
- v1-0005-pg_waldump-Add-support-for-archived-WAL-decoding.patch (application/x-patch) patch v1-0005
- v1-0006-WIP-pg_waldump-Remove-the-restriction-on-the-orde.patch (application/x-patch) patch v1-0006
- v1-0007-pg_verifybackup-Delay-default-WAL-directory-prepa.patch (application/x-patch) patch v1-0007
- v1-0008-pg_verifybackup-Rename-the-wal-directory-switch-t.patch (application/x-patch) patch v1-0008
- v1-0009-pg_verifybackup-enabled-WAL-parsing-for-tar-forma.patch (application/x-patch) patch v1-0009
Hi All, Attaching patch to support a new feature that let pg_waldump decode WAL files directly from a tar archive. This was worked to address a limitation in pg_verifybackup[1], which couldn't parse WAL files from tar-formatted backups. The implementation will align with pg_waldump's existing xlogreader design, which uses three callback functions to manage WAL segments: open, read, and close. For tar archives, however, the approach will be simpler. Instead of using separate callbacks for opening and closing, the tar archive will be opened once at the start and closed explicitly at the end. The core logic will be in the WAL page reading callback. When xlogreader requests a new WAL page, this callback will be invoked. It will then call the archive streamer routine to read the WAL data from the tar archive into a buffer. This data will then be copied into xlogreader's own buffer, completing the read. Essentially, this is plumbing work: the new code will be responsible for getting WAL data from the tar archive and feeding it to the existing xlogreader. All other WAL page and record decoding logic, which is already robust within xlogreader, will be reused as is. This feature is being implemented in a series of patches as: - Refactoring: The first few patches (0001-0004) are dedicated to refactoring and minor code changes. - 005: This patch introduces the core functionality for pg_waldump to read WAL from a tar archive using the same archive streamer (fe_utils/astreamer.h) used in pg_verifybackup. This version requires WAL files in the archive to be in sequential order. - 006: This patch removes the sequential order restriction. If pg_waldump encounters an out-of-order WAL file, it writes the file to a temporary directory. The utility will then continue decoding and read from this temporary location later. - 007 and onwards: These patches will update pg_verifybackup to remove the restriction on WAL parsing for tar-formatted backups. 008 patch renames the "--wal-directory" switch to "--wal-path" to make it more generic, allowing it accepts a directory path or a tar archive path. ----------------------------------- Known Issues & Status: ----------------------------------- - Timeline Switching: The current implementation in patch 006 does not correctly handle timeline switching. This is a known issue, especially when a timeline change occurs on a WAL file that has been written to a temporary location. - Testing: Local regression tests on CentOS and macOS M4 are passing. However, some tests on macOS Sonoma (specifically 008_untar.pl and 010_client_untar.pl) are failing in the GitHub workflow with a "WAL parsing failed for timeline 1" error. This issue is currently being investigated. Please take a look at the attached patch and let me know your thoughts. This is an initial version, and I am making incremental improvements to address known issues and limitations. 1] https://git.postgresql.org/pg/commitdiff/8dfd3129027969fdd2d9d294220c867d2efd84aa -- Regards, Amul Sul EDB: http://www.enterprisedb.com