Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

amul sul <sulamul@gmail.com>

View thread
From: Amul Sul <sulamul@gmail.com>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-19T10:20:59Z
Lists: pgsql-hackers

Attachments

On Wed, Mar 18, 2026 at 8:46 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Wed, Mar 18, 2026 at 5:15 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Wed, Mar 11, 2026 at 10:38 PM Andrew Dunstan <andrew@dunslane.net> wrote:
> > > [...]
> > > Looks pretty good. I have squashed them into three patches I think are committable. Also attached is a diff showing what's changed - mainly this:
> > >
> > > . --follow + tar archive rejected (pg_waldump.c) — new validation prevents a confusing pg_fatal when combining --follow with a tar archive
> > > . error messages split (archive_waldump.c) — the single "could not read file" error is now two distinct messages: "WAL segment is too short" (truncated file) vs "unexpected end of archive" (archive EOF) - Fixes an issue raised in review
> > > . hash table cleanup (archive_waldump.c) — free_archive_reader now iterates and frees all remaining hash entries and destroys the table
> > >
> >
> > The final squashed version looks good to me, thank you. But, I would
> > like to propose splitting the 0001 patch into two separate commits: a
> > preparatory refactoring of the pg_waldump code and a standalone commit
> > that moves the tar archive detection and compression logic to a common
> > location, as the latter is an independent improvement to the existing
> > codebase. Additionally, since the test file refactoring was only kept
> > separate to facilitate the review and has already been reviewed, I
> > suggest merging those changes into the main feature patch i.e. 0002.
> > All other elements should remain in a single preparatory refactoring
> > patch for pg_waldump.
> >
> > Attached is the version that includes the proposed split. No
> > additional changes to 0002 and 0003 patches.
> >
>
> Added the two missing 'Reviewed-by' lines to the credit section of the
> commit message and did a minor optimization in get_archive_wal_entry.
>

Attaching an updated version. It includes some tweaks to code
comments, adds an assert inside get_archive_wal_entry(), moves the
archive_read_buf_size declaration and usage into an assert-enabled
check, and makes a minor change to precheck_tar_backup_file() to
assign out-variables only after successful validation.

Regards,
Amul

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.