Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: Euler Taveira <euler@eulerto.com>
Cc: Amul Sul <sulamul@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-01-28T13:03:02Z
Lists: pgsql-hackers 
On Tue, Jan 27, 2026 at 10:02 PM Euler Taveira <euler@eulerto.com> wrote:
> + * archive_waldump.c
> + *     A generic facility for reading WAL data from tar archives via archive
> + *     streamer.
>
> The other tools (pg_basebackup and pg_verifybackup) that also use astreamer API
> named this similar file as astreamer_SOMETHING.c. It seems a good idea to
> follow the same pattern, no? Maybe astreamer_tar_archive.c or
> astreamer_archive.c.

There shouldn't be anything specific to tar files in here, and
astreamer_archive would be meaningless, since the "a" in "astreamer"
stands for archive. What this file is is an archive streamer specific
to pg_waldump, hence the name.

> Can it enforce a specific order? tar follows an arbitrary order in which the
> files is returned by the filesystem. You've been debating a solution to buffer
> the WAL contents using memory or spilled files. If it always create the tar in
> an alphabetical order, you can reduce the scope of this patch. (Didn't look
> what challenges are expected to use a sorted list to generate the tar file.)

It's posible to create a tar file in a specific order by specifying
command-line arguments to tar in the order you want the tar file to be
built. But I think the real thing here is that this limitation is
lifted by the following patch. Whether it's worth splitting it apart
into two patches this way is debatable. As I have pointed out in my
previous reviews, the split hasn't been done very cleanly.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.