Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: Amul Sul <sulamul@gmail.com>
Cc: Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-01-26T17:22:33Z
Lists: pgsql-hackers 
On Fri, Jan 23, 2026 at 7:27 AM Amul Sul <sulamul@gmail.com> wrote:
> Another option I previously considered was adding the filtration logic
> inside the archive streamer itself. However, since the very first read
> is required to calculate the WAL segment size, the filter check cannot
> be performed immediately. However, we could send a signal to the
> archive streamer via privateInfo (e.g., a read_any_wal or
> skip_wal_check boolean flag) to disable the filtration check until the
> size is calculated. But that approach isn't very elegant; if the first
> WAL page we read belongs to a segment we actually want to skip, we
> would still have to run the filter check and handle the skip/removal
> logic outside of the streamer (i.e., inside init_archive_reader()).
> This would result in performing the same filtration check in two
> different places.

I mean, I don't really buy this logic. If the information added to
privateInfo is "here's the LSN before which you can remove stuff," and
it starts out initialized to 0/0, then the read of the first WAL page
causes no problem at all, because nothing is before 0/0. After it gets
updated to some non-zero value, the next call to
astreamer_waldump_content() can handle discarding any data we don't
need.

IMHO, the best argument for keeping things are is that in some cases,
that decision might result in a bit of delay in discarding old data,
but I don't think that really matters. I think all that we care about
is the peak memory utilization of an operation, and I don't think that
an explicit signaling system should increase that at all.

That said, I'm certainly willing to consider other ideas about how
this can work. However, I feel strongly that the logic needs to be not
only correct, but clear and well-explained. Setting cur_wal to NULL to
make the astreamer skip without adequate comments doesn't meet that
standard. Maybe with some better comments it's all right, but frankly
I'm a bit skeptical. Right now, you're using whether or not cur_wal is
NULL as a signal to skip data or not skip data. How is that better
than passing down the LSN and TLI that you want to read next and
letting the astreamer figure out what to do itself? It's a signaling
mechanism either way, but it seems a lot easier to figure out whether
we always keep the LSN and TLI updated properly than to figure out
whether cur_wal is always NULL at exactly the right times.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.