Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-22T23:13:11Z
Lists: pgsql-hackers

Attachments

Everything's pushed, although it looks like it'll be a few hours
before we see results from batta or hachi.

In the meantime, I have one more change that I'd like to propose in
archive_waldump.c: I think we should drop read_archive_file's "count"
parameter and just always try to fill archive_read_buf completely,
as attached.  I don't believe that using a variable count is buying
anything.  I get that the idea was to not load more data than we have
to before init_archive_reader can identify the segment size and start
filtering data we don't need.  But I doubt that that helps a whole
lot: I think most of the cost in this pipeline is going into
decompression and tar format parsing, not into copying data into the
hashtable.  Also, I noticed while I was investigating the bugs we
just fixed that the tar file very likely contains a ton of other
data before the first WAL file.  In the regression test scenarios
it seems to reliably have all of the cluster's actual table data
ahead of the WAL directory.  By using a small read count, we are
slowing down scanning of all that data.  Maybe not by much, I've
not tried to measure it.  But I think the case for using a variable
count is very weak.

Even if you don't like that change, I think we should apply the
parts of the attached that change archive_read_buf_size into an
always-present, always-filled field of XLogDumpPrivate.  Having
its presence be dependent on USE_ASSERT_CHECKING is just asking
for trouble, ie different .c files having a different opinion
of whether it's there.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.