Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Michael Paquier <michael@paquier.xyz>
Cc: Andrew Dunstan <andrew@dunslane.net>, Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>, Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-25T17:25:21Z
Lists: pgsql-hackers

Attachments

Michael Paquier <michael@paquier.xyz> writes:
> The buildfarm has switched mostly to green, except on this one:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42

Yeah, there are some other weird failures on other machines too.
There's also the problem we already knew about of FD leakage breaking
cleanup of the temp file directory on Windows.

I wrote a patch to fix the FD leakage problem (v8-0001 attached).
I don't have Windows at hand, but I tested it by dint of having
the atexit callback invoke "lsof" to see if there were any open
files in the temp directory.

I also occasionally saw some of the weird errors mentioned above.
After much debugging, I believe that the issue is that
archive_waldump.c is unaware that inserting or deleting entries
in a simplehash.h hash table can cause other entries to move.
That can break privateInfo->cur_file, and it can also break
read_archive_wal_page which thought it could just re-use its
entry pointer after calling read_archive_file.  v8-0002 attached
fixes that, and I'm not seeing weird failures anymore.

Two additional thoughts:

1. The amount of data that pg_waldump's TAP tests use is not
sufficient to trigger these problems with any degree of reliability.
I'm hesitant to make the tests run longer, but clearly we do not
have adequate coverage now.

2. I didn't do it here, but I urgently think we should rip out
read_archive_wal_page's stanza that truncates the entry's
"buf" string (the "if (privateInfo->decoding_started)" part).
My faith in this code in general is at rock bottom, and my faith in
the extent to which we've tested it is somewhere below ground level.
I don't think we need rickety optimizations that serve only to
keep the active hashtable entry to less than 16MB, when we're going
to reclaim that space altogether as soon as we've finished dumping
that segment.  This truncation scares me because it adds a whole
'nother level of poorly-documented complexity to the invariants
around what is in entry->buf.  Also, while we theoretically should
not need to spill the entry after this point, if we did we would
write a corrupted spill file.

			regards, tom lane

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.