Horton
Menu

Re: pg_waldump: support decoding of WAL inside tarfile

Andrew Dunstan <andrew@dunslane.net>

View thread
From: Andrew Dunstan <andrew@dunslane.net>
To: Amul Sul <sulamul@gmail.com>, Zsolt Parragi <zsolt.parragi@percona.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Chao Li <li.evan.chao@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2026-03-20T19:33:18Z
Lists: pgsql-hackers 
On 2026-03-20 Fr 9:26 AM, Amul Sul wrote:
> On Fri, Mar 20, 2026 at 5:01 PM Amul Sul <sulamul@gmail.com> wrote:
>> On Fri, Mar 20, 2026 at 2:18 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
>>> Hello!
>>>
>>> Path is ignored with a positional argument, I think this is a bug?
>>>
>>> This fails:
>>>
>>> pg_waldump --path /wal/dir 000000010000000000000001
>>>
>>> And this works:
>>>
>>> pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8
>>>
>> Good catch! I've fixed this in the attached version and updated a test
>> case to cover this scenario.
>>
>>> +{
>>> + int fname_len = strlen(fname);
>>> +
>>>
>>> Shouldn't this use size_t?
>>>
>> Okay, that can be used. I’ve done the same in the attached version.
>>
>>> + /*
>>> + * Setup temporary directory to store WAL segments and set up an exit
>>> + * callback to remove it upon completion.
>>> + */
>>> + setup_tmpwal_dir(waldir);
>>>
>>> Maybe this could be deferred to be created only on first use? If I
>>> understand correctly, in a typical scenario waldump won't use this
>>> temporary directory, yet it always creates it.
>> Yeah, that optimization can be done, but passing the waldir  -- which
>> is only used once -- to the point where the first temp file is created
>> would require quite a bit of code refactoring that doesn't seem to
>> offer much gain, IMO.
>>
> Since Andrew also leans toward creating the directory only when
> needed, I have reconsidered the approach. I think we can pass waldir
> (the archive directory) via XLogDumpPrivate, and I’ve implemented that
> in the attached version.
>

Thanks, committed with very minor tweaks.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.