Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Use size_t instead of Size in pg_waldump

  2. More tar portability adjustments.

  3. Further harden tests that might use not-so-compatible tar versions.

  4. Harden astreamer tar parsing logic against archives it can't handle.

  5. Fix pg_waldump/t/001_basic.pl with BSD tar on ZFS.

  6. Remove a low-value, high-risk optimization in pg_waldump.

  7. Fix misuse of simplehash.h hash operations in pg_waldump.

  8. Fix file descriptor leakages in pg_waldump.

  9. Fix poorly-sized buffers in astreamer compression modules.

  10. Remove read_archive_file()'s "count" parameter.

  11. Report detailed errors from XLogFindNextRecord() failures.

  12. Fix assorted bugs in archive_waldump.c.

  13. Remove nonfunctional tar file trailer size check.

  14. Fix finalization of decompressor astreamers.

  15. Move tar detection and compression logic to common.

  16. pg_verifybackup: Enable WAL parsing for tar-format backups

  17. pg_waldump: Add support for reading WAL from tar archives

  18. pg_waldump: Preparatory refactoring for tar archive WAL decoding.

  19. pg_waldump: Remove file-level global WalSegSz.

  20. pg_verifybackup: Verify tar-format backups.

  1. pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-08-07T14:17:08Z

    Hi All,

Attaching patch to support a new feature that let pg_waldump decode
WAL files directly from a tar archive. This was worked to address a
limitation in pg_verifybackup[1], which couldn't parse WAL files from
tar-formatted backups.

The implementation will align with pg_waldump's existing xlogreader
design, which uses three callback functions to manage WAL segments:
open, read, and close. For tar archives, however, the approach will be
simpler. Instead of using separate callbacks for opening and closing,
the tar archive will be opened once at the start and closed explicitly
at the end.

The core logic will be in the WAL page reading callback. When
xlogreader requests a new WAL page, this callback will be invoked. It
will then call the archive streamer routine to read the WAL data from
the tar archive into a buffer. This data will then be copied into
xlogreader's own buffer, completing the read.

Essentially, this is plumbing work: the new code will be responsible
for getting WAL data from the tar archive and feeding it to the
existing xlogreader. All other WAL page and record decoding logic,
which is already robust within xlogreader, will be reused as is.

This feature is being implemented in a series of patches as:

- Refactoring: The first few patches (0001-0004) are dedicated to
refactoring and minor code changes.

- 005: This patch introduces the core functionality for pg_waldump to
read WAL from a tar archive using the same archive streamer
(fe_utils/astreamer.h) used in pg_verifybackup. This version requires
WAL files in the archive to be in sequential order.

- 006: This patch removes the sequential order restriction. If
pg_waldump encounters an out-of-order WAL file, it writes the file to
a temporary directory. The utility will then continue decoding and
read from this temporary location later.

- 007 and onwards: These patches will update pg_verifybackup to remove the
restriction on WAL parsing for tar-formatted backups. 008 patch renames the
"--wal-directory" switch to "--wal-path" to make it more generic, allowing
it accepts a directory path or a tar archive path.

-----------------------------------
Known Issues & Status:
-----------------------------------
- Timeline Switching: The current implementation in patch 006 does not
correctly handle timeline switching. This is a known issue, especially
when a timeline change occurs on a WAL file that has been written to a
temporary location.

- Testing: Local regression tests on CentOS and macOS M4 are passing.
However, some tests on macOS Sonoma (specifically 008_untar.pl and
010_client_untar.pl) are failing in the GitHub workflow with a "WAL
parsing failed for timeline 1" error. This issue is currently being
investigated.

Please take a look at the attached patch and let me know your
thoughts. This is an initial version, and I am making incremental
improvements to address known issues and limitations.


1] https://git.postgresql.org/pg/commitdiff/8dfd3129027969fdd2d9d294220c867d2efd84aa

--
Regards,
Amul Sul
EDB: http://www.enterprisedb.com

  2. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-08-25T12:28:44Z

    On Thu, Aug 7, 2025 at 7:47 PM Amul Sul <sulamul@gmail.com> wrote:
> [....]
> -----------------------------------
> Known Issues & Status:
> -----------------------------------
> - Timeline Switching: The current implementation in patch 006 does not
> correctly handle timeline switching. This is a known issue, especially
> when a timeline change occurs on a WAL file that has been written to a
> temporary location.
>

This is still pending and will be addressed in the next version.
Therefore, patch 0006 remains marked as WIP.

> - Testing: Local regression tests on CentOS and macOS M4 are passing.
> However, some tests on macOS Sonoma (specifically 008_untar.pl and
> 010_client_untar.pl) are failing in the GitHub workflow with a "WAL
> parsing failed for timeline 1" error. This issue is currently being
> investigated.
>

This has been fixed in the attached version; all GitHub workflow tests
are now fine.

Regards,
Amul

  3. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-08-26T11:52:20Z

    On Mon, Aug 25, 2025 at 5:58 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Thu, Aug 7, 2025 at 7:47 PM Amul Sul <sulamul@gmail.com> wrote:
> > [....]
> > -----------------------------------
> > Known Issues & Status:
> > -----------------------------------
> > - Timeline Switching: The current implementation in patch 006 does not
> > correctly handle timeline switching. This is a known issue, especially
> > when a timeline change occurs on a WAL file that has been written to a
> > temporary location.
> >
>
> This is still pending and will be addressed in the next version.
> Therefore, patch 0006 remains marked as WIP.
>

After testing pg_waldump, I have realised that my previous
understanding of its timeline handling was incorrect. I had mistakenly
assumed by reading xlogreader code that it would use the same
timeline-switching logic found in xlogreader, without first verifying
this behavior. In testing, I found that pg_waldump does not follow
timeline switches. Instead, it expects all WAL files to be from a
single timeline, which is either specified by the user or determined
from the starting segment or default 1.

This is a positive finding, as it means we don't need to make
significant changes to align pg_waldump's current behavior. The
attached patches are now complete and no longer works in progress --
read for review. Additionally, I've dropped patch v2-0004 because it is
no longer necessary. The primary patches that implement the proposed
feature are now 0004 and 0005 in the attached set.

Regards,
Amul

  4. Re: pg_waldump: support decoding of WAL inside tarfile

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-09-08T13:37:02Z

    On Tue, Aug 26, 2025 at 1:53 PM Amul Sul <sulamul@gmail.com> wrote:
>
[..patch]

Hi Amul!

0001: LGTM, maybe I would just slightly enhance the commit message
("This is in preparation for adding a second source file to this
directory.") -- maye bit a bit more verbose or use a message from
0002?
0002: LGTM
0003: LGTM

Tested here (after partial patch apply, and test suite did work fine).

0004:

    a.  Why should it be necessary to provide startLSN (-s) ? Couldn't
it autodetect the first WAL (tar file) inside and just use that with
some info message?
    $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar
    pg_waldump: error: no start WAL location given

    b. Why would it like to open "blah" dir if I wanted that "blah"
segment from the archive? Shouldn't it tell that it was looking in the
archive and couldn find it inside?
    $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar blah
    pg_waldump: error: could not open file "blah": Not a directory

    c. It doesnt work when using SEGSTART, but it's there:
    $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar
000000010000000000000059
    pg_waldump: error: could not open file "000000010000000000000059":
Not a directory
    $ tar tf /tmp/base/pg_wal.tar | head -1
    000000010000000000000059

    d.     I've later noticed that follow-up patches seem to use the
-s switch and there it seems to work OK. The above SEGSTART issue was
not detected, probably because tests need to be extended cover  of
segment name rather than just --start LSN (see test_pg_waldump):
    $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar --stats
-s 0/59000358
    pg_waldump: first record is after 0/59000358, at 0/590003E8,
skipping over 144 bytes
    WAL statistics between 0/590003E8 and 0/61000000:
    [..]

    e. Code around`if (walpath == NULL && directory != NULL)` needs
some comments.

    f. Code around `if (fname != NULL && is_tar_file(fname,
&compression))` , so if fname is WAL segment here
(00000001000000000000005A) and we do check again if that has been
tar-ed (is_tar_file())? Why?

    g. Just a question: the commit message says `Note that this patch
requires that the WAL files within the archive be in sequential order;
an error will be reported otherwise`. I'm wondering if such
occurrences are known to be happening in the wild? Or is it just an
assumption that if someone would modify the tar somehow? (either way
we could just add a reason why we need to handle such a case if we
know -- is manual alternation the only source of such state?). For the
record, I've tested crafting custom archives with out of sequence WAL
archives and the code seems to work (it was done using: tar --append
-f pg_wal.tar --format=ustar ..)

    h. Anyway, in case of typo/wrong LSN, 0004 emits wrong error
message I think:

    $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar --stats
-s 0/50000358
    pg_waldump: error: WAL files are not archived in sequential order
    pg_waldump: detail: Expecting segment number 80 but found 89.

    it's just that the 50000358 LSN above is below the minimal LSN
present in the WAL segments (first segment is 000000010000000000000059
there, i've just intentionally provided a bad value 50.. as a typo and
it causes the wrong message). Now it might not be an issue as with
0005 patch the same test behaves OK (`pg_waldump: error: could not
find a valid record after 0/50000358`). It is just relevant if this
would be committed not all at once.

    i. If I give wrong --timeline=999 to pg_waldump it fails with
misleading error message: could not read WAL data from "pg_wal.tar"
archive: read -1 of 8192

0005:
    a. I'm wondering if we shouldn't log (to stderr?) some kind of
notification message (just once) that non-sequential WAL files were
discovered and that pg_waldump is starting to write to $somewhere as
it may be causing bigger I/O than anticipated when running the
command. This can easily help when troubleshooting why it is not fast,
and also having set TMPDIR to usually /tmp can be slow or too small.

    b. IMHO member_prepare_tmp_write() / get_tmp_wal_file_path() with
TMPDIR can be prone to symlink attack. Consider setting TMPDIR=/tmp .
We are writing to e.g. /tmp/<WALsegment>.waldump.tmp in 0004 , but
that path is completely guessable. If an attacker prepares some
symlinks and links those to some other places, I think the code will
happily open and overwrite the contents of the rogue symlink. I think
using mkstemp(3)/tmpfile(3) would be a safer choice if TMPDIR needs to
be in play. Consider that pg_waldump can be run as root (there's no
mechanism preventing it from being used that way).

    c. IMHO that unlink() might be not guaranteed to always remove
files, as in case of any trouble and exit() , those files might be
left over. I think we need some atexit() handlers. This can be
triggered with combo of options of nonsequential files in tar + wrong
LSN given:

    $ tar tf pg_wal.tar
    00000001000000000000005A
    00000001000000000000005B
    00000001000000000000005C
    [..]
    000000010000000000000060
    000000010000000000000059 <-- out of order, appended last
    $ ls -lh 0*
    ls: cannot access '0*': No such file or directory
    $ /usr/pgsql19/bin/pg_waldump --path=/tmp/ble/pg_wal.tar --stats
-s 0/10000358 #wrong LSN
    pg_waldump: error: could not find a valid record after 0/10000358
    $ ls -lh 0*
    -rw------- 1 postgres postgres 16M Sep  8 14:44
000000010000000000000059.waldump.tmp
    -rw------- 1 postgres postgres 16M Sep  8 14:44
00000001000000000000005A.waldump.tmp
    [..]

0006: LGTM

0007:
    a. Commit message says `Future patches to pg_waldump will enable
it to decode WAL directly` , but those pg_waldump are earlier patches,
right?

    b. pg_verifybackup should print some info with --progress that it
is spawning pg_waldump (pg_verifybackup --progress mode does not
display anything related to verifing WALs, but it could)

    c. I'm wondering, but pg_waldump seems to be not complaining if
--end=LSN is made into such a future that it doesn't exist. E.g. If
the latest WAL segment is 60 (with end LSN 0/60A77A59), but I run
pg_waldump `--end=0/7000000` , it will return code 0 and nothing on
stderr. So how sure are we that the necessary WAL segments (as per
backup_manifest) are actually inside the tar? It's supposed to be
verified, but it isn't for this use case? Same happens if craft
special tar and remove just one WAL segment from pg_wal.tar (simulate
missing WAL segment), but ask the pg_verifybackup/pg_waldump to verify
it to exact last LSN sequence, e.g.:

    $ /usr/pgsql19/bin/pg_waldump --quiet
--path=/tmp/missing/pg_wal.tar --timeline=1 --start=0/59000028
--end=0/60A77A58 && echo OK # but it is not OK
    OK
    $ /usr/pgsql19/bin/pg_waldump --stats
--path=/tmp/missing/pg_wal.tar --timeline=1 --start=0/59000028
--end=0/60A77A58
    WAL statistics between 0/59000028 and 0/5CFFFFD0: # <-- 0/5C LSN
maximum detected
    [..]

    Notice it has read till 0/5C (but I've asked till 0/60), because
I've removed 0D:
    $ tar tf /tmp/missing/pg_wal.tar| grep ^0
    000000010000000000000059
    00000001000000000000005A
    00000001000000000000005B
    00000001000000000000005C
    00000001000000000000005E <-- missing 5D

    Yet it reported no errors.

0008:
    LGTM

Another open question I have is this: shouldn't backup_manifest come
with CRC checksum for the archived WALs? Or does that guarantee that
backup_manifest WAL-Ranges are present in pg_wal.tar is good enough
because individual WAL files are CRC-protected itself?

-J.

  5. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-09-12T10:55:28Z

    On Mon, Sep 8, 2025 at 7:07 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Tue, Aug 26, 2025 at 1:53 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> [..patch]
>
> Hi Amul!
>

Thanks for your review. I'm replying to a few of your comments now,
but for the rest, I need to think about them. I'm kind of in agreement
with some of them for the fix, but I won't be able to spend time on
that next week due to official travel. I'll try to get back as soon as
possible after that.

>     a.  Why should it be necessary to provide startLSN (-s) ? Couldn't
> it autodetect the first WAL (tar file) inside and just use that with
> some info message?
>     $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar
>     pg_waldump: error: no start WAL location given
>

There are two reasons. First,  existing pg_waldump
--path=some_directory would result in the same error. Second, it would
force us to re-read the archive twice just to locate the first WAL
segment, which is inefficient.

>     c. It doesnt work when using SEGSTART, but it's there:
>     $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar
> 000000010000000000000059
>     pg_waldump: error: could not open file "000000010000000000000059":
> Not a directory
>     $ tar tf /tmp/base/pg_wal.tar | head -1
>     000000010000000000000059
>

I don't believe this is the correct use case. The WAL files are inside
a tar archive, and the requirement is to use a starting LSN and a
timeline (if not the default).

>     d.     I've later noticed that follow-up patches seem to use the
> -s switch and there it seems to work OK. The above SEGSTART issue was
> not detected, probably because tests need to be extended cover  of
> segment name rather than just --start LSN (see test_pg_waldump):
>     $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar --stats
> -s 0/59000358
>     pg_waldump: first record is after 0/59000358, at 0/590003E8,
> skipping over 144 bytes
>     WAL statistics between 0/590003E8 and 0/61000000:
>     [..]
>

Hope previous reasoning makes sense to you.

>     e. Code around`if (walpath == NULL && directory != NULL)` needs
> some comments.
>

I think this is an existing one.

>     f. Code around `if (fname != NULL && is_tar_file(fname,
> &compression))` , so if fname is WAL segment here
> (00000001000000000000005A) and we do check again if that has been
> tar-ed (is_tar_file())? Why?
>

Again, how?

>     g. Just a question: the commit message says `Note that this patch
> requires that the WAL files within the archive be in sequential order;
> an error will be reported otherwise`. I'm wondering if such
> occurrences are known to be happening in the wild? Or is it just an
> assumption that if someone would modify the tar somehow? (either way
> we could just add a reason why we need to handle such a case if we
> know -- is manual alternation the only source of such state?). For the
> record, I've tested crafting custom archives with out of sequence WAL
> archives and the code seems to work (it was done using: tar --append
> -f pg_wal.tar --format=ustar ..)
>

This is an almost nonexistent occurrence. While pg_basebackup archives
WAL files in sequential order, we don't have an explicit code to
enforce that order within it. Furthermore, since we can't control how
external tools might handle the files, this extra precaution is
necessary.


> Another open question I have is this: shouldn't backup_manifest come
> with CRC checksum for the archived WALs? Or does that guarantee that
> backup_manifest WAL-Ranges are present in pg_wal.tar is good enough
> because individual WAL files are CRC-protected itself?
>

I don't know, I have to check pg_verifybackup.

Regards,
Amul

  6. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2025-09-12T18:28:15Z

    Here are some review comments on v3-0004:

In general, I think this looks pretty nice, but I think it needs more
cleanup and polishing.

There doesn't seem to be any reason for
astreamer_waldump_content_new() to take an astreamer *next argument.
If you look at astreamer.h, you'll see that some astreamer_BLAH_new()
functions take such an argument, and others don't. The ones that do
forward their input to another astreamer; the ones that don't, like
astreamer_plain_writer_new(), send it somewhere else. AFAICT, this
astreamer is never going to send its output to another astreamer, so
there's no reason for this argument.

I'm also a little confused by the choice of the name
astreamer_waldump_content_new(). I would have thought this would be
something like astreamer_waldump_new() or astreamer_xlogreader_new().
The word "content" doesn't seem to me to be adding much here, and it
invites confusion with the "content" callback.

I think you can merge setup_astreamer() into
init_tar_archive_reader(). The only other caller is
verify_tar_archive(), but that does exactly the same additional steps
as init_tar_archive_reader(), as far as I can see.

The return statement for astreamer_wal_read is really odd:

+       return (count - nbytes) ? (count - nbytes) : -1;

Since 0 is false in C, this is equivalent to: count != nbytes ? count
- nbytes : -1, but it's a strange way to write it. What makes it even
stranger is that it seems as though the intention here is to count the
number of bytes read, but you do that by taking the number of bytes
requested (count) and subtracting the number of bytes we didn't manage
to read (nbytes); and then you just up and return -1 instead of 0
whenever the answer would have been zero. This is all lacking in
comments and seems a bit more confusing than it needs to be. So my
suggestions are:

1. Consider redefining nbytes to be the number of bytes that you have
read instead of the number of bytes you haven't read. So the loop in
this function would be while (nbytes < count) instead of while (nbytes
> 0).

2. If you need to map 0 to -1, consider having the caller do this
instead of putting that inside this function.

3. Add a comment saying what the return value is supposed to be".

If you do both 1 and 2, then the return statement can just say "return
nbytes;" and the comment can say "Returns the number of bytes
successfully read."

I would suggest changing the name of the variable from "readBuff" to
"readBuf". There are no existing uses of readBuff in the code base.

I think this comment also needs improvement:

+               /*
+                * Ignore existing data if the required target page
has not yet been
+                * read.
+                */
+               if (recptr >= endPtr)
+               {
+                       len = 0;
+
+                       /* Reset the buffer */
+                       resetStringInfo(astreamer_buf);
+               }

This comment is problematic for a few reasons. First, we're not
ignoring the existing data: we're throwing it out. Second, the comment
doesn't say why we're doing what we're doing, only that we're doing
it. Here's my guess at the actual explanation -- please correct me if
I'm wrong: "pg_waldump never reads the same WAL bytes more than once,
so if we're now being asked for data beyond the end of what we've
already read, that means none of the data we currently have in the
buffer will ever be consulted again. So, we can discard the existing
buffer contents and start over." By the way, if this explanation is
correct, it might be nice to add an assertion someplace that verifies
it, like asserting that we're always reading from an LSN greater than
or equal to (or exactly equal to?) the LSN immediately following the
last data we read.

In general, I wonder whether there's a way to make the separation of
concerns between astreamer_wal_read() and TarWALDumpReadPage()
cleaner. Right now, the latter is basically a stub, but I'm not sure
that is the best thing here. I already mentioned one example of how to
do this: make the responsibility for 0 => -1 translation the job of
TarWALDumpReadPage() rather than astreamer_wal_read(). But I think
there might be a little more we can do. In particular, I wonder
whether we could say that astreamer_wal_read() is only responsible for
filling the buffer, and the caller, TarWALDumpReadPage() in this case,
needs to empty it. That seems like it might produce a cleaner
separation of duties.

Another thing that isn't so nice right now is that
verify_tar_archive() has to open and close the archive only for
init_tar_archive_reader() to be called to reopen it again just moments
later. It would be nicer to open the file just once and then keep it
open. Here again, I wonder if the separation of duties could be a bit
cleaner.

Is there a real need to pass XLogDumpPrivate to astreamer_wal_read or
astreamer_archive_read? The only things that they need are archive_fd,
archive_name, archive_streamer, archive_streamer_buf, and
archive_streamer_read_ptr. In other words, they really don't care
about any of the *existing* things that are in XLogDumpPrivate. This
makes me wonder whether we should actually try to make this new
astreamer completely independent of xlogreader. In other words,
instead of calling it astreamer_waldump() or astreamer_xlogreader() as
I proposed above, maybe it could be a completely generic astreamer,
say astreamer_stringinfo_new(StringInfo *buf) that just appends to the
buffer. That would require also moving the stuff out of
astreamer_wal_read() that knows about XLogRecPtr, but why does that
function need to know about XLogRecPtr? Couldn't the caller figure out
that part and just tell this function how many bytes are needed?

--
Robert Haas
EDB: http://www.enterprisedb.com

  7. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2025-09-12T20:27:07Z

    On Fri, Sep 12, 2025 at 2:28 PM Robert Haas <robertmhaas@gmail.com> wrote:
> Is there a real need to pass XLogDumpPrivate to astreamer_wal_read or
> astreamer_archive_read? The only things that they need are archive_fd,
> archive_name, archive_streamer, archive_streamer_buf, and
> archive_streamer_read_ptr. In other words, they really don't care
> about any of the *existing* things that are in XLogDumpPrivate. This
> makes me wonder whether we should actually try to make this new
> astreamer completely independent of xlogreader. In other words,
> instead of calling it astreamer_waldump() or astreamer_xlogreader() as
> I proposed above, maybe it could be a completely generic astreamer,
> say astreamer_stringinfo_new(StringInfo *buf) that just appends to the
> buffer. That would require also moving the stuff out of
> astreamer_wal_read() that knows about XLogRecPtr, but why does that
> function need to know about XLogRecPtr? Couldn't the caller figure out
> that part and just tell this function how many bytes are needed?

Hmm, on further thought, I think this was a silly idea. Part of the
intended function of this astreamer is to make sure we're only reading
WAL files from the archive, and eventually reordering them if
required, so obviously something completely generic isn't going to
work. Maybe there's a way to make this look a little cleaner and
tidier but this isn't it...

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  8. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-09-25T08:18:13Z

    On Fri, Sep 12, 2025 at 4:25 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Mon, Sep 8, 2025 at 7:07 PM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> >
> > On Tue, Aug 26, 2025 at 1:53 PM Amul Sul <sulamul@gmail.com> wrote:
> > >
> > [..patch]
> >
> > Hi Amul!
> >
>
> Thanks for your review. I'm replying to a few of your comments now,
> but for the rest, I need to think about them. I'm kind of in agreement
> with some of them for the fix, but I won't be able to spend time on
> that next week due to official travel. I'll try to get back as soon as
> possible after that.
>

Reverting on rest of review comments:

> 0001: LGTM, maybe I would just slightly enhance the commit message
> ("This is in preparation for adding a second source file to this
> directory.") -- maye bit a bit more verbose or use a message from
> 0002?

Done.

>     b. Why would it like to open "blah" dir if I wanted that "blah"
> segment from the archive? Shouldn't it tell that it was looking in the
> archive and couldn find it inside?
>     $ /usr/pgsql19/bin/pg_waldump --path=/tmp/base/pg_wal.tar blah
>     pg_waldump: error: could not open file "blah": Not a directory

Now, an error will be thrown if any additional command-line
arguments are provided when an archive is specified, similar to how
existing extra arguments are handled.

>     i. If I give wrong --timeline=999 to pg_waldump it fails with
> misleading error message: could not read WAL data from "pg_wal.tar"
> archive: read -1 of 8192

Now., added a much better error message for that case.

>     a. I'm wondering if we shouldn't log (to stderr?) some kind of
> notification message (just once) that non-sequential WAL files were
> discovered and that pg_waldump is starting to write to $somewhere as
> it may be causing bigger I/O than anticipated when running the
> command. This can easily help when troubleshooting why it is not fast,
> and also having set TMPDIR to usually /tmp can be slow or too small.

Now, emitting info messages, but I'm not sure whether we should have
info or debug.

>     b. IMHO member_prepare_tmp_write() / get_tmp_wal_file_path() with
> TMPDIR can be prone to symlink attack. Consider setting TMPDIR=/tmp .
> We are writing to e.g. /tmp/<WALsegment>.waldump.tmp in 0004 , but
> that path is completely guessable. If an attacker prepares some
> symlinks and links those to some other places, I think the code will
> happily open and overwrite the contents of the rogue symlink. I think
> using mkstemp(3)/tmpfile(3) would be a safer choice if TMPDIR needs to
> be in play. Consider that pg_waldump can be run as root (there's no
> mechanism preventing it from being used that way).

I am not sure what the worst-case scenario would be or what a good
alternative is.

>     c. IMHO that unlink() might be not guaranteed to always remove
> files, as in case of any trouble and exit() , those files might be
> left over. I think we need some atexit() handlers. This can be
> triggered with combo of options of nonsequential files in tar + wrong
> LSN given:

Done.

> 0007:
>     a. Commit message says `Future patches to pg_waldump will enable
> it to decode WAL directly` , but those pg_waldump are earlier patches,
> right?

Right, fixed.

>     b. pg_verifybackup should print some info with --progress that it
> is spawning pg_waldump (pg_verifybackup --progress mode does not
> display anything related to verifing WALs, but it could)

If we decide to do that, it could be a separate project, IMHO.

>     c. I'm wondering, but pg_waldump seems to be not complaining if
> --end=LSN is made into such a future that it doesn't exist.

The behavior will be kept as if a directory was provided with a start
and end LSN.

Thanks again for the review. I'll post the new patches in my next reply.

Regards,
Amul

  9. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-09-25T08:24:51Z

    On Fri, Sep 12, 2025 at 11:58 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> Here are some review comments on v3-0004:
>

Thanks for the review. My replies are below.

> There doesn't seem to be any reason for
> astreamer_waldump_content_new() to take an astreamer *next argument.
> If you look at astreamer.h, you'll see that some astreamer_BLAH_new()
> functions take such an argument, and others don't. The ones that do
> forward their input to another astreamer; the ones that don't, like
> astreamer_plain_writer_new(), send it somewhere else. AFAICT, this
> astreamer is never going to send its output to another astreamer, so
> there's no reason for this argument.
>

Done.

> I'm also a little confused by the choice of the name
> astreamer_waldump_content_new(). I would have thought this would be
> something like astreamer_waldump_new() or astreamer_xlogreader_new().
> The word "content" doesn't seem to me to be adding much here, and it
> invites confusion with the "content" callback.
>

Done -- renamed to astreamer_waldump_new().

> I think you can merge setup_astreamer() into
> init_tar_archive_reader(). The only other caller is
> verify_tar_archive(), but that does exactly the same additional steps
> as init_tar_archive_reader(), as far as I can see.
>

Done.

> The return statement for astreamer_wal_read is really odd:
>
> +       return (count - nbytes) ? (count - nbytes) : -1;
>

Agreed, that's a bit odd. This seems to be leftover code from the experimental
patch. The astreamer_wal_read() function should behave like WALRead():
it should either successfully read all the requested bytes or throw an
error. Corrected in the attached version.

>
> I would suggest changing the name of the variable from "readBuff" to
> "readBuf". There are no existing uses of readBuff in the code base.
>

The existing WALDumpReadPage() function has a "readBuff" argument, and
I've used it that way for consistency.

> I think this comment also needs improvement:
>
> +               /*
> +                * Ignore existing data if the required target page
> has not yet been
> +                * read.
> +                */
> +               if (recptr >= endPtr)
> +               {
> +                       len = 0;
> +
> +                       /* Reset the buffer */
> +                       resetStringInfo(astreamer_buf);
> +               }
>
> This comment is problematic for a few reasons. First, we're not
> ignoring the existing data: we're throwing it out. Second, the comment
> doesn't say why we're doing what we're doing, only that we're doing
> it. Here's my guess at the actual explanation -- please correct me if
> I'm wrong: "pg_waldump never reads the same WAL bytes more than once,
> so if we're now being asked for data beyond the end of what we've
> already read, that means none of the data we currently have in the
> buffer will ever be consulted again. So, we can discard the existing
> buffer contents and start over." By the way, if this explanation is
> correct, it might be nice to add an assertion someplace that verifies
> it, like asserting that we're always reading from an LSN greater than
> or equal to (or exactly equal to?) the LSN immediately following the
> last data we read.
>

Updated the comment. The similar assertion exists right before
copying to the readBuff.

>
> Another thing that isn't so nice right now is that
> verify_tar_archive() has to open and close the archive only for
> init_tar_archive_reader() to be called to reopen it again just moments
> later. It would be nicer to open the file just once and then keep it
> open. Here again, I wonder if the separation of duties could be a bit
> cleaner.
>

Prefer to keep those separate, assuming that reopening the file won't
cause any significant harm. Let me know if you think otherwise.

Attached the updated version, kindly have a look.

Regards,
Amul

  10. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2025-09-29T15:15:31Z

    On Thu, Sep 25, 2025 at 4:25 AM Amul Sul <sulamul@gmail.com> wrote:
> > Another thing that isn't so nice right now is that
> > verify_tar_archive() has to open and close the archive only for
> > init_tar_archive_reader() to be called to reopen it again just moments
> > later. It would be nicer to open the file just once and then keep it
> > open. Here again, I wonder if the separation of duties could be a bit
> > cleaner.
>
> Prefer to keep those separate, assuming that reopening the file won't
> cause any significant harm. Let me know if you think otherwise.

Well, I guess I'd like to know why we can't do better. I'm not really
worried about performance, but reopening the file means that you can
never make it work with reading from a pipe.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  11. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-09-29T16:17:10Z

    On Mon, Sep 29, 2025 at 8:45 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Sep 25, 2025 at 4:25 AM Amul Sul <sulamul@gmail.com> wrote:
> > > Another thing that isn't so nice right now is that
> > > verify_tar_archive() has to open and close the archive only for
> > > init_tar_archive_reader() to be called to reopen it again just moments
> > > later. It would be nicer to open the file just once and then keep it
> > > open. Here again, I wonder if the separation of duties could be a bit
> > > cleaner.
> >
> > Prefer to keep those separate, assuming that reopening the file won't
> > cause any significant harm. Let me know if you think otherwise.
>
> Well, I guess I'd like to know why we can't do better. I'm not really
> worried about performance, but reopening the file means that you can
> never make it work with reading from a pipe.

I have some skepticism regarding the extra coding that might be
introduced, as performance is not my primary concern here. If we aim
to keep the file open only once, that logic should be implemented
before calling verify_tar_archive(), not inside it. Implementing the
open and close logic within verify_tar_archive() and
free_tar_archive_reader() would create a confusing and scattered
pattern, especially since these separate operations require only two
lines of code each (open and close if it's a tar file). My second,
concern is that after verify_tar_archive(), we might need to reset the
file reader offset to the beginning. While reusing the buffered data
from the first iteration is technically possible, that only works if
the desired start LSN is at the absolute beginning of the archive, or
later in the sequence, which cannot be reliably guaranteed. Therefore,
for simplicity and avoid the complexity of managing that offset reset
code, I am thinking of a simpler approach.

Regards,
Amul

  12. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2025-10-10T18:01:59Z

    On Mon, Sep 29, 2025 at 12:17 PM Amul Sul <sulamul@gmail.com> wrote:
> While reusing the buffered data
> from the first iteration is technically possible, that only works if
> the desired start LSN is at the absolute beginning of the archive, or
> later in the sequence, which cannot be reliably guaranteed.

I spent a bunch of time studying this code today and I think that the
problem you're talking about here is evidence of a design problem with
astreamer_wal_read() and some of the other code in
astreamer_waldump.c. Your code calls astreamer_wal_read() when it
wants to peek at the first xlog block to determine the WAL segment
size, and it also calls astreamer_wal_read() when it wants read WAL
sequentially beginning at the start LSN and continuing until it
reaches the end LSN. However, these two cases have very different
requirements. verify_tar_archive(), which is misleadingly named and
really exists to determine the WAL segment size, just wants to read
the first xlog block that physically appears in the archive. Every
xlog block will have the same WAL segment size, so it does not matter
which one we read. On the other hand, TarWALDumpReadPage wants to read
WAL in sequential order. In other words, one call to
astreamer_wal_read() really wants to read a block without any block
reordering, and the other call wants to read a block with block
reordering.

To me, it looks like the problem here is that the block reordering
functionality should live on top of the astreamer, not inside of it.
Imagine that astreamer just spits out the bytes in the order in which
they physically appear in the archive, and then there's another
component that consumes and reorders those bytes. So, you read data
and push it into the astreamer until the number of bytes in the output
buffer is at least XLOG_BLCKSZ, and then from there you extract the
WAL segment size. Then, you call XLogReaderAllocate() and enter the
main loop. The reordering logic lives inside of TarWALDumpReadPage().
Each time it gets data from the astreamer's buffer, it either returns
it to the caller if it's in order or buffers it using temporary files
if not.

I found it's actually quite easy to write a patch that avoids
reopening the file. Here it is, on top of your v4:

diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
index 2c42df46d43..c4346a5e211 100644
--- a/src/bin/pg_waldump/pg_waldump.c
+++ b/src/bin/pg_waldump/pg_waldump.c
@@ -368,17 +368,8 @@ init_tar_archive_reader(XLogDumpPrivate *private,
const char *waldir,
                         XLogRecPtr startptr, XLogRecPtr endptr,
                         pg_compress_algorithm compression)
 {
-    int         fd;
     astreamer  *streamer;

-    /* Open tar archive and store its file descriptor */
-    fd = open_file_in_directory(waldir, private->archive_name);
-
-    if (fd < 0)
-        pg_fatal("could not open file \"%s\"", private->archive_name);
-
-    private->archive_fd = fd;
-
     /*
      * Create an appropriate chain of archive streamers for reading the given
      * tar archive.
@@ -1416,12 +1407,22 @@ main(int argc, char **argv)
     /* we have everything we need, start reading */
     if (is_tar)
     {
+        /* Open tar archive and store its file descriptor */
+        private.archive_fd =
+            open_file_in_directory(waldir, private.archive_name);
+        if (private.archive_fd < 0)
+            pg_fatal("could not open file \"%s\"", private.archive_name);
+
         /* Verify that the archive contains valid WAL files */
         waldir = waldir ? pg_strdup(waldir) : pg_strdup(".");
         init_tar_archive_reader(&private, waldir, InvalidXLogRecPtr,
                                 InvalidXLogRecPtr, compression);
         verify_tar_archive(&private);
-        free_tar_archive_reader(&private);
+        astreamer_free(private.archive_streamer);
+
+        if (lseek(private.archive_fd, 0, SEEK_SET) != 0)
+            pg_log_error("could not seek in file \"%s\": %m",
+                         private.archive_name);

         /* Set up for reading tar file */
         init_tar_archive_reader(&private, waldir, private.startptr,

Of course, this is not really what we want to do: it avoids reopening
the file, but because we can't back up the archive streamer once it's
been created, we have to lseek back to the beginning of the file. But
notice how silly this looks: with this patch, we free the archive
reader and immediately create a new archive reader that is exactly the
same in every way except that we call astreamer_waldump_new(startptr,
endptr, private) instead of astreamer_waldump_new(InvalidXLogRecPtr,
InvalidXLogRecPtr, private). We could arrange to update the original
archive streamer with new values of startSegNo and endSegNo after
verify_tar_archive(), but that's still not quite good enough, because
we might have already made some decisions on what to do with the data
that we read that it's too late to reverse. But, what that means is
that the astreamer_waldump machinery is not smart enough to read one
block of data without making irreversible decisions from which we
can't recover without recreating the entire object. I think we can,
and should, try to do better.

It's also worth noting that the unfortunate layering doesn't just
require us to read the first block of the file: it also complicates
the code in various places. The fact that astreamer_wal_read() needs a
special case for XLogRecPtrIsInvalid(recptr) is a direct result of
this problem, and the READ_ANY_WAL() macro and both the places that
test it are also direct results of this problem. In other words, I'm
arguing that astreamer_wal_read() is incorrectly defined, and that
error creates ugliness in the code both above and below
astreamer_wal_read().

While I'm on the topic of astreamer_wal_read(), here are a few other
problems I noticed:

* The return value is not documented, and it seems to always be count,
in which case it might as well return void. The caller already has the
value they passed for count.
* It seems like it would be more appropriate to assert that endPtr >=
len and just set startPtr = endPtr - len. I don't see how len > endPtr
can ever happen, and I bet bad things will happen if it does.
* "pg_waldump never ask the same" -> "pg_waldump never asks for the same"

Also, this is absolutely not OK with me:

                        /* Fetch more data */
                        if (astreamer_archive_read(privateInfo) == 0)
                        {
                                char            fname[MAXFNAMELEN];
                                XLogSegNo       segno;

                                XLByteToSeg(targetPagePtr, segno, WalSegSz); an
                                XLogFileName(fname,
privateInfo->timeline, segno, WalSegSz);

                                pg_fatal("could not find file \"%s\"
in \"%s\" archive",
                                                 fname,
privateInfo->archive_name);
                        }

astreamer_archive_read() will return 0 if we reach the end of the
tarfile, so this is saying that if we reach the end of the tar file
without finding the range of bytes for which we're looking, the
explanation must be that the relevant WAL file is missing from the
archive. But that is way too much action at a distance. I was able to
easily construct a counterexample by copying the first 81920 bytes of
a valid WAL file and then doing this:

[robert.haas pgsql-meson]$ tar tf pg_wal.tar
000000010000000000000005
[robert.haas pgsql-meson]$ pg_waldump -s 0/050008D8 -e 0/05FFED98
pg_wal.tar >/dev/null
pg_waldump: error: could not find file "000000010000000000000005" in
"pg_wal.tar" archive

Without the redirection to /dev/null, what happened was that
pg_waldump printed out a bunch of records from
000000010000000000000005 and then said that 000000010000000000000005
could not be found, which is obviously silly. But the fact that I
found a specific counterexample here isn't even really the point. The
point is that there's a big gap between what we actually know at this
point (which is that we've read the whole input file) and what the
message is claiming (which is that the reason must be that the file is
missing from the archive). Even if the counterexample above didn't
exist and that really were the only way for that to happen as of
today, that's very fragile. Maybe some future code change will make it
so that there's a second reason that could happen. How would somebody
realize that they had created a second condition by means of which
this code could be reached? If they did realize it, how would they get
the correct error to be reported?

I'm not quite sure how this should be fixed, but I strongly suspect
that the error report here needs to move closer to the code that is
doing the file reordering. Aside from the possibility of the file
being missing and the possibility of the file being too short, a third
possibility is that targetPagePtr retreats between one call and the
next. That really shouldn't happen, but there are no asserts here
verifying that it doesn't.

I also don't like the fact that one call to astreamer_archive_read()
checks the return value (but only whether it's zero, the specific
return value apparently doesn't matter, so why doesn't it return
bool?) and the other doesn't. That kind of coding pattern is very
rarely correct. The code says:

    /* Continue reading from the open WAL segment, if any */
    if (state->seg.ws_file >= 0)
    {
        /*
         * To prevent a race condition where the archive streamer is still
         * exporting a file that we are trying to read, we invoke the streamer
         * to ensure enough data is available.
         */
        if (private->curSegNo == state->seg.ws_segno)
            astreamer_archive_read(private);

        return WALDumpReadPage(state, targetPagePtr, reqLen, targetPtr,
                               readBuff);
    }

But it's unclear why this should be good enough to ensure that enough
data is available. astreamer_archive_read() might read zero bytes and
return 0, so this doesn't really guarantee anything at all. On the
other hand, even if astereamer_archive_read() returns a non-zero
value, it's only going to read READ_CHUNK_SIZE bytes from the
underlying file, so if more than that needs to be read in order for us
to have enough data, we won't. I think it's very hard to imagine a
situation in which you can call astreamer_archive_read() without using
some loop. That's what astreamer_wal_read() does: it calls
astreamer_archive_read() until it either returns 0 -- in which case we
know we've failed -- or until we have enough data. Here we just hope
that calling it once is enough, and that checking for errors is
unimportant. I also don't understand the reference to a race
condition, because there's only one process with one thread here, I
believe, so what would be racing against?

Another thing I noticed is that astreamer_archive_read() makes
reference to decrypting, but there's no cryptography involved in any
of this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  13. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-10-16T11:48:31Z

    On Fri, Oct 10, 2025 at 11:32 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Sep 29, 2025 at 12:17 PM Amul Sul <sulamul@gmail.com> wrote:
> > While reusing the buffered data
> > from the first iteration is technically possible, that only works if
> > the desired start LSN is at the absolute beginning of the archive, or
> > later in the sequence, which cannot be reliably guaranteed.
>
> I spent a bunch of time studying this code today and I think that the
> problem you're talking about here is evidence of a design problem with
> astreamer_wal_read() and some of the other code in
> astreamer_waldump.c. Your code calls astreamer_wal_read() when it
> wants to peek at the first xlog block to determine the WAL segment
> size, and it also calls astreamer_wal_read() when it wants read WAL
> sequentially beginning at the start LSN and continuing until it
> reaches the end LSN. However, these two cases have very different
> requirements. verify_tar_archive(), which is misleadingly named and
> really exists to determine the WAL segment size, just wants to read
> the first xlog block that physically appears in the archive. Every
> xlog block will have the same WAL segment size, so it does not matter
> which one we read. On the other hand, TarWALDumpReadPage wants to read
> WAL in sequential order. In other words, one call to
> astreamer_wal_read() really wants to read a block without any block
> reordering, and the other call wants to read a block with block
> reordering.
>
> To me, it looks like the problem here is that the block reordering
> functionality should live on top of the astreamer, not inside of it.
> Imagine that astreamer just spits out the bytes in the order in which
> they physically appear in the archive, and then there's another
> component that consumes and reorders those bytes. So, you read data
> and push it into the astreamer until the number of bytes in the output
> buffer is at least XLOG_BLCKSZ, and then from there you extract the
> WAL segment size. Then, you call XLogReaderAllocate() and enter the
> main loop. The reordering logic lives inside of TarWALDumpReadPage().
> Each time it gets data from the astreamer's buffer, it either returns
> it to the caller if it's in order or buffers it using temporary files
> if not.
>

I initially considered implementing the reordering logic outside of
astreamer when we first discussed this project, but the implementation
could get complicated -- or at least feel hacky. Let me explain why:

astreamer reads the archive in fixed-size chunks (here it is 128KB).
Sometimes, a single read can contain data from two WAL files --
specifically, the tail end of one file and the start of the next --
because of how they’re physically stored in the archive. astreamer
knows where one file ends and another begins through tags like
ASTREAMER_MEMBER_HEADER, ASTREAMER_MEMBER_CONTENTS, and
ASTREAMER_MEMBER_TRAILER. However, it can’t pause mid-chunk to hold
data from the next file once the previous one ends and for the caller;
it pushes the entire chunk it has read to the target buffer.

So, if we put the reordering logic outside the streamer, we’d
sometimes be receiving buffers containing mixed data from two WAL
files. The caller would then need to correctly identify WAL file
boundaries within those buffers. This would require passing extra
metadata -- like segment numbers for the WAL files in the buffer, plus
start and end offsets of those segments within the buffer. While not
impossible, it feels a bit hacky and I'm unsure if that’s the best
approach.

> I found it's actually quite easy to write a patch that avoids
> reopening the file. Here it is, on top of your v4:
>
> diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c
> index 2c42df46d43..c4346a5e211 100644
> --- a/src/bin/pg_waldump/pg_waldump.c
> +++ b/src/bin/pg_waldump/pg_waldump.c
> @@ -368,17 +368,8 @@ init_tar_archive_reader(XLogDumpPrivate *private,
> const char *waldir,
>                          XLogRecPtr startptr, XLogRecPtr endptr,
>                          pg_compress_algorithm compression)
>  {
> -    int         fd;
>      astreamer  *streamer;
>
> -    /* Open tar archive and store its file descriptor */
> -    fd = open_file_in_directory(waldir, private->archive_name);
> -
> -    if (fd < 0)
> -        pg_fatal("could not open file \"%s\"", private->archive_name);
> -
> -    private->archive_fd = fd;
> -
>      /*
>       * Create an appropriate chain of archive streamers for reading the given
>       * tar archive.
> @@ -1416,12 +1407,22 @@ main(int argc, char **argv)
>      /* we have everything we need, start reading */
>      if (is_tar)
>      {
> +        /* Open tar archive and store its file descriptor */
> +        private.archive_fd =
> +            open_file_in_directory(waldir, private.archive_name);
> +        if (private.archive_fd < 0)
> +            pg_fatal("could not open file \"%s\"", private.archive_name);
> +
>          /* Verify that the archive contains valid WAL files */
>          waldir = waldir ? pg_strdup(waldir) : pg_strdup(".");
>          init_tar_archive_reader(&private, waldir, InvalidXLogRecPtr,
>                                  InvalidXLogRecPtr, compression);
>          verify_tar_archive(&private);
> -        free_tar_archive_reader(&private);
> +        astreamer_free(private.archive_streamer);
> +
> +        if (lseek(private.archive_fd, 0, SEEK_SET) != 0)
> +            pg_log_error("could not seek in file \"%s\": %m",
> +                         private.archive_name);
>
>          /* Set up for reading tar file */
>          init_tar_archive_reader(&private, waldir, private.startptr,
>
> Of course, this is not really what we want to do: it avoids reopening
> the file, but because we can't back up the archive streamer once it's
> been created, we have to lseek back to the beginning of the file. But
> notice how silly this looks: with this patch, we free the archive
> reader and immediately create a new archive reader that is exactly the
> same in every way except that we call astreamer_waldump_new(startptr,
> endptr, private) instead of astreamer_waldump_new(InvalidXLogRecPtr,
> InvalidXLogRecPtr, private). We could arrange to update the original
> archive streamer with new values of startSegNo and endSegNo after
> verify_tar_archive(), but that's still not quite good enough, because
> we might have already made some decisions on what to do with the data
> that we read that it's too late to reverse. But, what that means is
> that the astreamer_waldump machinery is not smart enough to read one
> block of data without making irreversible decisions from which we
> can't recover without recreating the entire object. I think we can,
> and should, try to do better.
>

Agreed.

> It's also worth noting that the unfortunate layering doesn't just
> require us to read the first block of the file: it also complicates
> the code in various places. The fact that astreamer_wal_read() needs a
> special case for XLogRecPtrIsInvalid(recptr) is a direct result of
> this problem, and the READ_ANY_WAL() macro and both the places that
> test it are also direct results of this problem. In other words, I'm
> arguing that astreamer_wal_read() is incorrectly defined, and that
> error creates ugliness in the code both above and below
> astreamer_wal_read().
>
> While I'm on the topic of astreamer_wal_read(), here are a few other
> problems I noticed:
>
> * The return value is not documented, and it seems to always be count,
> in which case it might as well return void. The caller already has the
> value they passed for count.

The caller will be xlogreader, and I believe we shouldn't change that.
For the same reason, WALDumpReadPage() also returns the same.

> * It seems like it would be more appropriate to assert that endPtr >=
> len and just set startPtr = endPtr - len. I don't see how len > endPtr
> can ever happen, and I bet bad things will happen if it does.
> * "pg_waldump never ask the same" -> "pg_waldump never asks for the same"
>

Ok.

> Also, this is absolutely not OK with me:
>
>                         /* Fetch more data */
>                         if (astreamer_archive_read(privateInfo) == 0)
>                         {
>                                 char            fname[MAXFNAMELEN];
>                                 XLogSegNo       segno;
>
>                                 XLByteToSeg(targetPagePtr, segno, WalSegSz); an
>                                 XLogFileName(fname,
> privateInfo->timeline, segno, WalSegSz);
>
>                                 pg_fatal("could not find file \"%s\"
> in \"%s\" archive",
>                                                  fname,
> privateInfo->archive_name);
>                         }
>
> astreamer_archive_read() will return 0 if we reach the end of the
> tarfile, so this is saying that if we reach the end of the tar file
> without finding the range of bytes for which we're looking, the
> explanation must be that the relevant WAL file is missing from the
> archive. But that is way too much action at a distance. I was able to
> easily construct a counterexample by copying the first 81920 bytes of
> a valid WAL file and then doing this:
>
> [robert.haas pgsql-meson]$ tar tf pg_wal.tar
> 000000010000000000000005
> [robert.haas pgsql-meson]$ pg_waldump -s 0/050008D8 -e 0/05FFED98
> pg_wal.tar >/dev/null
> pg_waldump: error: could not find file "000000010000000000000005" in
> "pg_wal.tar" archive
>
> Without the redirection to /dev/null, what happened was that
> pg_waldump printed out a bunch of records from
> 000000010000000000000005 and then said that 000000010000000000000005
> could not be found, which is obviously silly. But the fact that I
> found a specific counterexample here isn't even really the point. The
> point is that there's a big gap between what we actually know at this
> point (which is that we've read the whole input file) and what the
> message is claiming (which is that the reason must be that the file is
> missing from the archive). Even if the counterexample above didn't
> exist and that really were the only way for that to happen as of
> today, that's very fragile. Maybe some future code change will make it
> so that there's a second reason that could happen. How would somebody
> realize that they had created a second condition by means of which
> this code could be reached? If they did realize it, how would they get
> the correct error to be reported?
>

Agreed, I'll think about this.

>
>     /* Continue reading from the open WAL segment, if any */
>     if (state->seg.ws_file >= 0)
>     {
>         /*
>          * To prevent a race condition where the archive streamer is still
>          * exporting a file that we are trying to read, we invoke the streamer
>          * to ensure enough data is available.
>          */
>         if (private->curSegNo == state->seg.ws_segno)
>             astreamer_archive_read(private);
>
>         return WALDumpReadPage(state, targetPagePtr, reqLen, targetPtr,
>                                readBuff);
>     }
>
> But it's unclear why this should be good enough to ensure that enough
> data is available. astreamer_archive_read() might read zero bytes and
> return 0, so this doesn't really guarantee anything at all. On the
> other hand, even if astereamer_archive_read() returns a non-zero
> value, it's only going to read READ_CHUNK_SIZE bytes from the
> underlying file, so if more than that needs to be read in order for us
> to have enough data, we won't. I think it's very hard to imagine a
> situation in which you can call astreamer_archive_read() without using
> some loop.

The loop isn't needed because the caller always requests 8KB of data,
while READ_CHUNK_SIZE is 128KB. It’s assumed that the astreamer has
already created the file with some initial data. For example, if only
a few bytes have been written so far, when we reach
TarWALDumpReadPage(), it detects that we’re reading the same file
that the astreamer is still writing to and hasn’t finished. It then request to
appends 128KB of data by calling astreamer_archive_read, even though we
only need 8KB at a time. This process repeats each time the next 8KBchunk is
requested: astreamer_archive_read() appends another 128KB,and continues until
the file has been fully read and written.

> That's what astreamer_wal_read() does: it calls
> astreamer_archive_read() until it either returns 0 -- in which case we
> know we've failed -- or until we have enough data. Here we just hope
> that calling it once is enough, and that checking for errors is
> unimportant. I also don't understand the reference to a race
> condition, because there's only one process with one thread here, I
> believe, so what would be racing against?
>

In the case where the astreamer is exporting a file to disk but hasn’t
finished writing it, and we call TarWALDumpReadPage() to request
block(s) from that WAL file, we can read only up to the existing
blocks in the file. Since the file is incomplete, reading may fail
later. To handle this, astreamer_archive_read() is invoked to append
more data -- usually more than the requested amount, as explained
earlier. That is the race condition I am trying to handle.

Now, regarding the concern of astreamer_archive_read() returning zero
without reading or appending any data: this can happen only if the WAL
is shorter than expected -- an incomplete. In that case,
WALDumpReadPage() will raise the appropriate error, we don't have to
check at that point, I think.

> Another thing I noticed is that astreamer_archive_read() makes
> reference to decrypting, but there's no cryptography involved in any
> of this.
>

I think that was a typo -- I meant decompression.

Regards,
Amul

  14. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2025-10-20T14:34:50Z

    On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
> astreamer reads the archive in fixed-size chunks (here it is 128KB).
> Sometimes, a single read can contain data from two WAL files --
> specifically, the tail end of one file and the start of the next --
> because of how they’re physically stored in the archive. astreamer
> knows where one file ends and another begins through tags like
> ASTREAMER_MEMBER_HEADER, ASTREAMER_MEMBER_CONTENTS, and
> ASTREAMER_MEMBER_TRAILER. However, it can’t pause mid-chunk to hold
> data from the next file once the previous one ends and for the caller;
> it pushes the entire chunk it has read to the target buffer.

Right, this makes sense.

> So, if we put the reordering logic outside the streamer, we’d
> sometimes be receiving buffers containing mixed data from two WAL
> files. The caller would then need to correctly identify WAL file
> boundaries within those buffers. This would require passing extra
> metadata -- like segment numbers for the WAL files in the buffer, plus
> start and end offsets of those segments within the buffer. While not
> impossible, it feels a bit hacky and I'm unsure if that’s the best
> approach.

I agree that we need that kind of metadata, but I don't see why our
need for it depends on where we do the reordering. That is, if we do
the reordering above the astreamer layer, we need to keep track of the
origin of each chunk of WAL bytes, and if we do the reordering within
the astreamer layer, we still need to keep track of the origin of the
WAL bytes. Doing the ordering properly requires that tracking, but it
doesn't say anything about where that tracking has to be performed.

I think it might be better if we didn't write to the astreamer's
buffer at all. For example, suppose we create a struct that looks
approximately like this:

struct ChunkOfDecodedWAL
{
     XLogSegNo segno; // could also be XLogRecPtr start_lsn or char
*walfilename or whatever
     StringInfoData buffer;
     char *spillfilename; // or whatever we use to identify the temporary files
     bool already_removed;
     // potentially other metadata
};

Then, create a hash table and key it on the segno whatever. Have the
astreamer write to the hash table: when it gets a chunk of WAL, it
looks up or creates the relevant hash table entry and appends the data
to the buffer. At any convenient point in the code, you can decide to
write the data from the buffer to a spill file, after which you
resetStringInfo() on the buffer and populate the spill file name. When
you've used up the data, you remove the spill file and set the
already_removed flag.

I think this could also help with the error reporting stuff. When you
get to the end of the file, you'll know all the files you saw and how
much data you read from each of them. So you could possibly do
something like

ERROR: LSN %08X/%08X not found in archive "\%s\"
DETAIL: WAL segment %s is not present in the archive
-or
DETAIL: WAL segment %s was expected to be %u bytes, but was only %u bytes
-or-
DETAIL: whatever else can go wrong

The point is that every file you've ever seen has a hash table entry,
and in that hash table entry you can store everything about that file
that you need to know, whether that's the file data, the disk file
that contains the file data, the fact that we already threw the data
away, or any other fact that you can imagine wanting to know.

Said differently, the astreamer buffer is not really a great place to
write data. It exists because when we're just forwarding data from one
astreamer to the next, we will often need to buffer a small amount of
data to avoid terrible performance. However, it's only there to be
used when we don't have something better. I don't think any astreamer
that is intended to be the last one in the chain currently writes to
the buffer -- they write to the output file, or whatever, because
using an in-memory buffer as your final output destination is not a
real good plan.

> > While I'm on the topic of astreamer_wal_read(), here are a few other
> > problems I noticed:
> >
> > * The return value is not documented, and it seems to always be count,
> > in which case it might as well return void. The caller already has the
> > value they passed for count.
>
> The caller will be xlogreader, and I believe we shouldn't change that.
> For the same reason, WALDumpReadPage() also returns the same.

OK, but then you can make that clear via a brief comment.

> The loop isn't needed because the caller always requests 8KB of data,
> while READ_CHUNK_SIZE is 128KB. It’s assumed that the astreamer has
> already created the file with some initial data. For example, if only
> a few bytes have been written so far, when we reach
> TarWALDumpReadPage(), it detects that we’re reading the same file
> that the astreamer is still writing to and hasn’t finished. It then request to
> appends 128KB of data by calling astreamer_archive_read, even though we
> only need 8KB at a time. This process repeats each time the next 8KBchunk is
> requested: astreamer_archive_read() appends another 128KB,and continues until
> the file has been fully read and written.

Sure, but you don't know how much data is going to come out the other
end of the astreamer pipeline. Since the data is (possibly)
compressed, you expect at least as many bytes to emerge from the
output end as you add to the input end, but it's not a good idea to
rely on assumptions like that. Sometimes compressors end up making the
data slightly larger instead of smaller. It's unlikely that the effect
would be so dramatic that adding 128kB to one end of the pipeline
would make less than 8kB emerge from the other end, but it's not a
good idea to rely on assumptions like that. Not that this is a real
thing, but imagine that the compressed file had something in the
middle of it that behaved like a comment in C code, i.e. it didn't
generate any output.

> In the case where the astreamer is exporting a file to disk but hasn’t
> finished writing it, and we call TarWALDumpReadPage() to request
> block(s) from that WAL file, we can read only up to the existing
> blocks in the file. Since the file is incomplete, reading may fail
> later. To handle this, astreamer_archive_read() is invoked to append
> more data -- usually more than the requested amount, as explained
> earlier. That is the race condition I am trying to handle.

That's not what a race condition is:
https://en.wikipedia.org/wiki/Race_condition

> Now, regarding the concern of astreamer_archive_read() returning zero
> without reading or appending any data: this can happen only if the WAL
> is shorter than expected -- an incomplete. In that case,
> WALDumpReadPage() will raise the appropriate error, we don't have to
> check at that point, I think.

I'm not going to accept that kind of justification -- it is too
fragile to assume that you don't need to check for an error because it
"can't happen". Sometimes that is reasonable, but there is quite a lot
of action-at-a-distance here, so it does not feel safe.

> > Another thing I noticed is that astreamer_archive_read() makes
> > reference to decrypting, but there's no cryptography involved in any
> > of this.
>
> I think that was a typo -- I meant decompression.

I figured as much, but it still needs fixing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  15. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-11-06T09:03:05Z

    On Mon, Oct 20, 2025 at 8:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
>
> > So, if we put the reordering logic outside the streamer, we’d
> > sometimes be receiving buffers containing mixed data from two WAL
> > files. The caller would then need to correctly identify WAL file
> > boundaries within those buffers. This would require passing extra
> > metadata -- like segment numbers for the WAL files in the buffer, plus
> > start and end offsets of those segments within the buffer. While not
> > impossible, it feels a bit hacky and I'm unsure if that’s the best
> > approach.
>
> I agree that we need that kind of metadata, but I don't see why our
> need for it depends on where we do the reordering. That is, if we do
> the reordering above the astreamer layer, we need to keep track of the
> origin of each chunk of WAL bytes, and if we do the reordering within
> the astreamer layer, we still need to keep track of the origin of the
> WAL bytes. Doing the ordering properly requires that tracking, but it
> doesn't say anything about where that tracking has to be performed.
>
> I think it might be better if we didn't write to the astreamer's
> buffer at all. For example, suppose we create a struct that looks
> approximately like this:
>
> struct ChunkOfDecodedWAL
> {
>      XLogSegNo segno; // could also be XLogRecPtr start_lsn or char
> *walfilename or whatever
>      StringInfoData buffer;
>      char *spillfilename; // or whatever we use to identify the temporary files
>      bool already_removed;
>      // potentially other metadata
> };
>
> Then, create a hash table and key it on the segno whatever. Have the
> astreamer write to the hash table: when it gets a chunk of WAL, it
> looks up or creates the relevant hash table entry and appends the data
> to the buffer. At any convenient point in the code, you can decide to
> write the data from the buffer to a spill file, after which you
> resetStringInfo() on the buffer and populate the spill file name. When
> you've used up the data, you remove the spill file and set the
> already_removed flag.
>
> I think this could also help with the error reporting stuff. When you
> get to the end of the file, you'll know all the files you saw and how
> much data you read from each of them. So you could possibly do
> something like
>
> ERROR: LSN %08X/%08X not found in archive "\%s\"
> DETAIL: WAL segment %s is not present in the archive
> -or
> DETAIL: WAL segment %s was expected to be %u bytes, but was only %u bytes
> -or-
> DETAIL: whatever else can go wrong
>
> The point is that every file you've ever seen has a hash table entry,
> and in that hash table entry you can store everything about that file
> that you need to know, whether that's the file data, the disk file
> that contains the file data, the fact that we already threw the data
> away, or any other fact that you can imagine wanting to know.
>
> Said differently, the astreamer buffer is not really a great place to
> write data. It exists because when we're just forwarding data from one
> astreamer to the next, we will often need to buffer a small amount of
> data to avoid terrible performance. However, it's only there to be
> used when we don't have something better. I don't think any astreamer
> that is intended to be the last one in the chain currently writes to
> the buffer -- they write to the output file, or whatever, because
> using an in-memory buffer as your final output destination is not a
> real good plan.
>

Make sense, I implemented this approach in the attached version, but
with a different structure name and a slightly different error
message. In the error output using the WAL file name instead of the
LSN. This is because the LSN at that point may differ from the
user-provided one (it might have been adjusted to the start of a WAL
page by xlogreader). This follows the same style used in the routine
that reads the WAL file. The LSN values (user provided) are only used
in error messages generated at the very beginning, specifically in the
main() function of pg_waldump.

I have also restructured the code by moving most of the tar file
reading logic out of pg_waldump.c into astreamer_waldump.c, which has
now been renamed to archive_waldump.c.

Kindly have a look at the attached version. Thank you !

Regards,
Amul

  16. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-11-17T04:50:48Z

    On Thu, Nov 6, 2025 at 2:33 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Mon, Oct 20, 2025 at 8:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
> > [....]
> Kindly have a look at the attached version. Thank you !
>

Attached is the rebased version against the latest master head (e76defbcf09).

Regards,
Amul

  17. Re: pg_waldump: support decoding of WAL inside tarfile

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-11-19T08:20:14Z

    On Mon, Nov 17, 2025 at 5:51 AM Amul Sul <sulamul@gmail.com> wrote:
>
> On Thu, Nov 6, 2025 at 2:33 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Mon, Oct 20, 2025 at 8:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > >
> > > On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
> > > [....]
> > Kindly have a look at the attached version. Thank you !
> >
>
> Attached is the rebased version against the latest master head (e76defbcf09).

Hi Amul, thanks for working on this. I haven't really looked at the
source code deeply (I trust Robert eyes much more than mine on this
one), just skimmed a little bit:

1. As stated earlier, get_tmp_walseg_path() is still vulnerable (it
uses predictable path that could be used by attacker in $TMPDIR)

2. On the usability front:

a. If you do `pg_waldump --path pg_wal.tar -s 0/31000000` it will dump
a lot of WAL records and then print final:
pg_waldump: error: could not find file "000000010000000000000034" in archive

However, with `pg_waldump --path pg_wal.tar -s 0/31000000
--stats=record` (not passing '-e') it will simply bailout without
printing stats and with error:
pg_waldump: error: could not find file "000000010000000000000034" in archive

IMHO, it could print stats if it was capable of getting at least 1 WAL record.

3. The most critical issue for me was the initial lack of error
pass-through from pg_waldump (when used with WALs in tar) to the
pg_verifybackup. Now it works fine, so thanks for this:

a. pg_waldump is capable of discovering missing WALs as requested and
throwing proper return code (good)
$ /usr/pgsql19/bin/pg_waldump --path pg_wal.tar -s 0/31005F70 -e 0/343D2650 -q
pg_waldump: error: could not find file "000000010000000000000034" in archive
$ echo $?
1
$

b. pg_verifybackup now also complains properly with missing WAL inside tar

$ tar --delete -f pg_wal.tar 000000010000000000000032 # simulate loss of file
$ tar -tf pg_wal.tar
000000010000000000000031
archive_status/000000010000000000000031.done
archive_status/000000010000000000000032.done
000000010000000000000033
$ grep Start-LSN backup_manifest
{ "Timeline": 1, "Start-LSN": "0/31005F70", "End-LSN": "0/333D2650" }
$ /usr/pgsql19/bin/pg_verifybackup -P /tmp/basebackup/
791372/791372 kB (100%) verified
pg_waldump: error: could not find file "000000010000000000000032" in archive
pg_verifybackup: error: WAL parsing failed for timeline 1
$ echo $?
1
$

-J.

  18. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-11-21T11:44:26Z

    On Wed, Nov 19, 2025 at 1:50 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Mon, Nov 17, 2025 at 5:51 AM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Thu, Nov 6, 2025 at 2:33 PM Amul Sul <sulamul@gmail.com> wrote:
> > >
> > > On Mon, Oct 20, 2025 at 8:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > > >
> > > > On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
> > > > [....]
> > > Kindly have a look at the attached version. Thank you !
> > >
> >
> > Attached is the rebased version against the latest master head (e76defbcf09).
>
> Hi Amul, thanks for working on this. I haven't really looked at the
> source code deeply (I trust Robert eyes much more than mine on this
> one), just skimmed a little bit:
>
> 1. As stated earlier, get_tmp_walseg_path() is still vulnerable (it
> uses predictable path that could be used by attacker in $TMPDIR)
>

Yeah, I haven't done anything regarding this since I am unsure of what
should be done and what the risks involved are. I am thinking of
taking Robert's opinion on this.

> 2. On the usability front:
>
> a. If you do `pg_waldump --path pg_wal.tar -s 0/31000000` it will dump
> a lot of WAL records and then print final:
> pg_waldump: error: could not find file "000000010000000000000034" in archive
>
> However, with `pg_waldump --path pg_wal.tar -s 0/31000000
> --stats=record` (not passing '-e') it will simply bailout without
> printing stats and with error:
> pg_waldump: error: could not find file "000000010000000000000034" in archive
>
> IMHO, it could print stats if it was capable of getting at least 1 WAL record.
>

The similar behavior in the current pg_waldump when using the --path
option with a WAL directory and a starting LSN. E.g:

$ pg_waldump -s 0/04FE36E0 --path=/tmp/backup/tmp/ --stats=record

pg_waldump: first record is after 0/04FE36E0, at 0/04FE3F90, skipping
over 2224 bytes
pg_waldump: error: could not find file "000000010000000000000009": No
such file or directory


> 3. The most critical issue for me was the initial lack of error
> pass-through from pg_waldump (when used with WALs in tar) to the
> pg_verifybackup. Now it works fine, so thanks for this:
>

Thanks, that was exactly the intention -- to complete pg_verifybackup
for tar-formatted backup verification.

Regards,
Amul

  19. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-11-21T12:16:24Z

    On Mon, Nov 17, 2025 at 10:20 AM Amul Sul <sulamul@gmail.com> wrote:
>
> On Thu, Nov 6, 2025 at 2:33 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Mon, Oct 20, 2025 at 8:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > >
> > > On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
> > > [....]
> > Kindly have a look at the attached version. Thank you !
> >
>

Attached is the updated version. I have fixed an assertion failure
that can occasionally occur with a partial WAL page read.

Regards,
Amul

  20. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-11-25T06:37:32Z

    On Fri, Nov 21, 2025 at 5:14 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Wed, Nov 19, 2025 at 1:50 PM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> >
> > On Mon, Nov 17, 2025 at 5:51 AM Amul Sul <sulamul@gmail.com> wrote:
> > >
> > > On Thu, Nov 6, 2025 at 2:33 PM Amul Sul <sulamul@gmail.com> wrote:
> > > >
> > > > On Mon, Oct 20, 2025 at 8:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > > > >
> > > > > On Thu, Oct 16, 2025 at 7:49 AM Amul Sul <sulamul@gmail.com> wrote:
> > > > > [....]
> > > > Kindly have a look at the attached version. Thank you !
> > > >
> > >
> > > Attached is the rebased version against the latest master head (e76defbcf09).
> >
> > Hi Amul, thanks for working on this. I haven't really looked at the
> > source code deeply (I trust Robert eyes much more than mine on this
> > one), just skimmed a little bit:
> >
> > 1. As stated earlier, get_tmp_walseg_path() is still vulnerable (it
> > uses predictable path that could be used by attacker in $TMPDIR)
> >
>
> Yeah, I haven't done anything regarding this since I am unsure of what
> should be done and what the risks involved are. I am thinking of
> taking Robert's opinion on this.
>

Per offline discussion with Robert and Jakub, I have updated the patch
to use mkdtemp() as suggested, which is already available in the tree
for similar purposes. Thanks !


Regards,
Amul

  21. Re: pg_waldump: support decoding of WAL inside tarfile

    Chao Li <li.evan.chao@gmail.com> — 2025-11-25T08:50:29Z

    Hi Amul,

I reviewed the patch and got some comments:

> On Nov 25, 2025, at 14:37, Amul Sul <sulamul@gmail.com> wrote:
> 
> 
> Regards,
> Amul
> <v8-0001-Refactor-pg_waldump-Move-some-declarations-to-new.patch><v8-0002-Refactor-pg_waldump-Separate-logic-used-to-calcul.patch><v8-0003-Refactor-pg_waldump-Restructure-TAP-tests.patch><v8-0004-pg_waldump-Add-support-for-archived-WAL-decoding.patch><v8-0005-pg_waldump-Remove-the-restriction-on-the-order-of.patch><v8-0006-pg_verifybackup-Delay-default-WAL-directory-prepa.patch><v8-0007-pg_verifybackup-Rename-the-wal-directory-switch-t.patch><v8-0008-pg_verifybackup-enabled-WAL-parsing-for-tar-forma.patch>

1 - 0001 - pg_waldump.h
```
+ * pg_waldump.h - decode and display WAL
+ *
+ * Copyright (c) 2013-2025, PostgreSQL Global Development Group
```

This header file is brand new, so copyright year should be only 2025.

2 - 0001 - pg_waldump.c
```
-static int	WalSegSz;
+int			WalSegSz = DEFAULT_XLOG_SEG_SIZE;
```

0001 claims a refactoring, but if you initialize WalSegSz with DEFAULT_XLOG_SEG_SIZE, then the behavior is changing, this change is no longer a pure refactor.

I would suggest leave WalSegSz uninitiated (compiler will set 0 to it), then no behavior change, so that 0001 stays a self-contained pure refactor.

The other nit thing is that, as “static” is removed, now “WalSegSz” is placed in middle of two static variables, which looks not good. If I were making the code change, I would have moved WalSegSz to after all static variables.

3 - 0002
```
@@ -383,21 +406,11 @@ WALDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
 				XLogRecPtr targetPtr, char *readBuff)
 {
 	XLogDumpPrivate *private = state->private_data;
-	int			count = XLOG_BLCKSZ;
+	int			count = required_read_len(private, targetPagePtr, reqLen);
 	WALReadError errinfo;
 
-	if (XLogRecPtrIsValid(private->endptr))
-	{
-		if (targetPagePtr + XLOG_BLCKSZ <= private->endptr)
-			count = XLOG_BLCKSZ;
-		else if (targetPagePtr + reqLen <= private->endptr)
-			count = private->endptr - targetPagePtr;
-		else
-		{
-			private->endptr_reached = true;
-			return -1;
-		}
-	}
+	if (private->endptr_reached)
+		return -1;
```

This change introduces a logic hole. In old code, it sets private->endptr_reached = true; and return -1. In the code code, count and private->endptr_reached assignments are wrapped into required_read_len(). However, required_read_len() doesn’t check if private->endptr_reached has already been true, so that the logic hole is that, if private->endptr_reached is already true when calling required_read_len(), and required_read_len() returns a positive count, if (private->endptr_reached) will also be satisfied and return -1 from the function.

So, to be safe, we should check “if (count < 0) return -1”.

4 - 0002
```
+/* Returns the size in bytes of the data to be read. */
+static inline int
+required_read_len(XLogDumpPrivate *private, XLogRecPtr targetPagePtr,
+				  int reqLen)
+{
```

The function comment is too simple. It doesn’t cover the case where -1 is returned.

5 - 0003
```
+my @scenario = (
+	{
+		'path' => $node->data_dir
+	});
 
-@lines = test_pg_waldump('--limit' => 6);
-is(@lines, 6, 'limit option observed');
+for my $scenario (@scenario)
+{
```

"my @scenario” should be "my @scenarios”, so that for line become "for my $scenario (@scenarios)”, a little bit clearer.

6 - 0003
```
+	SKIP:
+	{
```

Why SKIP label is defined here? A SKIP label usually follows a skip statement, for example: in bin/pg_ctl/t/001_start_stop.pl
```
SKIP:
{
skip "unix-style permissions not supported on Windows", 2
if ($windows_os);

ok(-f $logFileName);
ok(check_mode_recursive("$tempdir/data", 0700, 0600));
}
```

7 - 0004 - Makefile
```
 	$(WIN32RES) \
 	compat.o \
 	pg_waldump.o \
+	archive_waldump.o \
 	rmgrdesc.o \
 	xlogreader.o \
 	xlogstats.o
```

Obviously the list was in alphabetical order, so archive_waldump.o should be placed before compat.o.

8 - 0004
```
+/*
+ * pg_waldump's XLogReaderRoutine->page_read callback to support dumping WAL
+ * files from tar archives.
+ */
+static int
+TarWALDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
+				   XLogRecPtr targetPtr, char *readBuff)
+{
+	XLogDumpPrivate *private = state->private_data;
+	int			count = required_read_len(private, targetPagePtr, reqLen);
```

Looking the page_read’s spec:
```
	/*
	 * Data input callback
	 *
	 * This callback shall read at least reqLen valid bytes of the xlog page
	 * starting at targetPagePtr, and store them in readBuf.  The callback
	 * shall return the number of bytes read (never more than XLOG_BLCKSZ), or
	 * -1 on failure.  The callback shall sleep, if necessary, to wait for the
	 * requested bytes to become available.  The callback will not be invoked
	 * again for the same page unless more than the returned number of bytes
	 * are needed.
	 *
	 * targetRecPtr is the position of the WAL record we're reading.  Usually
	 * it is equal to targetPagePtr + reqLen, but sometimes xlogreader needs
	 * to read and verify the page or segment header, before it reads the
	 * actual WAL record it's interested in.  In that case, targetRecPtr can
	 * be used to determine which timeline to read the page from.
	 *
	 * The callback shall set ->seg.ws_tli to the TLI of the file the page was
	 * read from.
	 */
	XLogPageReadCB page_read;
```

It says that page_read must read reqLen bytes, otherwise it should wait for more bytes.

However, TarWALDumpReadPage just calculate how many bytes can read and only read that long, which breaks the protocol. Is it a problem?

9 - 0004
```
+/*
+ * Create an astreamer that can read WAL from tar file.
+ */
+static astreamer *
+astreamer_waldump_new(XLogDumpPrivate *privateInfo)
+{
+	astreamer_waldump *streamer;
+
+	streamer = palloc0(sizeof(astreamer_waldump));
+	*((const astreamer_ops **) &streamer->base.bbs_ops) =
+		&astreamer_waldump_ops;
+
+	streamer->privateInfo = privateInfo;
+
+	return &streamer->base;
+}
```

This function allocates memory for streamer but only returns &streamer->base, so memory of streamer is leaked.

Also, in the function comment, “from tar file” => “from a tar file”.

10 - 0004
```
+ * End-of-stream processing for a astreamer_waldump stream.
```

Nit typo: a => an

11 - 0004
```
+	if (!IsValidWalSegSize(WalSegSz))
+	{
+		pg_log_error(ngettext("invalid WAL segment size in WAL file from archive \"%s\" (%d byte)",
+							  "invalid WAL segment size in WAL file from archive \"%s\" (%d bytes)",
+							  WalSegSz),
+					 privateInfo->archive_name, WalSegSz);
+		pg_log_error_detail("The WAL segment size must be a power of two between 1 MB and 1 GB.");
+		exit(1);
+	}
```

Why don’t pg_fatal()?

12 - 0005
```
+		/* Create a temporary file if one does not already exist */
+		if (!entry->tmpseg_exists)
+		{
+			write_fp = prepare_tmp_write(entry->segno);
+			entry->tmpseg_exists = true;
+		}
+
+		/* Flush data from the buffer to the file */
+		perform_tmp_write(entry->segno, &entry->buf, write_fp);
+		resetStringInfo(&entry->buf);
+
+		/*
+		 * The change in the current segment entry indicates that the reading
+		 * of this file has ended.
+		 */
+		if (entry != privateInfo->cur_wal && write_fp != NULL)
+		{
+			fclose(write_fp);
+			write_fp = NULL;
+		}
```

When entry->tmpseg_exists is true, then write_fp will not be initialized, but there should be a check to make sure write_fp is not NULL before perform_tmp_write(). 

Also, if write_fp != NULL, should we anyway close the file without considering entry != privateInfo->cur_wal? Otherwise write_fp may be left open.

13 - 0005
```
+	 * Use the directory specified by the TEMDIR environment variable. If it’s
```

Typo: TEMDIR => TMPDIR

14 - 0005
```
+ * Set up a temporary directory to temporarily store WAL segments.
```

temporary and temporarily are redundant.

No comment for 0007.

15 - 0007

I wonder why we need to manually po files? This is the first time I see a patch including po file changes.

16 - 0008
```
+		{
+			pg_log_error("wal archive not found");
+			pg_log_error_hint("Specify the correct path using the option -w/--wal-path."
+							  "Or you must use -n/--no-parse-wal when verifying a tar-format backup.");
+			exit(1);
+		}
```

“wal” should be “WAL”.

In the hint message, there should be a white space between the two sentences.

Again, why not pg_fatal().

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  22. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2025-11-26T06:02:04Z

    On Tue, Nov 25, 2025 at 2:21 PM Chao Li <li.evan.chao@gmail.com> wrote:
>
> Hi Amul,
>
> I reviewed the patch and got some comments:
>

Thanks for the review. Replying inline below.

> 1 - 0001 - pg_waldump.h
> ```
> + * pg_waldump.h - decode and display WAL
> + *
> + * Copyright (c) 2013-2025, PostgreSQL Global Development Group
> ```
>
> This header file is brand new, so copyright year should be only 2025.
>

Fixed in the attached version.

> 2 - 0001 - pg_waldump.c
> ```
> -static int     WalSegSz;
> +int                    WalSegSz = DEFAULT_XLOG_SEG_SIZE;
> ```
>
> 0001 claims a refactoring, but if you initialize WalSegSz with DEFAULT_XLOG_SEG_SIZE, then the behavior is changing, this change is no longer a pure refactor.
>
> I would suggest leave WalSegSz uninitiated (compiler will set 0 to it), then no behavior change, so that 0001 stays a self-contained pure refactor.
>

Agreed.

> The other nit thing is that, as “static” is removed, now “WalSegSz” is placed in middle of two static variables, which looks not good. If I were making the code change, I would have moved WalSegSz to after all static variables.
>

I placed it before the static declaration.

> 3 - 0002
> ```
> @@ -383,21 +406,11 @@ WALDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
>                                 XLogRecPtr targetPtr, char *readBuff)
>  {
>         XLogDumpPrivate *private = state->private_data;
> -       int                     count = XLOG_BLCKSZ;
> +       int                     count = required_read_len(private, targetPagePtr, reqLen);
>         WALReadError errinfo;
>
> -       if (XLogRecPtrIsValid(private->endptr))
> -       {
> -               if (targetPagePtr + XLOG_BLCKSZ <= private->endptr)
> -                       count = XLOG_BLCKSZ;
> -               else if (targetPagePtr + reqLen <= private->endptr)
> -                       count = private->endptr - targetPagePtr;
> -               else
> -               {
> -                       private->endptr_reached = true;
> -                       return -1;
> -               }
> -       }
> +       if (private->endptr_reached)
> +               return -1;
> ```
>
> This change introduces a logic hole. In old code, it sets private->endptr_reached = true; and return -1. In the code code, count and private->endptr_reached assignments are wrapped into required_read_len(). However, required_read_len() doesn’t check if private->endptr_reached has already been true, so that the logic hole is that, if private->endptr_reached is already true when calling required_read_len(), and required_read_len() returns a positive count, if (private->endptr_reached) will also be satisfied and return -1 from the function.
>
> So, to be safe, we should check “if (count < 0) return -1”.
>

I do not really understand the logical hole where the behaviour is the
same as the previous, but I like the idea of checking endptr_reached.
This is quite unlikely to be true, but it looks like good practice to
check that flag before setting it. Did it that way in the attached
version.

> 4 - 0002
> ```
> +/* Returns the size in bytes of the data to be read. */
> +static inline int
> +required_read_len(XLogDumpPrivate *private, XLogRecPtr targetPagePtr,
> +                                 int reqLen)
> +{
> ```
>
> The function comment is too simple. It doesn’t cover the case where -1 is returned.
>

Okay.

> 5 - 0003
> ```
> +my @scenario = (
> +       {
> +               'path' => $node->data_dir
> +       });
>
> -@lines = test_pg_waldump('--limit' => 6);
> -is(@lines, 6, 'limit option observed');
> +for my $scenario (@scenario)
> +{
> ```
>
> "my @scenario” should be "my @scenarios”, so that for line become "for my $scenario (@scenarios)”, a little bit clearer.
>

Done.

> 6 - 0003
> ```
> +       SKIP:
> +       {
> ```
>
> Why SKIP label is defined here? A SKIP label usually follows a skip statement, for example: in bin/pg_ctl/t/001_start_stop.pl
> ```
> SKIP:
> {
> skip "unix-style permissions not supported on Windows", 2
> if ($windows_os);
>
> ok(-f $logFileName);
> ok(check_mode_recursive("$tempdir/data", 0700, 0600));
> }
> ```
>

Yeah, I knew that, but that is needed in the next patch where I wanted
to avoid a large diff when introducing SKIP and the associated
indentation. This patch is not expected to be committed independently,
and I have added a note in the commit message for the same.

> 7 - 0004 - Makefile
> ```
>         $(WIN32RES) \
>         compat.o \
>         pg_waldump.o \
> +       archive_waldump.o \
>         rmgrdesc.o \
>         xlogreader.o \
>         xlogstats.o
> ```
>
> Obviously the list was in alphabetical order, so archive_waldump.o should be placed before compat.o.
>

Done.

> 8 - 0004
> ```
> +/*
> + * pg_waldump's XLogReaderRoutine->page_read callback to support dumping WAL
> + * files from tar archives.
> + */
> +static int
> +TarWALDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen,
> +                                  XLogRecPtr targetPtr, char *readBuff)
> +{
> +       XLogDumpPrivate *private = state->private_data;
> +       int                     count = required_read_len(private, targetPagePtr, reqLen);
> ```
>
> Looking the page_read’s spec:
> ```
>         /*
>          * Data input callback
>          *
>          * This callback shall read at least reqLen valid bytes of the xlog page
>          * starting at targetPagePtr, and store them in readBuf.  The callback
>          * shall return the number of bytes read (never more than XLOG_BLCKSZ), or
>          * -1 on failure.  The callback shall sleep, if necessary, to wait for the
>          * requested bytes to become available.  The callback will not be invoked
>          * again for the same page unless more than the returned number of bytes
>          * are needed.
>          *
>          * targetRecPtr is the position of the WAL record we're reading.  Usually
>          * it is equal to targetPagePtr + reqLen, but sometimes xlogreader needs
>          * to read and verify the page or segment header, before it reads the
>          * actual WAL record it's interested in.  In that case, targetRecPtr can
>          * be used to determine which timeline to read the page from.
>          *
>          * The callback shall set ->seg.ws_tli to the TLI of the file the page was
>          * read from.
>          */
>         XLogPageReadCB page_read;
> ```
>
> It says that page_read must read reqLen bytes, otherwise it should wait for more bytes.
>
> However,
 just calculate how many bytes can read and only read that long, which
breaks the protocol. Is it a problem?
>

The behaviour is the same as the routine used to read the bare WAL
file. I don't think there will be any problem for the pg_waldump.

> 9 - 0004
> ```
> +/*
> + * Create an astreamer that can read WAL from tar file.
> + */
> +static astreamer *
> +astreamer_waldump_new(XLogDumpPrivate *privateInfo)
> +{
> +       astreamer_waldump *streamer;
> +
> +       streamer = palloc0(sizeof(astreamer_waldump));
> +       *((const astreamer_ops **) &streamer->base.bbs_ops) =
> +               &astreamer_waldump_ops;
> +
> +       streamer->privateInfo = privateInfo;
> +
> +       return &streamer->base;
> +}
> ```
>
> This function allocates memory for streamer but only returns &streamer->base, so memory of streamer is leaked.
>

May I know why you think there would be a memory leak? I believe the
address of the structure is the same as the address of its first
member, base. I am returning base because the goal is to return a
generic astreamer type, which is the standard approach used in other
archive streamer code.

> Also, in the function comment, “from tar file” => “from a tar file”.
>
> 10 - 0004
> ```
> + * End-of-stream processing for a astreamer_waldump stream.
> ```
>
> Nit typo: a => an
>

Done.

> 11 - 0004
> ```
> +       if (!IsValidWalSegSize(WalSegSz))
> +       {
> +               pg_log_error(ngettext("invalid WAL segment size in WAL file from archive \"%s\" (%d byte)",
> +                                                         "invalid WAL segment size in WAL file from archive \"%s\" (%d bytes)",
> +                                                         WalSegSz),
> +                                        privateInfo->archive_name, WalSegSz);
> +               pg_log_error_detail("The WAL segment size must be a power of two between 1 MB and 1 GB.");
> +               exit(1);
> +       }
> ```
>
> Why don’t pg_fatal()?
>

This is how we do when we need to emit error details as well.

> 12 - 0005
> ```
> +               /* Create a temporary file if one does not already exist */
> +               if (!entry->tmpseg_exists)
> +               {
> +                       write_fp = prepare_tmp_write(entry->segno);
> +                       entry->tmpseg_exists = true;
> +               }
> +
> +               /* Flush data from the buffer to the file */
> +               perform_tmp_write(entry->segno, &entry->buf, write_fp);
> +               resetStringInfo(&entry->buf);
> +
> +               /*
> +                * The change in the current segment entry indicates that the reading
> +                * of this file has ended.
> +                */
> +               if (entry != privateInfo->cur_wal && write_fp != NULL)
> +               {
> +                       fclose(write_fp);
> +                       write_fp = NULL;
> +               }
> ```
>
> When entry->tmpseg_exists is true, then write_fp will not be initialized, but there should be a check to make sure write_fp is not NULL before perform_tmp_write().
>

perform_tmp_write() has assert for the same.

> Also, if write_fp != NULL, should we anyway close the file without considering entry != privateInfo->cur_wal? Otherwise write_fp may be left open.
>

We read the WAL from the tar file in chunks, and those same chunks are
written to the temporary file within the loop. If we close the
temporary file now, we will have to open it again later for the next
chunk write. Could you elaborate on a scenario where you believe this
file might be left open unintentionally?

> 13 - 0005
> ```
> +        * Use the directory specified by the TEMDIR environment variable. If it’s
> ```
>
> Typo: TEMDIR => TMPDIR
>

Done.

> 14 - 0005
> ```
> + * Set up a temporary directory to temporarily store WAL segments.
> ```
>
> temporary and temporarily are redundant.
>

I believe that is grammatically correct and clear.

> No comment for 0007.
>
> 15 - 0007
>
> I wonder why we need to manually po files? This is the first time I see a patch including po file changes.
>

Okay, I included that initially to ensure the PO file update wasn't
overlooked during commit. I have removed it to minimize the diff and
added the note in the patch commit message.

> 16 - 0008
> ```
> +               {
> +                       pg_log_error("wal archive not found");
> +                       pg_log_error_hint("Specify the correct path using the option -w/--wal-path."
> +                                                         "Or you must use -n/--no-parse-wal when verifying a tar-format backup.");
> +                       exit(1);
> +               }
> ```
>
> “wal” should be “WAL”.
>
> In the hint message, there should be a white space between the two sentences.
>

Done.

Thanks again for your review comments; they are quite helpful. Kindly
take a look at the attached version.

Regards,
Amul

  23. Re: pg_waldump: support decoding of WAL inside tarfile

    Chao Li <li.evan.chao@gmail.com> — 2025-11-26T07:23:39Z

    
> On Nov 26, 2025, at 14:02, Amul Sul <sulamul@gmail.com> wrote:
> 
>> 9 - 0004
>> ```
>> +/*
>> + * Create an astreamer that can read WAL from tar file.
>> + */
>> +static astreamer *
>> +astreamer_waldump_new(XLogDumpPrivate *privateInfo)
>> +{
>> +       astreamer_waldump *streamer;
>> +
>> +       streamer = palloc0(sizeof(astreamer_waldump));
>> +       *((const astreamer_ops **) &streamer->base.bbs_ops) =
>> +               &astreamer_waldump_ops;
>> +
>> +       streamer->privateInfo = privateInfo;
>> +
>> +       return &streamer->base;
>> +}
>> ```
>> 
>> This function allocates memory for streamer but only returns &streamer->base, so memory of streamer is leaked.
>> 
> 
> May I know why you think there would be a memory leak? I believe the
> address of the structure is the same as the address of its first
> member, base. I am returning base because the goal is to return a
> generic astreamer type, which is the standard approach used in other
> archive streamer code.

Ah… Got it.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  24. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-01-02T12:29:54Z

    Attached is the rebased version, includes some code comment improvements.

Regards,
Amul

  25. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-01-16T20:38:03Z

    On Fri, Jan 2, 2026 at 7:30 AM Amul Sul <sulamul@gmail.com> wrote:
> Attached is the rebased version, includes some code comment improvements.

Regarding 0001, I don't think that passing the WAL segment size around
in a global variable is a good practice. It makes it hard to tell
whether the variable is guaranteed to have been set properly at all
the places where it is used. Of course, this patch set does not
introduce that problem, but I think it could avoid making it worse by
adding WalSegSz to XLogDumpPrivate instead of exporting the global
variable everywhere. That struct seems to be available everyplace
where we need the value. Actually, though, I'm inclined to remove the
file-level global as well. Patch for that attached. I'm not against
global variables in general, but I'm against global variables that are
just there to avoid passing the right stuff around via function
argument lists.

0002 claims to be just a refactoring but if
(unlikely(private->endptr_reached)) return -1 is net new code, so I
object. I realize this contradicts the review comment from Chao Li,
but the principal that refactoring shouldn't change the logic is more
important than whatever value we might get out of adding this sanity
check here -- and I bet if we look into it we'll find that there are
lots of other places in PostgreSQL that would also need to be changed
if we wanted such a sanity check everywhere that we have code like
this. I'd also like to point out that the commit message isn't really
adequate to understand why this helps achieve the overall goal of the
patch set.

0003 looks functionally correct but notice that you've stolen some but
not all of the tests from the "# various ways of specifying WAL range"
section without copying the comment or explaining in the commit
message why you've moved some tests but not others.

Regarding 0004:

Instead of making ArchivedWAL_HTAB a global variable, can we just
store a pointer to it in XLogDumpPrivate? (I would change the name to
something else, something all lower case.) Or maybe the pointer should
go someplace else, but I'm skeptical of a global variable here for the
same reasons as 0001.

I think ArchivedWALFile would be a better name than ArchivedWALEntry,
and cur_wal_file or cur_file would be a better name than cur_wal.

I'm extremely happy with the way that the hash table looks and the new
logic in init_archive_reader() to make sure that we don't need to
reread the beginning of the WAL but can reuse the data already read.
That seems like a huge improvement to me.

+               if (recptr >= endPtr)
+               {
+                       /* Discard the buffered data */
+                       resetStringInfo(&entry->buf);
+                       len = 0;
+
+                       /*
+                        * Push back the partial page data for the
current page to the
+                        * buffer, ensuring it remains full page
available for re-reading
+                        * if requested.
+                        */
+                       if (p > readBuff)
+                       {
+                               Assert((count - nbytes) > 0);
+                               appendBinaryStringInfo(&entry->buf,
readBuff, count - nbytes);
+                       }
+               }

It feels pretty inelegant to me to pull data back into &entry->buf
from readBuff here. I think what you should do is just do this test
once, before entering the while (nbytes > 0) loop. Then you'll always
have p == readBuff, so the nested if statement is no longer required.
Also, the way you have it, if this triggers for the first loop
iteration, it will trigger for every subsquent loop iteration, which
seems wasteful.

+               if (len > 0 && recptr > startPtr)
+               {
+                       int                     skipBytes = 0;
+
+                       /*
+                        * The required offset is not at the start of
the buffer, so skip
+                        * bytes until reaching the desired offset of
the target page.
+                        */
+                       skipBytes = recptr - startPtr;
+
+                       buf += skipBytes;
+                       len -= skipBytes;
+               }

This logic seems pretty confusing. It looks as though len can go
negative, because what if skipBytes > len? I'm not sure that's really
possible here, but there's no comments explaining why it can't and no
assertion verifying that it doesn't. I think you should probably try
to unify this with the following if statement that is gated by (len >
0). In other places where we have code like this, what I've usually
found useful is to compute the number of bytes to copy as Min(bytes
available from source, bytes that will fit into the target), and I
think that technique might be useful here. For example, in
BlockRefTableRead, length is always the number of bytes that the
caller wants which we haven't yet read, and we compute bytes_to_copy =
Min(length, buffer->used - buffer->cursor). We then memcpy(data,
&buffer->data[buffer->cursor], bytes_to_copy) and then adjust data and
length. A similar bit of code not written by me can be found in
WALReadFromBuffers, which maintains nbytes as the remaining number of
bytes to be copied and then computes the size of the next copy as
npagebytes = Min(nbytes, XLOG_BLCKSZ - offset). I think your if (len >
0 && recptr > startPtr) block is trying to do something similar to
what the Min() is doing in those examples, but I find it hard to
understand whether it is doing it correctly. I think you need some
combination of clearer variable names, more comments, and/or rewriting
this to be easier to read and understand without help.

+               entry = privateInfo->cur_wal;
+
+               /* Fetch more data */
+               if (read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
+                       break;                  /* archive file ended */

Why do we set entry to privateInfo->cur_wal and then call
read_archive_file() afterwards? Doesn't read_archive_file() have the
potential to change cur_wal? If so, don't we want to look at the entry
that is current after we've called read_archive_file(), rather than
before? Also, to harp on the naming a bit more, stuff like entry =
privateInfo->cur_wal kind of shows that you haven't made the naming
real consistent. This could be more clear if it said something like
wal_file = privateInfo->cur_wal_file.

+               /* Found the required entry */
+               if (entry->segno == segno)
+                       return entry;
+
+               /*
+                * Ignore if the timeline is different or the current
segment is not
+                * the desired one.
+                */
+               if (privateInfo->timeline != entry->timeline ||
+                       privateInfo->startSegNo > entry->segno ||
+                       privateInfo->endSegNo < entry->segno)
+               {
+                       privateInfo->cur_wal = NULL;
+                       continue;
+               }
+

I don't really understand what's happening here. One thing that's odd
is that the first of these two if statements considers the timeline
and the second does not. It seems unlikely that this is correct. I
suspect that the hash table needs to be keyed by (TLI, segno) rather
than just by TLI, and that get_archive_wal_entry needs to take both a
TLI and a segment number as an argument.  Otherwise, what could
possibly prevent the first if statement from returning the correct
segment from the wrong timeline?

(Likewise, I think stuff in 0005 like prepare_tmp_write() should work
on a TLI and a segno, not just a segno. There probably should be very
few places in the patch where you pass only a segno and no TLI.
get_tmp_walseg_path() is a pretty clear sign of the problem: If you
were including the TLI in the file name, you would just use
XLogFileName() here, but because you left out the TLI, you end up
having to invent your own file naming convention. That seems pretty
bad: wouldn't we like the temporary file names to have the same names
as the files inside the archive? What happens if the same WAL file
exists in the archive with two different TLIs?)

Another thing that's odd is: why is this function resetting
privateInfo->cur_wal to NULL? I feel like it should be the job of
astreamer_waldump_content() to manage the value of
privateInfo->cur_wal, and this function doesn't seem to have any
business touching it. Even if it thinks that we want to ignore that
WAL file for purposes of *this* call to get_archive_wal_entry(),
surely it has no right to decide that the *next* call to this function
should ignore that entry as well.

Honestly, I don't really understand why we need to poke into
privateInfo at all here. I feel like the shape of this loop should be:
while (1) { if (read_archive_file(...) == 0) break; if (we have the
correct entry) return entry; } }. It seems to me that it ought to be
the job of the archive streamer to handle everything else, including
buffering more data and spilling to files. This should just iterate,
repeatedly calling read_archive_file(), until we either get to the
file we want or run out of archive.

+               /*
+                * XXX: If the segment being read not the requested
one, the data must
+                * be buffered, as we currently lack the mechanism to
write it to a
+                * temporary file. This is a known limitation that
will be fixed in the
+                * next patch, as the buffer could grow up to the full
WAL segment
+                * size.
+                */
+               if (segno > entry->segno)
+                       continue;

I couldn't figure out what was going on here for a while. Then I
realized that this makes some sense: if (segno > entry->segno), that
means that the segment number that we've found in the archive is older
than the requested segment number. Because we don't back up, that
means we don't need the data. I think the idea is that we would fall
through here and buffer the data if this if-test fails, but patch 0005
actually removes these two lines of code, so I think something here is
kind of messed up. I think the result will be that with both patches
applied, we'll buffer data we don't actually need for anything.

Generally, I think that some details here should be revisited:

+typedef struct ArchivedWALEntry
+{
+       uint32          status;                 /* hash status */
+       XLogSegNo       segno;                  /* hash key: WAL
segment number */
+       TimeLineID      timeline;               /* timeline of this wal file */
+
+       StringInfoData buf;
+       bool            tmpseg_exists;  /* spill file exists? */
+
+       int                     total_read;             /* total read
of archived WAL segment */
+} ArchivedWALEntry;

The first few members are great, except that timeline needs to be part
of the hash key as well, so you probably need to make a two-element
struct with a segno and a TLI and make that struct be the thing that
comes right after status. After that, I'm not so sure. Essentially, I
think what's happening here is that segno*file_size+total_read is the
end LSN for buf, and the start LSN value is that value minus buf.len.
And if tmpseg_exists is true, then the buffer might be empty, and you
can read however much of the segment we found from the temp file. I
find that confusing. If we're going to do something like this, I feel
like it would make more sense to store XLogRecPtr starttli instead of
int total_read, so that we explicitly mention the first LSN stored in
the buffer, and you infer the ending LSN from the length of the
buffer.

But, stepping back a bit, why do we have a StringInfoData in every
ArchivedWALEntry? It makes sense to do it that way if we're going to
buffer all of the WAL data in memory. In that case, you need a
separate buffer for every ArchivedWALEntry, and so this is logical.
But if we're going to spill to disk, then there should only ever be
one buffer in use at any given time, so why have a buffer in every
ArchivedWALEntry instead of a single buffer inside of XLogDumpPrivate?
We could choose to treat the hash table as the set of previous
segments for which we have written data to disk, and the current
segment and any associated buffer would be tracked inside the
XLogDumpPrivate or astreamer_waldump, which would mean that only one
copy of them would exist. I'm tempted to say that the direction I'm
sketching here is the right way to go, but I'm not 100% certain that's
true. It could be write to have a buffer per WAL file, as you have
here, but if so, we should know why we're doing that and how it fits
into the design, and I'm not currently seeing a real reason for it.

+       if (strstr(member->pathname, "PaxHeaders."))
+               return false;

There is no way that a filename containing the string "PaxHeaders."
could ever pass the IsXLogFileName test just above. We shouldn't need
this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  26. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-01-19T11:13:30Z

    On Sat, Jan 17, 2026 at 2:08 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Jan 2, 2026 at 7:30 AM Amul Sul <sulamul@gmail.com> wrote:
> > Attached is the rebased version, includes some code comment improvements.
>

Thanks for the feedback. I am replying to a few comments inline below
regarding elements that are unlikely to change -- explaining the
reasoning why I believe they are correct -- just to ensure we are
aligned. For the remaining points, I generally agree and will either
implement the improvements or provide my reasoning if I don't.

>
> +               if (len > 0 && recptr > startPtr)
> +               {
> +                       int                     skipBytes = 0;
> +
> +                       /*
> +                        * The required offset is not at the start of
> the buffer, so skip
> +                        * bytes until reaching the desired offset of
> the target page.
> +                        */
> +                       skipBytes = recptr - startPtr;
> +
> +                       buf += skipBytes;
> +                       len -= skipBytes;
> +               }
>
> This logic seems pretty confusing. It looks as though len can go
> negative, because what if skipBytes > len? I'm not sure that's really
> possible here, but there's no comments explaining why it can't and no
> assertion verifying that it doesn't. I think you should probably try
> to unify this with the following if statement that is gated by (len >
> 0).

The preceding check if (recptr >= endPtr) ensures that within this
block, recptr is always less than endPtr. Since startPtr is derived as
endPtr - len, recptr naturally falls within the expected range (i.e.,
recptr - startPtr <= len). However, I agree that we can simplify this
by combining it with the subsequent if statement and/or by adding an
assert there.

>
> +               entry = privateInfo->cur_wal;
> +
> +               /* Fetch more data */
> +               if (read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
> +                       break;                  /* archive file ended */
>
> Why do we set entry to privateInfo->cur_wal and then call
> read_archive_file() afterwards? Doesn't read_archive_file() have the
> potential to change cur_wal? If so, don't we want to look at the entry
> that is current after we've called read_archive_file(), rather than
> before? Also, to harp on the naming a bit more, stuff like entry =
> privateInfo->cur_wal kind of shows that you haven't made the naming
> real consistent. This could be more clear if it said something like
> wal_file = privateInfo->cur_wal_file.

I see your point, and you’re right that assigning the entry to
privateInfo->cur_wal here seems useless; however, it becomes necessary
for the logic in patch 0005. I assigned it just before
read_archive_file() because that function can change where the current
WAL ends and a new one starts. I need to track the completed WAL entry
at that moment, as it may contain data that needs to be spilled to a
temporary file. I'll add a comment to clarify this. I also agree with
your feedback on variable naming and will improve that in the next
revision.

>
> +               /* Found the required entry */
> +               if (entry->segno == segno)
> +                       return entry;
> +
> +               /*
> +                * Ignore if the timeline is different or the current
> segment is not
> +                * the desired one.
> +                */
> +               if (privateInfo->timeline != entry->timeline ||
> +                       privateInfo->startSegNo > entry->segno ||
> +                       privateInfo->endSegNo < entry->segno)
> +               {
> +                       privateInfo->cur_wal = NULL;
> +                       continue;
> +               }
> +
>
> I don't really understand what's happening here. One thing that's odd
> is that the first of these two if statements considers the timeline
> and the second does not. It seems unlikely that this is correct. I
> suspect that the hash table needs to be keyed by (TLI, segno) rather
> than just by TLI, and that get_archive_wal_entry needs to take both a
> TLI and a segment number as an argument.  Otherwise, what could
> possibly prevent the first if statement from returning the correct
> segment from the wrong timeline?

Yeah, I need to swap the positions of these checks.

>
> Another thing that's odd is: why is this function resetting
> privateInfo->cur_wal to NULL? I feel like it should be the job of
> astreamer_waldump_content() to manage the value of
> privateInfo->cur_wal, and this function doesn't seem to have any
> business touching it. Even if it thinks that we want to ignore that
> WAL file for purposes of *this* call to get_archive_wal_entry(),
> surely it has no right to decide that the *next* call to this function
> should ignore that entry as well.
>
> Honestly, I don't really understand why we need to poke into
> privateInfo at all here. I feel like the shape of this loop should be:
> while (1) { if (read_archive_file(...) == 0) break; if (we have the
> correct entry) return entry; } }. It seems to me that it ought to be
> the job of the archive streamer to handle everything else, including
> buffering more data and spilling to files. This should just iterate,
> repeatedly calling read_archive_file(), until we either get to the
> file we want or run out of archive.
>

We previously tried having the astreamer function perform all
necessary checks (e.g., TLI and segment number), but this became
problematic in init_archive_reader when we needed to read a single WAL
page just to determine the WAL segment size without any validation. To
bypass those checks, I initially used a READ_ANY_WAL() macro, but it
felt like a hack.

In the current design, the archive streamer is responsible only for
reading and copying data to the buffer. To optimize this and avoid
unnecessary memcpy and  data buffering for WAL files that aren't
needed, we set privateInfo->cur_wal to NULL, signaling the streamer
code to skip those operations.

But, I am thinking that instead of setting privateInfo->cur_wal to
NULL, we could simply discard the buffer data at that place and let
memcpy in astreamer_content copy if it would be that much of an issue.

> +               /*
> +                * XXX: If the segment being read not the requested
> one, the data must
> +                * be buffered, as we currently lack the mechanism to
> write it to a
> +                * temporary file. This is a known limitation that
> will be fixed in the
> +                * next patch, as the buffer could grow up to the full
> WAL segment
> +                * size.
> +                */
> +               if (segno > entry->segno)
> +                       continue;
>
> I couldn't figure out what was going on here for a while. Then I
> realized that this makes some sense: if (segno > entry->segno), that
> means that the segment number that we've found in the archive is older
> than the requested segment number. Because we don't back up, that
> means we don't need the data. I think the idea is that we would fall
> through here and buffer the data if this if-test fails, but patch 0005
> actually removes these two lines of code, so I think something here is
> kind of messed up. I think the result will be that with both patches
> applied, we'll buffer data we don't actually need for anything.
>

Yeah, should discard buffered data too.

> Generally, I think that some details here should be revisited:
>
> +typedef struct ArchivedWALEntry
> +{
> +       uint32          status;                 /* hash status */
> +       XLogSegNo       segno;                  /* hash key: WAL
> segment number */
> +       TimeLineID      timeline;               /* timeline of this wal file */
> +
> +       StringInfoData buf;
> +       bool            tmpseg_exists;  /* spill file exists? */
> +
> +       int                     total_read;             /* total read
> of archived WAL segment */
> +} ArchivedWALEntry;
>
> The first few members are great, except that timeline needs to be part
> of the hash key as well, so you probably need to make a two-element
> struct with a segno and a TLI and make that struct be the thing that
> comes right after status. After that, I'm not so sure. Essentially, I
> think what's happening here is that segno*file_size+total_read is the
> end LSN for buf, and the start LSN value is that value minus buf.len.
> And if tmpseg_exists is true, then the buffer might be empty, and you
> can read however much of the segment we found from the temp file. I
> find that confusing. If we're going to do something like this, I feel
> like it would make more sense to store XLogRecPtr starttli instead of
> int total_read, so that we explicitly mention the first LSN stored in
> the buffer, and you infer the ending LSN from the length of the
> buffer.
>

total_read is not only the total of buffered data; it also includes
data that has been spilled to a temporary file. If the buffer is not
empty, the most recently read data from the archive exists there;
otherwise, if a temporary file exists, that data will be found there.
In short, it tracks the total bytes of WAL that have been read from
the archive. I will improve the code comment to reflect this.

> But, stepping back a bit, why do we have a StringInfoData in every
> ArchivedWALEntry? It makes sense to do it that way if we're going to
> buffer all of the WAL data in memory. In that case, you need a
> separate buffer for every ArchivedWALEntry, and so this is logical.
> But if we're going to spill to disk, then there should only ever be
> one buffer in use at any given time, so why have a buffer in every
> ArchivedWALEntry instead of a single buffer inside of XLogDumpPrivate?
> We could choose to treat the hash table as the set of previous
> segments for which we have written data to disk, and the current
> segment and any associated buffer would be tracked inside the
> XLogDumpPrivate or astreamer_waldump, which would mean that only one
> copy of them would exist. I'm tempted to say that the direction I'm
> sketching here is the right way to go, but I'm not 100% certain that's
> true. It could be write to have a buffer per WAL file, as you have
> here, but if so, we should know why we're doing that and how it fits
> into the design, and I'm not currently seeing a real reason for it.
>

In the earlier version, we had only one buffer where we copied
streamed data, but a problem (previously discussed [1]) arose when the
archive streamer reads data from two WAL files -- i.e., one is ending
and the next is starting. We have to keep track of where the previous
file ends and the next one starts, along with the WAL segment
information that each belongs to.

> +       if (strstr(member->pathname, "PaxHeaders."))
> +               return false;
>
> There is no way that a filename containing the string "PaxHeaders."
> could ever pass the IsXLogFileName test just above. We shouldn't need
> this.
>

This checks the directory name (e.g.,
x_dir/y_dir/PaxHeaders.NNNN/wal_file_name). The name of that metadata
file is exactly the same as the WAL file name, which is why
IsXLogFileName() doesn't help here.

Regards,
Amul

1] http://postgr.es/m/CA+Tgmoardk4VuthHc23vov+AVkhq7eT0mFUs-2ctAnP1uiTaog@mail.gmail.com

  27. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-01-19T13:30:48Z

    On Mon, Jan 19, 2026 at 6:14 AM Amul Sul <sulamul@gmail.com> wrote:
> We previously tried having the astreamer function perform all
> necessary checks (e.g., TLI and segment number), but this became
> problematic in init_archive_reader when we needed to read a single WAL
> page just to determine the WAL segment size without any validation. To
> bypass those checks, I initially used a READ_ANY_WAL() macro, but it
> felt like a hack.
>
> In the current design, the archive streamer is responsible only for
> reading and copying data to the buffer. To optimize this and avoid
> unnecessary memcpy and  data buffering for WAL files that aren't
> needed, we set privateInfo->cur_wal to NULL, signaling the streamer
> code to skip those operations.

Hmm. In that case, the code definitely deserves a comment, as that was
not at all clear to me.

> But, I am thinking that instead of setting privateInfo->cur_wal to
> NULL, we could simply discard the buffer data at that place and let
> memcpy in astreamer_content copy if it would be that much of an issue.

I don't think doing unnecessary copying is the right way forward. The
copying itself could be expensive, but I am a little concerned about
the memory utilization of this code. Suppose the user has increased
the WAL segment size to 1GB or even higher. It seems like we could
buffer a whole segment or maybe even more in some scenarios. If we
avoid copying data we don't need, then we also avoid buffering it in
memory.

In terms of the separation of concerns, we could view setting
privateInfo->cur_wal = NULL as a form of signaling, a way for this
code to tell the astreamer that it doesn't need the data buffering.
However, I think it might be better to make the signaling more
explicit. Instead of having the caller directly set the buffer to
NULL, or directly trim data out of the buffer, maybe it should set
some value in privateInfo that tells the astreamer what to do. For
instance, suppose it sets the oldest LSN that it might still care
about in privateInfo, and then the astreamer is free to do skip
copying of any data prior to that LSN, and discard any that it already
has. Especially if properly commented, I think this might be more
clear than what you have now. I am not 100% sure that's the right
idea, though. I just think right now it's a bit murky who does what
and why the division of responsibilities is what it is.

For example, read_archive_wal_page() currently has some responsibility
for discarding data, but is that because it is the right place to
discard data, or is that just because that's where we have an LSN
available that tells us what we can discard? I don't know, but an
explicit signaling mechanism like what I describe in the previous
paragraph can make it possible for those two things to happen in
different places.

> In the earlier version, we had only one buffer where we copied
> streamed data, but a problem (previously discussed [1]) arose when the
> archive streamer reads data from two WAL files -- i.e., one is ending
> and the next is starting. We have to keep track of where the previous
> file ends and the next one starts, along with the WAL segment
> information that each belongs to.

Hmm, interesting. My first thought was that if we were starting a new
WAL file, isn't it time to flush the data from the old one to a spill
file? But maybe it's not, if the WAL reader can reread records, and a
record can span file boundaries. But probably we need some comment so
that the next reader is not confused, and we definitely need to make
sure that the structure members in ArchivedWALEntry are well-chosen
and well-documented.

> > +       if (strstr(member->pathname, "PaxHeaders."))
> > +               return false;
> >
> > There is no way that a filename containing the string "PaxHeaders."
> > could ever pass the IsXLogFileName test just above. We shouldn't need
> > this.
> >
>
> This checks the directory name (e.g.,
> x_dir/y_dir/PaxHeaders.NNNN/wal_file_name). The name of that metadata
> file is exactly the same as the WAL file name, which is why
> IsXLogFileName() doesn't help here.

I think this code should only be considering files in the toplevel
directory, and skipping over any directories it finds. I absolutely
promise I am not going to commit anything that is specifically looking
for PaxHeaders. Nothing we've ever done with tar files up to now has
required that, and I don't think this should, either.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  28. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-01-23T12:27:11Z

    On Mon, Jan 19, 2026 at 7:01 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> > But, I am thinking that instead of setting privateInfo->cur_wal to
> > NULL, we could simply discard the buffer data at that place and let
> > memcpy in astreamer_content copy if it would be that much of an issue.
>
> I don't think doing unnecessary copying is the right way forward. The
> copying itself could be expensive, but I am a little concerned about
> the memory utilization of this code. Suppose the user has increased
> the WAL segment size to 1GB or even higher. It seems like we could
> buffer a whole segment or maybe even more in some scenarios. If we
> avoid copying data we don't need, then we also avoid buffering it in
> memory.
>
> In terms of the separation of concerns, we could view setting
> privateInfo->cur_wal = NULL as a form of signaling, a way for this
> code to tell the astreamer that it doesn't need the data buffering.
> However, I think it might be better to make the signaling more
> explicit. Instead of having the caller directly set the buffer to
> NULL, or directly trim data out of the buffer, maybe it should set
> some value in privateInfo that tells the astreamer what to do. For
> instance, suppose it sets the oldest LSN that it might still care
> about in privateInfo, and then the astreamer is free to do skip
> copying of any data prior to that LSN, and discard any that it already
> has. Especially if properly commented, I think this might be more
> clear than what you have now. I am not 100% sure that's the right
> idea, though. I just think right now it's a bit murky who does what
> and why the division of responsibilities is what it is.
>

The current implementation of astreamer_waldump_content() does not
have sufficient information to skip WAL segments during the initial
hash entry preparation and data-copying phase. Because the filtration
parameters -- which determine if a segment should be skipped -- depend
on the WAL segment size, we must first read a WAL page through the
streamer to calculate that size, which is done in
init_archive_reader(). Therefore, the responsibility of the archive
streamer is strictly to copy the WAL segment data to the buffer.

The skipping decision is handled inside get_archive_wal_entry(), which
sets privateInfo->cur_wal to NULL. In the next version, I am planning
to add a separate routine (with better commenting) that, along with
setting the pointer to NULL, releases that hash entry to avoid
unnecessary memory usage.

Another option I previously considered was adding the filtration logic
inside the archive streamer itself. However, since the very first read
is required to calculate the WAL segment size, the filter check cannot
be performed immediately. However, we could send a signal to the
archive streamer via privateInfo (e.g., a read_any_wal or
skip_wal_check boolean flag) to disable the filtration check until the
size is calculated. But that approach isn't very elegant; if the first
WAL page we read belongs to a segment we actually want to skip, we
would still have to run the filter check and handle the skip/removal
logic outside of the streamer (i.e., inside init_archive_reader()).
This would result in performing the same filtration check in two
different places.

Therefore, I believe performing the filtration check through
get_archive_wal_entry() and then calling a routine to clear
privateInfo->cur_wal and the hash entry is the better approach, IMO.
Additionally, once we consume a WAL file and move to the next one, the
hash entry and buffer for that WAL can be released to prevent
unnecessary memory consumption by calling the same routine that I am
planning to add.

> > > +       if (strstr(member->pathname, "PaxHeaders."))
> > > +               return false;
> > >
> > > There is no way that a filename containing the string "PaxHeaders."
> > > could ever pass the IsXLogFileName test just above. We shouldn't need
> > > this.
> > >
> >
> > This checks the directory name (e.g.,
> > x_dir/y_dir/PaxHeaders.NNNN/wal_file_name). The name of that metadata
> > file is exactly the same as the WAL file name, which is why
> > IsXLogFileName() doesn't help here.
>
> I think this code should only be considering files in the toplevel
> directory, and skipping over any directories it finds. I absolutely
> promise I am not going to commit anything that is specifically looking
> for PaxHeaders. Nothing we've ever done with tar files up to now has
> required that, and I don't think this should, either.

Ok, fair enough, my intention was to allow decoding of valid WAL data
from any directory in the tar archive, but I will go ahead and add
that restriction as suggested.

Regards,
Amul

  29. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-01-26T17:22:33Z

    On Fri, Jan 23, 2026 at 7:27 AM Amul Sul <sulamul@gmail.com> wrote:
> Another option I previously considered was adding the filtration logic
> inside the archive streamer itself. However, since the very first read
> is required to calculate the WAL segment size, the filter check cannot
> be performed immediately. However, we could send a signal to the
> archive streamer via privateInfo (e.g., a read_any_wal or
> skip_wal_check boolean flag) to disable the filtration check until the
> size is calculated. But that approach isn't very elegant; if the first
> WAL page we read belongs to a segment we actually want to skip, we
> would still have to run the filter check and handle the skip/removal
> logic outside of the streamer (i.e., inside init_archive_reader()).
> This would result in performing the same filtration check in two
> different places.

I mean, I don't really buy this logic. If the information added to
privateInfo is "here's the LSN before which you can remove stuff," and
it starts out initialized to 0/0, then the read of the first WAL page
causes no problem at all, because nothing is before 0/0. After it gets
updated to some non-zero value, the next call to
astreamer_waldump_content() can handle discarding any data we don't
need.

IMHO, the best argument for keeping things are is that in some cases,
that decision might result in a bit of delay in discarding old data,
but I don't think that really matters. I think all that we care about
is the peak memory utilization of an operation, and I don't think that
an explicit signaling system should increase that at all.

That said, I'm certainly willing to consider other ideas about how
this can work. However, I feel strongly that the logic needs to be not
only correct, but clear and well-explained. Setting cur_wal to NULL to
make the astreamer skip without adequate comments doesn't meet that
standard. Maybe with some better comments it's all right, but frankly
I'm a bit skeptical. Right now, you're using whether or not cur_wal is
NULL as a signal to skip data or not skip data. How is that better
than passing down the LSN and TLI that you want to read next and
letting the astreamer figure out what to do itself? It's a signaling
mechanism either way, but it seems a lot easier to figure out whether
we always keep the LSN and TLI updated properly than to figure out
whether cur_wal is always NULL at exactly the right times.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  30. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-01-27T12:06:34Z

    On Mon, Jan 26, 2026 at 10:52 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Jan 23, 2026 at 7:27 AM Amul Sul <sulamul@gmail.com> wrote:
> > Another option I previously considered was adding the filtration logic
> > inside the archive streamer itself. However, since the very first read
> > is required to calculate the WAL segment size, the filter check cannot
> > be performed immediately. However, we could send a signal to the
> > archive streamer via privateInfo (e.g., a read_any_wal or
> > skip_wal_check boolean flag) to disable the filtration check until the
> > size is calculated. But that approach isn't very elegant; if the first
> > WAL page we read belongs to a segment we actually want to skip, we
> > would still have to run the filter check and handle the skip/removal
> > logic outside of the streamer (i.e., inside init_archive_reader()).
> > This would result in performing the same filtration check in two
> > different places.
>
> I mean, I don't really buy this logic. If the information added to
> privateInfo is "here's the LSN before which you can remove stuff," and
> it starts out initialized to 0/0, then the read of the first WAL page
> causes no problem at all, because nothing is before 0/0. After it gets
> updated to some non-zero value, the next call to
> astreamer_waldump_content() can handle discarding any data we don't
> need.
>
> IMHO, the best argument for keeping things are is that in some cases,
> that decision might result in a bit of delay in discarding old data,
> but I don't think that really matters. I think all that we care about
> is the peak memory utilization of an operation, and I don't think that
> an explicit signaling system should increase that at all.
>
> That said, I'm certainly willing to consider other ideas about how
> this can work. However, I feel strongly that the logic needs to be not
> only correct, but clear and well-explained. Setting cur_wal to NULL to
> make the astreamer skip without adequate comments doesn't meet that
> standard. Maybe with some better comments it's all right, but frankly
> I'm a bit skeptical. Right now, you're using whether or not cur_wal is
> NULL as a signal to skip data or not skip data. How is that better
> than passing down the LSN and TLI that you want to read next and
> letting the astreamer figure out what to do itself? It's a signaling
> mechanism either way, but it seems a lot easier to figure out whether
> we always keep the LSN and TLI updated properly than to figure out
> whether cur_wal is always NULL at exactly the right times.
>

I am not sure how to calculate the LSN of the very first page it reads
at this point, since we don't have the segment size yet. Furthermore,
the previous implementation using the segment number as the hash key
was incorrect, as extracting the segment number and TLI is impossible
without knowing the segment size, IIUC.

In the attached version, I am using the WAL segment name as the hash
key, which is much more straightforward. I have rewritten
read_archive_wal_page(), and it looks much cleaner than before. The
logic to discard irrelevant WAL files is still within
get_archive_wal_entry. I added an explanation for setting cur_wal to
NULL, which is now handled in the separate function I mentioned
previously.

Kindly have a look at the attached version; let me know if you are
still not happy with the current approach for filtering/discarding
irrelevant WAL segments. It isn't much different from the previous
version, but I have tried to keep it in a separate routine for better
code readability, with comments to make it easier to understand. I
also added a comment for ArchivedWALFile.

I have included your patch (removing WalSegSz) to the attached patch
set as 0001; all my subsequent patches are now bumped by one.

Regards,
Amul

  31. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-01-27T12:23:19Z

    On Sat, Jan 17, 2026 at 2:08 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Jan 2, 2026 at 7:30 AM Amul Sul <sulamul@gmail.com> wrote:
> > Attached is the rebased version, includes some code comment improvements.
>

Here are my replies to the remaining review comments. An updated
version of the patch was posted in the previous email [1].

> Regarding 0001, I don't think that passing the WAL segment size around
> in a global variable is a good practice. It makes it hard to tell
> whether the variable is guaranteed to have been set properly at all
> the places where it is used. Of course, this patch set does not
> introduce that problem, but I think it could avoid making it worse by
> adding WalSegSz to XLogDumpPrivate instead of exporting the global
> variable everywhere. That struct seems to be available everyplace
> where we need the value. Actually, though, I'm inclined to remove the
> file-level global as well. Patch for that attached. I'm not against
> global variables in general, but I'm against global variables that are
> just there to avoid passing the right stuff around via function
> argument lists.
>

Yes, it looks good. Added to the v11 patch set.

> 0002 claims to be just a refactoring but if
> (unlikely(private->endptr_reached)) return -1 is net new code, so I
> object. I realize this contradicts the review comment from Chao Li,
> but the principal that refactoring shouldn't change the logic is more
> important than whatever value we might get out of adding this sanity
> check here -- and I bet if we look into it we'll find that there are
> lots of other places in PostgreSQL that would also need to be changed
> if we wanted such a sanity check everywhere that we have code like
> this. I'd also like to point out that the commit message isn't really
> adequate to understand why this helps achieve the overall goal of the
> patch set.
>

Understood, removed it.

> 0003 looks functionally correct but notice that you've stolen some but
> not all of the tests from the "# various ways of specifying WAL range"
> section without copying the comment or explaining in the commit
> message why you've moved some tests but not others.
>

The excluded tests provide WAL files as input to pg_waldump, which is
incompatible with tar archive input. I have updated the commit message
to reflect this.

> Regarding 0004:
>
> Instead of making ArchivedWAL_HTAB a global variable, can we just
> store a pointer to it in XLogDumpPrivate? (I would change the name to
> something else, something all lower case.) Or maybe the pointer should
> go someplace else, but I'm skeptical of a global variable here for the
> same reasons as 0001.
>

Done.

> I think ArchivedWALFile would be a better name than ArchivedWALEntry,
> and cur_wal_file or cur_file would be a better name than cur_wal.
>

Done.

> I'm extremely happy with the way that the hash table looks and the new
> logic in init_archive_reader() to make sure that we don't need to
> reread the beginning of the WAL but can reuse the data already read.
> That seems like a huge improvement to me.
>
> +               if (recptr >= endPtr)
> +               {
> +                       /* Discard the buffered data */
> +                       resetStringInfo(&entry->buf);
> +                       len = 0;
> +
> +                       /*
> +                        * Push back the partial page data for the
> current page to the
> +                        * buffer, ensuring it remains full page
> available for re-reading
> +                        * if requested.
> +                        */
> +                       if (p > readBuff)
> +                       {
> +                               Assert((count - nbytes) > 0);
> +                               appendBinaryStringInfo(&entry->buf,
> readBuff, count - nbytes);
> +                       }
> +               }
>
> It feels pretty inelegant to me to pull data back into &entry->buf
> from readBuff here. I think what you should do is just do this test
> once, before entering the while (nbytes > 0) loop. Then you'll always
> have p == readBuff, so the nested if statement is no longer required.
> Also, the way you have it, if this triggers for the first loop
> iteration, it will trigger for every subsquent loop iteration, which
> seems wasteful.
>

I'm not sure I follow the logic. How would this trigger a copy from
readBuff back into &entry->buf on every iteration? I  believe that
this is a rare case: it should only happen when we reach the end of
the buffer with less than the requested 8kB available. In that
scenario, we copy the remaining bytes, reset the buffer, and then --
since (p > readBuff)  -- perform the pull-back before invoking the
archive stream for the next segment. Am I missing something in the
loop condition?


I replied to the rest of the review comments previously, but I’m not
sure whether replying in multiple emails is the correct approach.
Perhaps, since the patch is taking a bit longer, I wanted to keep the
conversation moving.

Regards,
Amul

1] http://postgr.es/m/CAAJ_b97mWgAkj7CqdRsof4N_J-gxqjKh79b0Aw3JRo0PoZZCJg@mail.gmail.com

  32. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-01-27T15:55:23Z

    On Tue, Jan 27, 2026 at 7:23 AM Amul Sul <sulamul@gmail.com> wrote:
> Yes, it looks good. Added to the v11 patch set.

I'll respond to the rest of this later, but I went ahead and committed that one.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  33. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-01-27T21:11:26Z

    On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
> In the attached version, I am using the WAL segment name as the hash
> key, which is much more straightforward. I have rewritten
> read_archive_wal_page(), and it looks much cleaner than before. The
> logic to discard irrelevant WAL files is still within
> get_archive_wal_entry. I added an explanation for setting cur_wal to
> NULL, which is now handled in the separate function I mentioned
> previously.
>
> Kindly have a look at the attached version; let me know if you are
> still not happy with the current approach for filtering/discarding
> irrelevant WAL segments. It isn't much different from the previous
> version, but I have tried to keep it in a separate routine for better
> code readability, with comments to make it easier to understand. I
> also added a comment for ArchivedWALFile.

I feel like the division of labor between get_archive_wal_entry() and
read_archive_wal_page() is odd. I noticed this in the last version,
too, and it still seems to be the case. get_archive_wal_entry() first
calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
If it doesn't, it loops until an entry for the requested file shows up
and then returns it. Then control returns to read_archive_wal_page()
which loops some more until we have all the data we need for the
requested file. But it seems odd to me to have two separate loops
here. I think that the first loop is going to call read_archive_file()
until we find the beginning of the file that we care about and then
the second one is going to call read_archive_file() some more until we
have read enough of it to satisfy the request. It feels odd to me to
do it that way, as if we told somebody to first wait until 9 o'clock
and then wait another 30 minutes, instead of just telling them to wait
until 9:30. I realize it's not quite the same thing, because apart
from calling read_archive_file(), the two loops do different things,
but I still think it looks odd.

+ /*
+ * Ignore if the timeline is different or the current segment is not
+ * the desired one.
+ */
+ XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
+ if (privateInfo->timeline != curSegTimeline ||
+ privateInfo->startSegNo > curSegNo ||
+ privateInfo->endSegNo < curSegNo ||
+ segno > curSegNo)
+ {
+ free_archive_wal_entry(entry->fname, privateInfo);
+ continue;
+ }

The comment doesn't match the code. If it did, the test would be
(privateInfo->timeline != curSegTimeline || segno != curSegno). But
instead the segno test is > rather than !=, and the checks against
startSegNo and endSegNo aren't explained at all. I think I understand
why the segno test uses > rather than !=, but it's the point of the
comment to explain things like that, rather than leaving the reader to
guess. And I don't know why we also need to test startSegNo and
endSegNo.

I also wonder what the point is of doing XLogFromFileName() on the
fname provided by the caller and then again on entry->fname. Couldn't
you just compare the strings?

Again, the division of labor is really odd here. It's the job of
astreamer_waldump_content() to skip things that aren't WAL files at
all, but it's the job of get_archive_wal_entry() to skip things that
are WAL files but not the one we want. I disagree with putting those
checks in completely separate parts of the code.

+static bool
+member_is_wal_file(astreamer_waldump *mystreamer, astreamer_member *member,
+    char **fname)
+{
+ int pathlen;
+ char    *filename;
+
+ /* We are only interested in normal files. */
+ if (member->is_directory || member->is_link)
+ return false;
+
+ pathlen = strlen(member->pathname);
+ if (pathlen < XLOG_FNAME_LEN)
+ return false;
+
+ /* WAL files from the top-level or pg_wal directory will be decoded */
+ if (pathlen > XLOG_FNAME_LEN &&
+ strncmp(member->pathname, XLOGDIR, strlen(XLOGDIR)) != 0)
+ return false;
+
+ /* WAL file could be with full path */
+ filename = member->pathname + (pathlen - XLOG_FNAME_LEN);
+ if (!IsXLogFileName(filename))
+ return false;
+
+ *fname = pnstrdup(filename, XLOG_FNAME_LEN);
+
+ return true;
+}

I don't think this code is fully correct. Note that pg_verifybackup's
member_verify_header() prepends "./" and then applies
canonicalize_path(). I think the same considerations apply here.
Consider:

[rhaas pgdata]$ (cd pg_wal; tar -cf ../pg_wal1.tar 00*)
[rhaas pgdata]$ (cd pg_wal; tar -cf ../pg_wal2.tar .)
[rhaas pgdata]$ tar tf pg_wal1.tar
00000001000000000000002C
00000001000000000000002D
00000001000000000000002E
00000001000000000000002F
000000010000000000000030
000000010000000000000031
000000010000000000000032
000000010000000000000033
[rhaas pgdata]$ tar tf pg_wal2.tar
./
./00000001000000000000002C
./00000001000000000000002D
./00000001000000000000002E
./archive_status/
./000000010000000000000031
./000000010000000000000030
./summaries/
./00000001000000000000002F
./000000010000000000000032
./000000010000000000000033

--
Robert Haas
EDB: http://www.enterprisedb.com

  34. Re: pg_waldump: support decoding of WAL inside tarfile

    Euler Taveira <euler@eulerto.com> — 2026-01-28T03:01:39Z

    On Tue, Jan 27, 2026, at 9:06 AM, Amul Sul wrote:
>
> I have included your patch (removing WalSegSz) to the attached patch
> set as 0001; all my subsequent patches are now bumped by one.
>

I started looking at this patch. This is not a complete review.

0002. LGTM.

0003. If this is a refactor, you should move the unlikely logic (if necessary)
to 0005.

0004. You mentioned in the commit message that this patch should be merged into
0005. Reviewing it is a bit hard if you just looked at the patch. However, it
is clear if you use 'git show -w $HASH0004'. I would expect that this patch
contains

+my $tar = $ENV{TAR};
+

...

+       skip "tar command is not available", 3
+         if !defined $tar;

because you add the SKIP. As a refactor I expect it to work independently.

0005.

+           pg_log_error("unnecessary command-line arguments specified with tar archive (first is \"%s\")",
+                        argv[optind]);

Which command-line arguments? This message isn't clear. How about using "option
%s cannot be used together with tar archive".

-   if (waldir != NULL)                                                         
+   if (walpath != NULL)                                                        
    {                                                                           
+       /* validate path points to tar archive */                               
+       if (is_archive_file(walpath, &compression))                             
+       {                                                                       
+           char       *fname = NULL;                                           
+                                                                               
+           split_path(walpath, &waldir, &fname);                               
+                                                                               
+           private.archive_name = fname;                                       
+           is_archive = true;                                                  
+       }                                                                       
        /* validate path points to directory */                                 
-       if (!verify_directory(waldir))                                          
+       else if (!verify_directory(walpath))

Maybe I'm confused about the fact that the function name is is_archive_file()
and there is a variable called is_archive. Couldn't you test
private.archive_name != NULL instead of using is_archive variable?

-       if (!verify_directory(waldir))                                          
+       else if (!verify_directory(walpath))                                    
        {                                                                       
            pg_log_error("could not open directory \"%s\": %m", waldir);        
            goto bad_argument;

This is not a problem of this patch but I just want to point out that it should
be pg_fatal() just like similar code path a few lines below.

+               /* set segno to endsegno for check of --end */                  
+               segno = endsegno;                                               
+           }                                                                   
+                                                                               
+                                                                               
+           if (!XLByteInSeg(private.endptr, segno, WalSegSz) &&                
+               private.endptr != (segno + 1) * WalSegSz)

2 blank lines. Remove one.

+ * archive_waldump.c
+ *     A generic facility for reading WAL data from tar archives via archive
+ *     streamer.

The other tools (pg_basebackup and pg_verifybackup) that also use astreamer API
named this similar file as astreamer_SOMETHING.c. It seems a good idea to
follow the same pattern, no? Maybe astreamer_tar_archive.c or
astreamer_archive.c.

+/* Hash entry structure for holding WAL segment data read from the archive */
+typedef struct ArchivedWALEntry
+{
+   uint32      status;         /* hash status */
+   XLogSegNo   segno;          /* hash key: WAL segment number */
+   TimeLineID  timeline;       /* timeline of this wal file */
+
+   StringInfoData buf;
+   bool        tmpseg_exists;  /* spill file exists? */
+
+   int         total_read;     /* total read of archived WAL segment */
+} ArchivedWALEntry;

s/wal file/WAL file/

What buf is for? It is the only member that doesn't have a description. I think
total_read gives the impression that you've read the file and that's the number
of bytes it got. That's not true; it represents the accumulative length. Maybe
read_len is a good candidate.

+bool
+is_archive_file(const char *fname, pg_compress_algorithm *compression)
+{
+   int         fname_len = strlen(fname);
+   pg_compress_algorithm compress_algo;
+

Why do you need an extra variable here? I think compress_algo is better than
compression. Besides that, it is a good opportunity to move this function to a
common file (common/compression.c?) and use it in other places like
CreateBackupStreamer() -- pg_basebackup.

+   /* Before that we must parse the tar archive. */
+   streamer = astreamer_tar_parser_new(streamer);
+
+   /* Before that we must decompress, if archive is compressed. */
+   if (compression == PG_COMPRESSION_GZIP)

It reads better if you suppress 'Before that we must'. (I think you want to say
'After' instead of 'Before'.)

+   /* Hash table storing WAL entries read from the archive */
+   ArchivedWAL_HTAB = ArchivedWAL_create(16, NULL);

Any reason for 16? Comment says nothing about it.

+       if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
+           pg_fatal("could not find WAL in \"%s\" archive",
+                    privateInfo->archive_name);

Could we have a better message here? The read_archive_file() is already testing
the error cases and you are testing the EOF case. I would use 'archive \"%s\"'
or even 'archive file \"%s\".

+   /* Needed WAL yet to be decoded from archive, do the same */
+   while (1)
+   {
+       entry = privateInfo->cur_wal;

This comment is not clear. Could you rephrase it?

+       /*
+        * XXX: If the segment being read not the requested one, the data must
+        * be buffered, as we currently lack the mechanism to write it to a
+        * temporary file. This is a known limitation that will be fixed in the
+        * next patch, as the buffer could grow up to the full WAL segment
+        * size.
+        */
+       if (segno > entry->segno)
+           continue;
+
+       /* WAL segments must be archived in order */
+       pg_log_error("WAL files are not archived in sequential order");
+       pg_log_error_detail("Expecting segment number " UINT64_FORMAT " but found " UINT64_FORMAT ".",
+                           segno, entry->segno);

Can it enforce a specific order? tar follows an arbitrary order in which the
files is returned by the filesystem. You've been debating a solution to buffer
the WAL contents using memory or spilled files. If it always create the tar in
an alphabetical order, you can reduce the scope of this patch. (Didn't look
what challenges are expected to use a sorted list to generate the tar file.)

-  dependencies: [frontend_code, lz4, zstd],
+  dependencies: [frontend_code, lz4, zstd, libpq],

Put libpq after frontend_code.

+
+   /* Fields required to read WAL from archive */
+   char       *archive_name;   /* Tar archive name */
+   int         archive_fd;     /* File descriptor for the open tar file */
+
+   astreamer  *archive_streamer;
+
+   /* What the archive streamer is currently reading */
+   struct ArchivedWALEntry *cur_wal;
+
+   /*
+    * Although these values can be easily derived from startptr and endptr,
+    * doing so repeatedly for each archived member would be inefficient, as
+    * it would involve recalculating and filtering out irrelevant WAL
+    * segments.
+    */
+   XLogSegNo   startSegNo;
+   XLogSegNo   endSegNo;

It is a matter of style but consistency is good. Per current style snake case
is preferred instead of CamelCase for these last members.

-       @lines = test_pg_waldump('--path' => $path);
+       my @lines;
+       @lines = test_pg_waldump($path);

my @lines = test_pg_waldump($path);

+/* Forward declaration */
+struct ArchivedWALFile;

Why don't you move ArchivedWALFile to pg_waldump.h? pg_waldump.h is included
into archive_waldump.c.

0006. As I said before I would avoid increasing the size of this patch if an
ordered tar file is viable. Didn't look in deep this patch but I have a few
suggestions:

* I don't like tmpfile_exists as a member name. A suggestion is 'spilled'.

+#ifndef WIN32
+   if (chmod(fpath, pg_file_create_mode))
+       pg_fatal("could not set permissions on file \"%s\": %m",
+                fpath);
+#endif

s/on/of/
My suggestion is to use the same sentence in initdb.

  could not change permissions of \"%s\": %m

+   pg_log_debug("temporarily exporting file \"%s\"", fpath);                   

spilling to temporary file \"%s\"

0007. LGTM.

0008. I'm concerned about this patch. It is breaking backward compatibility if
you are using a long option (--wal-directory). Your proposal is a generic word
that represents both cases (file and directory). I agree. However, I wouldn't
remove --wal-directory from the tool. Instead, I would keep it with the same
short option ('w') but add a sentence saying this long option is deprecated and
will be removed in the future or even remove any traces of this long option
from the help and documentation but silently accept the old long option. I
prefer the latter because it is not a required argument so a deprecation
warning is not necessary IMO.

0009. LGTM.


-- 
Euler Taveira
EDB   https://www.enterprisedb.com/

  35. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-01-28T13:03:02Z

    On Tue, Jan 27, 2026 at 10:02 PM Euler Taveira <euler@eulerto.com> wrote:
> + * archive_waldump.c
> + *     A generic facility for reading WAL data from tar archives via archive
> + *     streamer.
>
> The other tools (pg_basebackup and pg_verifybackup) that also use astreamer API
> named this similar file as astreamer_SOMETHING.c. It seems a good idea to
> follow the same pattern, no? Maybe astreamer_tar_archive.c or
> astreamer_archive.c.

There shouldn't be anything specific to tar files in here, and
astreamer_archive would be meaningless, since the "a" in "astreamer"
stands for archive. What this file is is an archive streamer specific
to pg_waldump, hence the name.

> Can it enforce a specific order? tar follows an arbitrary order in which the
> files is returned by the filesystem. You've been debating a solution to buffer
> the WAL contents using memory or spilled files. If it always create the tar in
> an alphabetical order, you can reduce the scope of this patch. (Didn't look
> what challenges are expected to use a sorted list to generate the tar file.)

It's posible to create a tar file in a specific order by specifying
command-line arguments to tar in the order you want the tar file to be
built. But I think the real thing here is that this limitation is
lifted by the following patch. Whether it's worth splitting it apart
into two patches this way is debatable. As I have pointed out in my
previous reviews, the split hasn't been done very cleanly.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  36. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-02-04T13:09:10Z

    On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
> > In the attached version, I am using the WAL segment name as the hash
> > key, which is much more straightforward. I have rewritten
> > read_archive_wal_page(), and it looks much cleaner than before. The
> > logic to discard irrelevant WAL files is still within
> > get_archive_wal_entry. I added an explanation for setting cur_wal to
> > NULL, which is now handled in the separate function I mentioned
> > previously.
> >
> > Kindly have a look at the attached version; let me know if you are
> > still not happy with the current approach for filtering/discarding
> > irrelevant WAL segments. It isn't much different from the previous
> > version, but I have tried to keep it in a separate routine for better
> > code readability, with comments to make it easier to understand. I
> > also added a comment for ArchivedWALFile.
>
> I feel like the division of labor between get_archive_wal_entry() and
> read_archive_wal_page() is odd. I noticed this in the last version,
> too, and it still seems to be the case. get_archive_wal_entry() first
> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> If it doesn't, it loops until an entry for the requested file shows up
> and then returns it. Then control returns to read_archive_wal_page()
> which loops some more until we have all the data we need for the
> requested file. But it seems odd to me to have two separate loops
> here. I think that the first loop is going to call read_archive_file()
> until we find the beginning of the file that we care about and then
> the second one is going to call read_archive_file() some more until we
> have read enough of it to satisfy the request. It feels odd to me to
> do it that way, as if we told somebody to first wait until 9 o'clock
> and then wait another 30 minutes, instead of just telling them to wait
> until 9:30. I realize it's not quite the same thing, because apart
> from calling read_archive_file(), the two loops do different things,
> but I still think it looks odd.
>
> + /*
> + * Ignore if the timeline is different or the current segment is not
> + * the desired one.
> + */
> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> + if (privateInfo->timeline != curSegTimeline ||
> + privateInfo->startSegNo > curSegNo ||
> + privateInfo->endSegNo < curSegNo ||
> + segno > curSegNo)
> + {
> + free_archive_wal_entry(entry->fname, privateInfo);
> + continue;
> + }
>
> The comment doesn't match the code. If it did, the test would be
> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> instead the segno test is > rather than !=, and the checks against
> startSegNo and endSegNo aren't explained at all. I think I understand
> why the segno test uses > rather than !=, but it's the point of the
> comment to explain things like that, rather than leaving the reader to
> guess. And I don't know why we also need to test startSegNo and
> endSegNo.
>
> I also wonder what the point is of doing XLogFromFileName() on the
> fname provided by the caller and then again on entry->fname. Couldn't
> you just compare the strings?
>
> Again, the division of labor is really odd here. It's the job of
> astreamer_waldump_content() to skip things that aren't WAL files at
> all, but it's the job of get_archive_wal_entry() to skip things that
> are WAL files but not the one we want. I disagree with putting those
> checks in completely separate parts of the code.
>

Keeping the timeline and segment start-end range checks inside the
archive streamer creates a circular dependency that cannot be resolved
without a 'dirty hack'. We must read the first available WAL file page
to determine the wal_segment_size before it can calculate the target
segment range. Moving the checks inside the streamer would make it
impossible to process that initial file, as the necessary filtering
parameters --  would still be unknown which would need to be skipped
for the first read somehow. What if later we realized that the first
WAL file which was allowed to be streamed by skipping that check is
irrelevant and doesn't fall under the start-end segment range?

> +static bool
> +member_is_wal_file(astreamer_waldump *mystreamer, astreamer_member *member,
> +    char **fname)
> +{
> + int pathlen;
> + char    *filename;
> +
> + /* We are only interested in normal files. */
> + if (member->is_directory || member->is_link)
> + return false;
> +
> + pathlen = strlen(member->pathname);
> + if (pathlen < XLOG_FNAME_LEN)
> + return false;
> +
> + /* WAL files from the top-level or pg_wal directory will be decoded */
> + if (pathlen > XLOG_FNAME_LEN &&
> + strncmp(member->pathname, XLOGDIR, strlen(XLOGDIR)) != 0)
> + return false;
> +
> + /* WAL file could be with full path */
> + filename = member->pathname + (pathlen - XLOG_FNAME_LEN);
> + if (!IsXLogFileName(filename))
> + return false;
> +
> + *fname = pnstrdup(filename, XLOG_FNAME_LEN);
> +
> + return true;
> +}
>
> I don't think this code is fully correct. Note that pg_verifybackup's
> member_verify_header() prepends "./" and then applies
> canonicalize_path(). I think the same considerations apply here.
> Consider:
>

Understood, will fix that.

Regards,
Amul

  37. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-02-04T13:52:41Z

    On Wed, Jan 28, 2026 at 8:32 AM Euler Taveira <euler@eulerto.com> wrote:
>
> On Tue, Jan 27, 2026, at 9:06 AM, Amul Sul wrote:
> >

Hi Euler,

Thanks for looking into this, and apologies for the delayed response
due to some unavoidable circumstances.

The updated version of the patch could take a bit more time, as I am
trying to fix and improve the code based on the concerns Robert raised
in his previous post.

> > I have included your patch (removing WalSegSz) to the attached patch
> > set as 0001; all my subsequent patches are now bumped by one.
> >
> 0003. If this is a refactor, you should move the unlikely logic (if necessary)
> to 0005.
>

Yes, that can be done. I have kept them separate for now to make the
review easier. I can merge them once the patch is ready for the
committer stage, unless the committer prefers to keep them as separate
commit.

> 0004. You mentioned in the commit message that this patch should be merged into
> 0005. Reviewing it is a bit hard if you just looked at the patch. However, it
> is clear if you use 'git show -w $HASH0004'. I would expect that this patch
> contains
>
> +my $tar = $ENV{TAR};
> +
>
> ...
>
> +       skip "tar command is not available", 3
> +         if !defined $tar;
>
> because you add the SKIP. As a refactor I expect it to work independently.
>

Okay, will do it in the next updated version.

> 0005.
>
> +           pg_log_error("unnecessary command-line arguments specified with tar archive (first is \"%s\")",
> +                        argv[optind]);
>
> Which command-line arguments? This message isn't clear. How about using "option
> %s cannot be used together with tar archive".
>

In the error message, we simply print only the first one -- we do have
similar error messages that use "command-line arguments".

> -   if (waldir != NULL)
> +   if (walpath != NULL)
>     {
> +       /* validate path points to tar archive */
> +       if (is_archive_file(walpath, &compression))
> +       {
> +           char       *fname = NULL;
> +
> +           split_path(walpath, &waldir, &fname);
> +
> +           private.archive_name = fname;
> +           is_archive = true;
> +       }
>         /* validate path points to directory */
> -       if (!verify_directory(waldir))
> +       else if (!verify_directory(walpath))
>
> Maybe I'm confused about the fact that the function name is is_archive_file()
> and there is a variable called is_archive. Couldn't you test
> private.archive_name != NULL instead of using is_archive variable?
>

Yes, can do that. I hope that doesn't confuse anyone.


> -       if (!verify_directory(waldir))
> +       else if (!verify_directory(walpath))
>         {
>             pg_log_error("could not open directory \"%s\": %m", waldir);
>             goto bad_argument;
>
> This is not a problem of this patch but I just want to point out that it should
> be pg_fatal() just like similar code path a few lines below.
>
> +               /* set segno to endsegno for check of --end */
> +               segno = endsegno;
> +           }
> +
> +
> +           if (!XLByteInSeg(private.endptr, segno, WalSegSz) &&
> +               private.endptr != (segno + 1) * WalSegSz)
>
> 2 blank lines. Remove one.
>

That's pre-existing; I'm not sure if I should change this.

> + * archive_waldump.c
> + *     A generic facility for reading WAL data from tar archives via archive
> + *     streamer.
>
> The other tools (pg_basebackup and pg_verifybackup) that also use astreamer API
> named this similar file as astreamer_SOMETHING.c. It seems a good idea to
> follow the same pattern, no? Maybe astreamer_tar_archive.c or
> astreamer_archive.c.
>

Robert responded to this -- thanks, Robert.

> +/* Hash entry structure for holding WAL segment data read from the archive */
> +typedef struct ArchivedWALEntry
> +{
> +   uint32      status;         /* hash status */
> +   XLogSegNo   segno;          /* hash key: WAL segment number */
> +   TimeLineID  timeline;       /* timeline of this wal file */
> +
> +   StringInfoData buf;
> +   bool        tmpseg_exists;  /* spill file exists? */
> +
> +   int         total_read;     /* total read of archived WAL segment */
> +} ArchivedWALEntry;
>
> s/wal file/WAL file/
>

Okay.

> What buf is for? It is the only member that doesn't have a description. I think
> total_read gives the impression that you've read the file and that's the number
> of bytes it got. That's not true; it represents the accumulative length. Maybe
> read_len is a good candidate.
>

Okay.

> +bool
> +is_archive_file(const char *fname, pg_compress_algorithm *compression)
> +{
> +   int         fname_len = strlen(fname);
> +   pg_compress_algorithm compress_algo;
> +
>
> Why do you need an extra variable here? I think compress_algo is better than
> compression.

Well, I follow the practice of updating the output variable at the end
to avoid any garbage assignment once all operations are successful. If
there are strong objections to this, I am happy to change it.

> Besides that, it is a good opportunity to move this function to a
> common file (common/compression.c?) and use it in other places like
> CreateBackupStreamer() -- pg_basebackup.
>

Yes, it would be a good idea to make it common, let me think about this.

> +   /* :. */
> +   streamer = astreamer_tar_parser_new(streamer);
> +
> +   /* Before that we must decompress, if archive is compressed. */
> +   if (compression == PG_COMPRESSION_GZIP)
>
> It reads better if you suppress 'Before that we must'. (I think you want to say
> 'After' instead of 'Before'.)
>

I meant to say 'Before'. This is the archive streamer stack, so the
latter one is executed before it.

> +   /* Hash table storing WAL entries read from the archive */
> +   ArchivedWAL_HTAB = ArchivedWAL_create(16, NULL);
>
> Any reason for 16? Comment says nothing about it.
>

It's an arbitrary choice; I will update the comment.

> +       if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
> +           pg_fatal("could not find WAL in \"%s\" archive",
> +                    privateInfo->archive_name);
>
> Could we have a better message here? The read_archive_file() is already testing
> the error cases and you are testing the EOF case. I would use 'archive \"%s\"'
> or even 'archive file \"%s\".
>

Okay.

> +   /* Needed WAL yet to be decoded from archive, do the same */
> +   while (1)
> +   {
> +       entry = privateInfo->cur_wal;
>
> This comment is not clear. Could you rephrase it?
>

Okay.

> -  dependencies: [frontend_code, lz4, zstd],
> +  dependencies: [frontend_code, lz4, zstd, libpq],
>
> Put libpq after frontend_code.
>

Okay.

> +
> +   /* Fields required to read WAL from archive */
> +   char       *archive_name;   /* Tar archive name */
> +   int         archive_fd;     /* File descriptor for the open tar file */
> +
> +   astreamer  *archive_streamer;
> +
> +   /* What the archive streamer is currently reading */
> +   struct ArchivedWALEntry *cur_wal;
> +
> +   /*
> +    * Although these values can be easily derived from startptr and endptr,
> +    * doing so repeatedly for each archived member would be inefficient, as
> +    * it would involve recalculating and filtering out irrelevant WAL
> +    * segments.
> +    */
> +   XLogSegNo   startSegNo;
> +   XLogSegNo   endSegNo;
>
> It is a matter of style but consistency is good. Per current style snake case
> is preferred instead of CamelCase for these last members.
>

Okay.

> -       @lines = test_pg_waldump('--path' => $path);
> +       my @lines;
> +       @lines = test_pg_waldump($path);
>
> my @lines = test_pg_waldump($path);
>

Okay.

> +/* Forward declaration */
> +struct ArchivedWALFile;
>
> Why don't you move ArchivedWALFile to pg_waldump.h? pg_waldump.h is included
> into archive_waldump.c.
>

The full structure is needed in archive_waldump.c only, so I added a
forward declaration to declare the pointer inside XLogDumpPrivate. I
believe we follow such practices elsewhere in the codebase as well.

> 0006. As I said before I would avoid increasing the size of this patch if an
> ordered tar file is viable. Didn't look in deep this patch but I have a few
> suggestions:
>
> * I don't like tmpfile_exists as a member name. A suggestion is 'spilled'.
>

Okay.

> +#ifndef WIN32
> +   if (chmod(fpath, pg_file_create_mode))
> +       pg_fatal("could not set permissions on file \"%s\": %m",
> +                fpath);
> +#endif
>
> s/on/of/
> My suggestion is to use the same sentence in initdb.
>

I think 'on' seems to be much more relevant than 'of,' and we do have
'could not set permissions on' error messages in other places as wel

>   could not change permissions of \"%s\": %m
>
> +   pg_log_debug("temporarily exporting file \"%s\"", fpath);
>
> spilling to temporary file \"%s\"
>

Okay.

> 0008. I'm concerned about this patch. It is breaking backward compatibility if
> you are using a long option (--wal-directory). Your proposal is a generic word
> that represents both cases (file and directory). I agree. However, I wouldn't
> remove --wal-directory from the tool. Instead, I would keep it with the same
> short option ('w') but add a sentence saying this long option is deprecated and
> will be removed in the future or even remove any traces of this long option
> from the help and documentation but silently accept the old long option. I
> prefer the latter because it is not a required argument so a deprecation
> warning is not necessary IMO.
>

Yeah, that was discussed with Robert offline and we believe that it is
better to make it more generalized; since we can now use the same
option to accept both wal-directory and wal-archived. pg_waldump has
much more generic options for the same, such as -- path=PATH.

Regards,
Amul

  38. Re: pg_waldump: support decoding of WAL inside tarfile

    Robert Haas <robertmhaas@gmail.com> — 2026-02-07T13:04:32Z

    On Wed, Feb 4, 2026 at 8:53 AM Amul Sul <sulamul@gmail.com> wrote:
> > 0008. I'm concerned about this patch. It is breaking backward compatibility if
> > you are using a long option (--wal-directory). Your proposal is a generic word
> > that represents both cases (file and directory). I agree. However, I wouldn't
> > remove --wal-directory from the tool. Instead, I would keep it with the same
> > short option ('w') but add a sentence saying this long option is deprecated and
> > will be removed in the future or even remove any traces of this long option
> > from the help and documentation but silently accept the old long option. I
> > prefer the latter because it is not a required argument so a deprecation
> > warning is not necessary IMO.
>
> Yeah, that was discussed with Robert offline and we believe that it is
> better to make it more generalized; since we can now use the same
> option to accept both wal-directory and wal-archived. pg_waldump has
> much more generic options for the same, such as -- path=PATH.

Of course, the fact that we discussed it doesn't mean that the issue
is completely settled. However, I don't think there would be general
support from other people on the project for the idea of getting rid
of the long option entirely, or even just the documentation for it. We
have long options for almost all short options these days, and I agree
with that as a general practice. Sometimes we have ONLY a long option,
but we very rarely have ONLY a short option, which is good because
sometimes we have tools with too many options for it to be viable to
give everything a short option, and the long options really help to
make things more self-documenting.

What I would consider a more viable option is to not do the rename and
leave this as --wal-directory even though the argument could really be
a directory or a file. That would avoid the backward compatibility
break that is troubling Euler. And, you could argue that a tar file is
enough like a directory that it won't really cause much confusion.
Personally, I favor renaming it. I think the number of people using
pg_waldump is fairly small, and the number of those people who are
using the long form of the option is very small. Hence, I don't
believe think the rename will inconvenience many users, and I think it
will improve clarity for future users. But, Euler (or someone else)
might take the opposite viewpoint.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  39. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-02-10T09:36:12Z

    On Wed, Feb 4, 2026 at 6:39 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
> > > In the attached version, I am using the WAL segment name as the hash
> > > key, which is much more straightforward. I have rewritten
> > > read_archive_wal_page(), and it looks much cleaner than before. The
> > > logic to discard irrelevant WAL files is still within
> > > get_archive_wal_entry. I added an explanation for setting cur_wal to
> > > NULL, which is now handled in the separate function I mentioned
> > > previously.
> > >
> > > Kindly have a look at the attached version; let me know if you are
> > > still not happy with the current approach for filtering/discarding
> > > irrelevant WAL segments. It isn't much different from the previous
> > > version, but I have tried to keep it in a separate routine for better
> > > code readability, with comments to make it easier to understand. I
> > > also added a comment for ArchivedWALFile.
> >
> > I feel like the division of labor between get_archive_wal_entry() and
> > read_archive_wal_page() is odd. I noticed this in the last version,
> > too, and it still seems to be the case. get_archive_wal_entry() first
> > calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> > If it doesn't, it loops until an entry for the requested file shows up
> > and then returns it. Then control returns to read_archive_wal_page()
> > which loops some more until we have all the data we need for the
> > requested file. But it seems odd to me to have two separate loops
> > here. I think that the first loop is going to call read_archive_file()
> > until we find the beginning of the file that we care about and then
> > the second one is going to call read_archive_file() some more until we
> > have read enough of it to satisfy the request. It feels odd to me to
> > do it that way, as if we told somebody to first wait until 9 o'clock
> > and then wait another 30 minutes, instead of just telling them to wait
> > until 9:30. I realize it's not quite the same thing, because apart
> > from calling read_archive_file(), the two loops do different things,
> > but I still think it looks odd.
> >
> > + /*
> > + * Ignore if the timeline is different or the current segment is not
> > + * the desired one.
> > + */
> > + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> > + if (privateInfo->timeline != curSegTimeline ||
> > + privateInfo->startSegNo > curSegNo ||
> > + privateInfo->endSegNo < curSegNo ||
> > + segno > curSegNo)
> > + {
> > + free_archive_wal_entry(entry->fname, privateInfo);
> > + continue;
> > + }
> >
> > The comment doesn't match the code. If it did, the test would be
> > (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> > instead the segno test is > rather than !=, and the checks against
> > startSegNo and endSegNo aren't explained at all. I think I understand
> > why the segno test uses > rather than !=, but it's the point of the
> > comment to explain things like that, rather than leaving the reader to
> > guess. And I don't know why we also need to test startSegNo and
> > endSegNo.
> >
> > I also wonder what the point is of doing XLogFromFileName() on the
> > fname provided by the caller and then again on entry->fname. Couldn't
> > you just compare the strings?
> >
> > Again, the division of labor is really odd here. It's the job of
> > astreamer_waldump_content() to skip things that aren't WAL files at
> > all, but it's the job of get_archive_wal_entry() to skip things that
> > are WAL files but not the one we want. I disagree with putting those
> > checks in completely separate parts of the code.
> >
>
> Keeping the timeline and segment start-end range checks inside the
> archive streamer creates a circular dependency that cannot be resolved
> without a 'dirty hack'. We must read the first available WAL file page
> to determine the wal_segment_size before it can calculate the target
> segment range. Moving the checks inside the streamer would make it
> impossible to process that initial file, as the necessary filtering
> parameters --  would still be unknown which would need to be skipped
> for the first read somehow. What if later we realized that the first
> WAL file which was allowed to be streamed by skipping that check is
> irrelevant and doesn't fall under the start-end segment range?
>

Please have a look at the attached version, specifically patch 0005.
In astreamer_waldump_content(), I have moved the WAL file filtration
check from get_archive_wal_entry(). This check will be skipped during
the initial read in init_archive_reader(), which instead performs it
explicitly once it determines the WAL segment size and the start/end
segments.

To access the WAL segment size inside astreamer_waldump_content(), I
have moved the WAL segment size variable into the XLogDumpPrivate
structure in the separate 0004 patch.

Regards,
Amul

  40. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-02-18T06:58:27Z

    On Tue, Feb 10, 2026 at 3:06 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > >
> > > On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
> > > > In the attached version, I am using the WAL segment name as the hash
> > > > key, which is much more straightforward. I have rewritten
> > > > read_archive_wal_page(), and it looks much cleaner than before. The
> > > > logic to discard irrelevant WAL files is still within
> > > > get_archive_wal_entry. I added an explanation for setting cur_wal to
> > > > NULL, which is now handled in the separate function I mentioned
> > > > previously.
> > > >
> > > > Kindly have a look at the attached version; let me know if you are
> > > > still not happy with the current approach for filtering/discarding
> > > > irrelevant WAL segments. It isn't much different from the previous
> > > > version, but I have tried to keep it in a separate routine for better
> > > > code readability, with comments to make it easier to understand. I
> > > > also added a comment for ArchivedWALFile.
> > >
> > > I feel like the division of labor between get_archive_wal_entry() and
> > > read_archive_wal_page() is odd. I noticed this in the last version,
> > > too, and it still seems to be the case. get_archive_wal_entry() first
> > > calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> > > If it doesn't, it loops until an entry for the requested file shows up
> > > and then returns it. Then control returns to read_archive_wal_page()
> > > which loops some more until we have all the data we need for the
> > > requested file. But it seems odd to me to have two separate loops
> > > here. I think that the first loop is going to call read_archive_file()
> > > until we find the beginning of the file that we care about and then
> > > the second one is going to call read_archive_file() some more until we
> > > have read enough of it to satisfy the request. It feels odd to me to
> > > do it that way, as if we told somebody to first wait until 9 o'clock
> > > and then wait another 30 minutes, instead of just telling them to wait
> > > until 9:30. I realize it's not quite the same thing, because apart
> > > from calling read_archive_file(), the two loops do different things,
> > > but I still think it looks odd.
> > >
> > > + /*
> > > + * Ignore if the timeline is different or the current segment is not
> > > + * the desired one.
> > > + */
> > > + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> > > + if (privateInfo->timeline != curSegTimeline ||
> > > + privateInfo->startSegNo > curSegNo ||
> > > + privateInfo->endSegNo < curSegNo ||
> > > + segno > curSegNo)
> > > + {
> > > + free_archive_wal_entry(entry->fname, privateInfo);
> > > + continue;
> > > + }
> > >
> > > The comment doesn't match the code. If it did, the test would be
> > > (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> > > instead the segno test is > rather than !=, and the checks against
> > > startSegNo and endSegNo aren't explained at all. I think I understand
> > > why the segno test uses > rather than !=, but it's the point of the
> > > comment to explain things like that, rather than leaving the reader to
> > > guess. And I don't know why we also need to test startSegNo and
> > > endSegNo.
> > >
> > > I also wonder what the point is of doing XLogFromFileName() on the
> > > fname provided by the caller and then again on entry->fname. Couldn't
> > > you just compare the strings?
> > >
> > > Again, the division of labor is really odd here. It's the job of
> > > astreamer_waldump_content() to skip things that aren't WAL files at
> > > all, but it's the job of get_archive_wal_entry() to skip things that
> > > are WAL files but not the one we want. I disagree with putting those
> > > checks in completely separate parts of the code.
> > >
> >
> > Keeping the timeline and segment start-end range checks inside the
> > archive streamer creates a circular dependency that cannot be resolved
> > without a 'dirty hack'. We must read the first available WAL file page
> > to determine the wal_segment_size before it can calculate the target
> > segment range. Moving the checks inside the streamer would make it
> > impossible to process that initial file, as the necessary filtering
> > parameters --  would still be unknown which would need to be skipped
> > for the first read somehow. What if later we realized that the first
> > WAL file which was allowed to be streamed by skipping that check is
> > irrelevant and doesn't fall under the start-end segment range?
> >
>
> Please have a look at the attached version, specifically patch 0005.
> In astreamer_waldump_content(), I have moved the WAL file filtration
> check from get_archive_wal_entry(). This check will be skipped during
> the initial read in init_archive_reader(), which instead performs it
> explicitly once it determines the WAL segment size and the start/end
> segments.
>
> To access the WAL segment size inside astreamer_waldump_content(), I
> have moved the WAL segment size variable into the XLogDumpPrivate
> structure in the separate 0004 patch.

Attached is an updated version including the aforesaid changes. It
includes a new refactoring patch (0001) that moves the logic for
identifying tar archives and their compression types from
pg_basebackup and pg_verifybackup into a separate-reusable function,
per a suggestion from Euler [1].  Additionally, I have added a test
for the contrecord decoding to the main patch (now 0006).

1] http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com

Regards,
Amul

  41. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-02T13:00:24Z

    On Wed, Feb 18, 2026 at 12:28 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Wed, Feb 4, 2026 at 6:39 PM Amul Sul <sulamul@gmail.com> wrote:
> > >
> > > On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > > >
> > > > On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
> > > > > In the attached version, I am using the WAL segment name as the hash
> > > > > key, which is much more straightforward. I have rewritten
> > > > > read_archive_wal_page(), and it looks much cleaner than before. The
> > > > > logic to discard irrelevant WAL files is still within
> > > > > get_archive_wal_entry. I added an explanation for setting cur_wal to
> > > > > NULL, which is now handled in the separate function I mentioned
> > > > > previously.
> > > > >
> > > > > Kindly have a look at the attached version; let me know if you are
> > > > > still not happy with the current approach for filtering/discarding
> > > > > irrelevant WAL segments. It isn't much different from the previous
> > > > > version, but I have tried to keep it in a separate routine for better
> > > > > code readability, with comments to make it easier to understand. I
> > > > > also added a comment for ArchivedWALFile.
> > > >
> > > > I feel like the division of labor between get_archive_wal_entry() and
> > > > read_archive_wal_page() is odd. I noticed this in the last version,
> > > > too, and it still seems to be the case. get_archive_wal_entry() first
> > > > calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> > > > If it doesn't, it loops until an entry for the requested file shows up
> > > > and then returns it. Then control returns to read_archive_wal_page()
> > > > which loops some more until we have all the data we need for the
> > > > requested file. But it seems odd to me to have two separate loops
> > > > here. I think that the first loop is going to call read_archive_file()
> > > > until we find the beginning of the file that we care about and then
> > > > the second one is going to call read_archive_file() some more until we
> > > > have read enough of it to satisfy the request. It feels odd to me to
> > > > do it that way, as if we told somebody to first wait until 9 o'clock
> > > > and then wait another 30 minutes, instead of just telling them to wait
> > > > until 9:30. I realize it's not quite the same thing, because apart
> > > > from calling read_archive_file(), the two loops do different things,
> > > > but I still think it looks odd.
> > > >
> > > > + /*
> > > > + * Ignore if the timeline is different or the current segment is not
> > > > + * the desired one.
> > > > + */
> > > > + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> > > > + if (privateInfo->timeline != curSegTimeline ||
> > > > + privateInfo->startSegNo > curSegNo ||
> > > > + privateInfo->endSegNo < curSegNo ||
> > > > + segno > curSegNo)
> > > > + {
> > > > + free_archive_wal_entry(entry->fname, privateInfo);
> > > > + continue;
> > > > + }
> > > >
> > > > The comment doesn't match the code. If it did, the test would be
> > > > (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> > > > instead the segno test is > rather than !=, and the checks against
> > > > startSegNo and endSegNo aren't explained at all. I think I understand
> > > > why the segno test uses > rather than !=, but it's the point of the
> > > > comment to explain things like that, rather than leaving the reader to
> > > > guess. And I don't know why we also need to test startSegNo and
> > > > endSegNo.
> > > >
> > > > I also wonder what the point is of doing XLogFromFileName() on the
> > > > fname provided by the caller and then again on entry->fname. Couldn't
> > > > you just compare the strings?
> > > >
> > > > Again, the division of labor is really odd here. It's the job of
> > > > astreamer_waldump_content() to skip things that aren't WAL files at
> > > > all, but it's the job of get_archive_wal_entry() to skip things that
> > > > are WAL files but not the one we want. I disagree with putting those
> > > > checks in completely separate parts of the code.
> > > >
> > >
> > > Keeping the timeline and segment start-end range checks inside the
> > > archive streamer creates a circular dependency that cannot be resolved
> > > without a 'dirty hack'. We must read the first available WAL file page
> > > to determine the wal_segment_size before it can calculate the target
> > > segment range. Moving the checks inside the streamer would make it
> > > impossible to process that initial file, as the necessary filtering
> > > parameters --  would still be unknown which would need to be skipped
> > > for the first read somehow. What if later we realized that the first
> > > WAL file which was allowed to be streamed by skipping that check is
> > > irrelevant and doesn't fall under the start-end segment range?
> > >
> >
> > Please have a look at the attached version, specifically patch 0005.
> > In astreamer_waldump_content(), I have moved the WAL file filtration
> > check from get_archive_wal_entry(). This check will be skipped during
> > the initial read in init_archive_reader(), which instead performs it
> > explicitly once it determines the WAL segment size and the start/end
> > segments.
> >
> > To access the WAL segment size inside astreamer_waldump_content(), I
> > have moved the WAL segment size variable into the XLogDumpPrivate
> > structure in the separate 0004 patch.
>
> Attached is an updated version including the aforesaid changes. It
> includes a new refactoring patch (0001) that moves the logic for
> identifying tar archives and their compression types from
> pg_basebackup and pg_verifybackup into a separate-reusable function,
> per a suggestion from Euler [1].  Additionally, I have added a test
> for the contrecord decoding to the main patch (now 0006).
>
> 1] http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
>

Rebased against the latest master, fixed typos in code comments, and
replaced palloc0 with palloc0_object.

Regards,
Amul

  42. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-04T00:37:48Z

    On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul <sulamul@gmail.com> wrote:
>> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul <sulamul@gmail.com> wrote:
>>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul <sulamul@gmail.com> wrote:
>>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <robertmhaas@gmail.com> wrote:
>>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
>>>>>> In the attached version, I am using the WAL segment name as the hash
>>>>>> key, which is much more straightforward. I have rewritten
>>>>>> read_archive_wal_page(), and it looks much cleaner than before. The
>>>>>> logic to discard irrelevant WAL files is still within
>>>>>> get_archive_wal_entry. I added an explanation for setting cur_wal to
>>>>>> NULL, which is now handled in the separate function I mentioned
>>>>>> previously.
>>>>>>
>>>>>> Kindly have a look at the attached version; let me know if you are
>>>>>> still not happy with the current approach for filtering/discarding
>>>>>> irrelevant WAL segments. It isn't much different from the previous
>>>>>> version, but I have tried to keep it in a separate routine for better
>>>>>> code readability, with comments to make it easier to understand. I
>>>>>> also added a comment for ArchivedWALFile.
>>>>> I feel like the division of labor between get_archive_wal_entry() and
>>>>> read_archive_wal_page() is odd. I noticed this in the last version,
>>>>> too, and it still seems to be the case. get_archive_wal_entry() first
>>>>> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
>>>>> If it doesn't, it loops until an entry for the requested file shows up
>>>>> and then returns it. Then control returns to read_archive_wal_page()
>>>>> which loops some more until we have all the data we need for the
>>>>> requested file. But it seems odd to me to have two separate loops
>>>>> here. I think that the first loop is going to call read_archive_file()
>>>>> until we find the beginning of the file that we care about and then
>>>>> the second one is going to call read_archive_file() some more until we
>>>>> have read enough of it to satisfy the request. It feels odd to me to
>>>>> do it that way, as if we told somebody to first wait until 9 o'clock
>>>>> and then wait another 30 minutes, instead of just telling them to wait
>>>>> until 9:30. I realize it's not quite the same thing, because apart
>>>>> from calling read_archive_file(), the two loops do different things,
>>>>> but I still think it looks odd.
>>>>>
>>>>> + /*
>>>>> + * Ignore if the timeline is different or the current segment is not
>>>>> + * the desired one.
>>>>> + */
>>>>> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
>>>>> + if (privateInfo->timeline != curSegTimeline ||
>>>>> + privateInfo->startSegNo > curSegNo ||
>>>>> + privateInfo->endSegNo < curSegNo ||
>>>>> + segno > curSegNo)
>>>>> + {
>>>>> + free_archive_wal_entry(entry->fname, privateInfo);
>>>>> + continue;
>>>>> + }
>>>>>
>>>>> The comment doesn't match the code. If it did, the test would be
>>>>> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
>>>>> instead the segno test is > rather than !=, and the checks against
>>>>> startSegNo and endSegNo aren't explained at all. I think I understand
>>>>> why the segno test uses > rather than !=, but it's the point of the
>>>>> comment to explain things like that, rather than leaving the reader to
>>>>> guess. And I don't know why we also need to test startSegNo and
>>>>> endSegNo.
>>>>>
>>>>> I also wonder what the point is of doing XLogFromFileName() on the
>>>>> fname provided by the caller and then again on entry->fname. Couldn't
>>>>> you just compare the strings?
>>>>>
>>>>> Again, the division of labor is really odd here. It's the job of
>>>>> astreamer_waldump_content() to skip things that aren't WAL files at
>>>>> all, but it's the job of get_archive_wal_entry() to skip things that
>>>>> are WAL files but not the one we want. I disagree with putting those
>>>>> checks in completely separate parts of the code.
>>>>>
>>>> Keeping the timeline and segment start-end range checks inside the
>>>> archive streamer creates a circular dependency that cannot be resolved
>>>> without a 'dirty hack'. We must read the first available WAL file page
>>>> to determine the wal_segment_size before it can calculate the target
>>>> segment range. Moving the checks inside the streamer would make it
>>>> impossible to process that initial file, as the necessary filtering
>>>> parameters --  would still be unknown which would need to be skipped
>>>> for the first read somehow. What if later we realized that the first
>>>> WAL file which was allowed to be streamed by skipping that check is
>>>> irrelevant and doesn't fall under the start-end segment range?
>>>>
>>> Please have a look at the attached version, specifically patch 0005.
>>> In astreamer_waldump_content(), I have moved the WAL file filtration
>>> check from get_archive_wal_entry(). This check will be skipped during
>>> the initial read in init_archive_reader(), which instead performs it
>>> explicitly once it determines the WAL segment size and the start/end
>>> segments.
>>>
>>> To access the WAL segment size inside astreamer_waldump_content(), I
>>> have moved the WAL segment size variable into the XLogDumpPrivate
>>> structure in the separate 0004 patch.
>> Attached is an updated version including the aforesaid changes. It
>> includes a new refactoring patch (0001) that moves the logic for
>> identifying tar archives and their compression types from
>> pg_basebackup and pg_verifybackup into a separate-reusable function,
>> per a suggestion from Euler [1].  Additionally, I have added a test
>> for the contrecord decoding to the main patch (now 0006).
>>
>> 1] http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
>>
> Rebased against the latest master, fixed typos in code comments, and
> replaced palloc0 with palloc0_object.
>

Hi Amul.


I think this looks in pretty good shape.

Attached are patches for a few things I think could be fixed. They are 
mostly self-explanatory. The TAP test fix is the only sane way I could 
come up with stopping the skip code you had from reporting a wildly 
inaccurate number of tests skipped. The sane way to do this from a 
Test::More perspective is a subtest, but unfortunately meson does not 
like subtest output, which is why we don't use it elsewhere, so the only 
way I could come up with was to split this out into a separate test. Of 
course, we might just say we don't care about the misreport, in which 
case we could just live with things as they are.


cheers


andrew



--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  43. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-04T12:52:00Z

    On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
> > On Wed, Feb 18, 2026 at 12:28 PM Amul Sul <sulamul@gmail.com> wrote:
> >> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul <sulamul@gmail.com> wrote:
> >>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul <sulamul@gmail.com> wrote:
> >>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas <robertmhaas@gmail.com> wrote:
> >>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul <sulamul@gmail.com> wrote:
> >>>>>> In the attached version, I am using the WAL segment name as the hash
> >>>>>> key, which is much more straightforward. I have rewritten
> >>>>>> read_archive_wal_page(), and it looks much cleaner than before. The
> >>>>>> logic to discard irrelevant WAL files is still within
> >>>>>> get_archive_wal_entry. I added an explanation for setting cur_wal to
> >>>>>> NULL, which is now handled in the separate function I mentioned
> >>>>>> previously.
> >>>>>>
> >>>>>> Kindly have a look at the attached version; let me know if you are
> >>>>>> still not happy with the current approach for filtering/discarding
> >>>>>> irrelevant WAL segments. It isn't much different from the previous
> >>>>>> version, but I have tried to keep it in a separate routine for better
> >>>>>> code readability, with comments to make it easier to understand. I
> >>>>>> also added a comment for ArchivedWALFile.
> >>>>> I feel like the division of labor between get_archive_wal_entry() and
> >>>>> read_archive_wal_page() is odd. I noticed this in the last version,
> >>>>> too, and it still seems to be the case. get_archive_wal_entry() first
> >>>>> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> >>>>> If it doesn't, it loops until an entry for the requested file shows up
> >>>>> and then returns it. Then control returns to read_archive_wal_page()
> >>>>> which loops some more until we have all the data we need for the
> >>>>> requested file. But it seems odd to me to have two separate loops
> >>>>> here. I think that the first loop is going to call read_archive_file()
> >>>>> until we find the beginning of the file that we care about and then
> >>>>> the second one is going to call read_archive_file() some more until we
> >>>>> have read enough of it to satisfy the request. It feels odd to me to
> >>>>> do it that way, as if we told somebody to first wait until 9 o'clock
> >>>>> and then wait another 30 minutes, instead of just telling them to wait
> >>>>> until 9:30. I realize it's not quite the same thing, because apart
> >>>>> from calling read_archive_file(), the two loops do different things,
> >>>>> but I still think it looks odd.
> >>>>>
> >>>>> + /*
> >>>>> + * Ignore if the timeline is different or the current segment is not
> >>>>> + * the desired one.
> >>>>> + */
> >>>>> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> >>>>> + if (privateInfo->timeline != curSegTimeline ||
> >>>>> + privateInfo->startSegNo > curSegNo ||
> >>>>> + privateInfo->endSegNo < curSegNo ||
> >>>>> + segno > curSegNo)
> >>>>> + {
> >>>>> + free_archive_wal_entry(entry->fname, privateInfo);
> >>>>> + continue;
> >>>>> + }
> >>>>>
> >>>>> The comment doesn't match the code. If it did, the test would be
> >>>>> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> >>>>> instead the segno test is > rather than !=, and the checks against
> >>>>> startSegNo and endSegNo aren't explained at all. I think I understand
> >>>>> why the segno test uses > rather than !=, but it's the point of the
> >>>>> comment to explain things like that, rather than leaving the reader to
> >>>>> guess. And I don't know why we also need to test startSegNo and
> >>>>> endSegNo.
> >>>>>
> >>>>> I also wonder what the point is of doing XLogFromFileName() on the
> >>>>> fname provided by the caller and then again on entry->fname. Couldn't
> >>>>> you just compare the strings?
> >>>>>
> >>>>> Again, the division of labor is really odd here. It's the job of
> >>>>> astreamer_waldump_content() to skip things that aren't WAL files at
> >>>>> all, but it's the job of get_archive_wal_entry() to skip things that
> >>>>> are WAL files but not the one we want. I disagree with putting those
> >>>>> checks in completely separate parts of the code.
> >>>>>
> >>>> Keeping the timeline and segment start-end range checks inside the
> >>>> archive streamer creates a circular dependency that cannot be resolved
> >>>> without a 'dirty hack'. We must read the first available WAL file page
> >>>> to determine the wal_segment_size before it can calculate the target
> >>>> segment range. Moving the checks inside the streamer would make it
> >>>> impossible to process that initial file, as the necessary filtering
> >>>> parameters --  would still be unknown which would need to be skipped
> >>>> for the first read somehow. What if later we realized that the first
> >>>> WAL file which was allowed to be streamed by skipping that check is
> >>>> irrelevant and doesn't fall under the start-end segment range?
> >>>>
> >>> Please have a look at the attached version, specifically patch 0005.
> >>> In astreamer_waldump_content(), I have moved the WAL file filtration
> >>> check from get_archive_wal_entry(). This check will be skipped during
> >>> the initial read in init_archive_reader(), which instead performs it
> >>> explicitly once it determines the WAL segment size and the start/end
> >>> segments.
> >>>
> >>> To access the WAL segment size inside astreamer_waldump_content(), I
> >>> have moved the WAL segment size variable into the XLogDumpPrivate
> >>> structure in the separate 0004 patch.
> >> Attached is an updated version including the aforesaid changes. It
> >> includes a new refactoring patch (0001) that moves the logic for
> >> identifying tar archives and their compression types from
> >> pg_basebackup and pg_verifybackup into a separate-reusable function,
> >> per a suggestion from Euler [1].  Additionally, I have added a test
> >> for the contrecord decoding to the main patch (now 0006).
> >>
> >> 1] http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
> >>
> > Rebased against the latest master, fixed typos in code comments, and
> > replaced palloc0 with palloc0_object.
> >
>
> Hi Amul.
>
>
> I think this looks in pretty good shape.
>

Thank you very much for looking at the patch.

> Attached are patches for a few things I think could be fixed. They are
> mostly self-explanatory. The TAP test fix is the only sane way I could
> come up with stopping the skip code you had from reporting a wildly
> inaccurate number of tests skipped. The sane way to do this from a
> Test::More perspective is a subtest, but unfortunately meson does not
> like subtest output, which is why we don't use it elsewhere, so the only
> way I could come up with was to split this out into a separate test. Of
> course, we might just say we don't care about the misreport, in which
> case we could just live with things as they are.
>

I agree that the reported skip number was incorrect, and I have
corrected it in the attached patch. I haven't applied your patch for
the TAP test improvements yet because I wanted to double-check it
first with you; the patch as it stood created duplicate tests already
present in 001_basic.pl. To avoid this duplication, I have added a
loop that performs tests for both plain and tar WAL directory inputs,
similar to the approach used in pg_verifybackup for different
compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
don't have any objection to doing so if you feel the duplication is
acceptable, but I feel that using a loop for the tests in 001_basic.pl
is a bit tidier. Let me know your thoughts.

I have applied all your other patches but skipped the changes to
pg_verifybackup.c from cf5955-fixes.patch.no-cfbot, as they seem
unrelated or perhaps I have misunderstood them.


Regards,
Amul

  44. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-04T21:50:30Z

    On 2026-03-04 We 7:52 AM, Amul Sul wrote:
> On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan<andrew@dunslane.net> wrote:
>>
>> On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
>>> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul<sulamul@gmail.com> wrote:
>>>> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas<robertmhaas@gmail.com> wrote:
>>>>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>>> In the attached version, I am using the WAL segment name as the hash
>>>>>>>> key, which is much more straightforward. I have rewritten
>>>>>>>> read_archive_wal_page(), and it looks much cleaner than before. The
>>>>>>>> logic to discard irrelevant WAL files is still within
>>>>>>>> get_archive_wal_entry. I added an explanation for setting cur_wal to
>>>>>>>> NULL, which is now handled in the separate function I mentioned
>>>>>>>> previously.
>>>>>>>>
>>>>>>>> Kindly have a look at the attached version; let me know if you are
>>>>>>>> still not happy with the current approach for filtering/discarding
>>>>>>>> irrelevant WAL segments. It isn't much different from the previous
>>>>>>>> version, but I have tried to keep it in a separate routine for better
>>>>>>>> code readability, with comments to make it easier to understand. I
>>>>>>>> also added a comment for ArchivedWALFile.
>>>>>>> I feel like the division of labor between get_archive_wal_entry() and
>>>>>>> read_archive_wal_page() is odd. I noticed this in the last version,
>>>>>>> too, and it still seems to be the case. get_archive_wal_entry() first
>>>>>>> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
>>>>>>> If it doesn't, it loops until an entry for the requested file shows up
>>>>>>> and then returns it. Then control returns to read_archive_wal_page()
>>>>>>> which loops some more until we have all the data we need for the
>>>>>>> requested file. But it seems odd to me to have two separate loops
>>>>>>> here. I think that the first loop is going to call read_archive_file()
>>>>>>> until we find the beginning of the file that we care about and then
>>>>>>> the second one is going to call read_archive_file() some more until we
>>>>>>> have read enough of it to satisfy the request. It feels odd to me to
>>>>>>> do it that way, as if we told somebody to first wait until 9 o'clock
>>>>>>> and then wait another 30 minutes, instead of just telling them to wait
>>>>>>> until 9:30. I realize it's not quite the same thing, because apart
>>>>>>> from calling read_archive_file(), the two loops do different things,
>>>>>>> but I still think it looks odd.
>>>>>>>
>>>>>>> + /*
>>>>>>> + * Ignore if the timeline is different or the current segment is not
>>>>>>> + * the desired one.
>>>>>>> + */
>>>>>>> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
>>>>>>> + if (privateInfo->timeline != curSegTimeline ||
>>>>>>> + privateInfo->startSegNo > curSegNo ||
>>>>>>> + privateInfo->endSegNo < curSegNo ||
>>>>>>> + segno > curSegNo)
>>>>>>> + {
>>>>>>> + free_archive_wal_entry(entry->fname, privateInfo);
>>>>>>> + continue;
>>>>>>> + }
>>>>>>>
>>>>>>> The comment doesn't match the code. If it did, the test would be
>>>>>>> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
>>>>>>> instead the segno test is > rather than !=, and the checks against
>>>>>>> startSegNo and endSegNo aren't explained at all. I think I understand
>>>>>>> why the segno test uses > rather than !=, but it's the point of the
>>>>>>> comment to explain things like that, rather than leaving the reader to
>>>>>>> guess. And I don't know why we also need to test startSegNo and
>>>>>>> endSegNo.
>>>>>>>
>>>>>>> I also wonder what the point is of doing XLogFromFileName() on the
>>>>>>> fname provided by the caller and then again on entry->fname. Couldn't
>>>>>>> you just compare the strings?
>>>>>>>
>>>>>>> Again, the division of labor is really odd here. It's the job of
>>>>>>> astreamer_waldump_content() to skip things that aren't WAL files at
>>>>>>> all, but it's the job of get_archive_wal_entry() to skip things that
>>>>>>> are WAL files but not the one we want. I disagree with putting those
>>>>>>> checks in completely separate parts of the code.
>>>>>>>
>>>>>> Keeping the timeline and segment start-end range checks inside the
>>>>>> archive streamer creates a circular dependency that cannot be resolved
>>>>>> without a 'dirty hack'. We must read the first available WAL file page
>>>>>> to determine the wal_segment_size before it can calculate the target
>>>>>> segment range. Moving the checks inside the streamer would make it
>>>>>> impossible to process that initial file, as the necessary filtering
>>>>>> parameters --  would still be unknown which would need to be skipped
>>>>>> for the first read somehow. What if later we realized that the first
>>>>>> WAL file which was allowed to be streamed by skipping that check is
>>>>>> irrelevant and doesn't fall under the start-end segment range?
>>>>>>
>>>>> Please have a look at the attached version, specifically patch 0005.
>>>>> In astreamer_waldump_content(), I have moved the WAL file filtration
>>>>> check from get_archive_wal_entry(). This check will be skipped during
>>>>> the initial read in init_archive_reader(), which instead performs it
>>>>> explicitly once it determines the WAL segment size and the start/end
>>>>> segments.
>>>>>
>>>>> To access the WAL segment size inside astreamer_waldump_content(), I
>>>>> have moved the WAL segment size variable into the XLogDumpPrivate
>>>>> structure in the separate 0004 patch.
>>>> Attached is an updated version including the aforesaid changes. It
>>>> includes a new refactoring patch (0001) that moves the logic for
>>>> identifying tar archives and their compression types from
>>>> pg_basebackup and pg_verifybackup into a separate-reusable function,
>>>> per a suggestion from Euler [1].  Additionally, I have added a test
>>>> for the contrecord decoding to the main patch (now 0006).
>>>>
>>>> 1]http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
>>>>
>>> Rebased against the latest master, fixed typos in code comments, and
>>> replaced palloc0 with palloc0_object.
>>>
>> Hi Amul.
>>
>>
>> I think this looks in pretty good shape.
>>
> Thank you very much for looking at the patch.
>
>> Attached are patches for a few things I think could be fixed. They are
>> mostly self-explanatory. The TAP test fix is the only sane way I could
>> come up with stopping the skip code you had from reporting a wildly
>> inaccurate number of tests skipped. The sane way to do this from a
>> Test::More perspective is a subtest, but unfortunately meson does not
>> like subtest output, which is why we don't use it elsewhere, so the only
>> way I could come up with was to split this out into a separate test. Of
>> course, we might just say we don't care about the misreport, in which
>> case we could just live with things as they are.
>>
> I agree that the reported skip number was incorrect, and I have
> corrected it in the attached patch. I haven't applied your patch for
> the TAP test improvements yet because I wanted to double-check it
> first with you; the patch as it stood created duplicate tests already
> present in 001_basic.pl. To avoid this duplication, I have added a
> loop that performs tests for both plain and tar WAL directory inputs,
> similar to the approach used in pg_verifybackup for different
> compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
> don't have any objection to doing so if you feel the duplication is
> acceptable, but I feel that using a loop for the tests in 001_basic.pl
> is a bit tidier. Let me know your thoughts.


I will take a look.

>
> I have applied all your other patches but skipped the changes to
> pg_verifybackup.c from cf5955-fixes.patch.no-cfbot, as they seem
> unrelated or perhaps I have misunderstood them.


<brown-paper-bag> That's what I get for using a poorly written tool.


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  45. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-06T22:21:09Z

    On 2026-03-04 We 4:50 PM, Andrew Dunstan wrote:
>
>
> On 2026-03-04 We 7:52 AM, Amul Sul wrote:
>> On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan<andrew@dunslane.net> wrote:
>>> On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
>>>> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas<robertmhaas@gmail.com> wrote:
>>>>>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>>>> In the attached version, I am using the WAL segment name as the hash
>>>>>>>>> key, which is much more straightforward. I have rewritten
>>>>>>>>> read_archive_wal_page(), and it looks much cleaner than before. The
>>>>>>>>> logic to discard irrelevant WAL files is still within
>>>>>>>>> get_archive_wal_entry. I added an explanation for setting cur_wal to
>>>>>>>>> NULL, which is now handled in the separate function I mentioned
>>>>>>>>> previously.
>>>>>>>>>
>>>>>>>>> Kindly have a look at the attached version; let me know if you are
>>>>>>>>> still not happy with the current approach for filtering/discarding
>>>>>>>>> irrelevant WAL segments. It isn't much different from the previous
>>>>>>>>> version, but I have tried to keep it in a separate routine for better
>>>>>>>>> code readability, with comments to make it easier to understand. I
>>>>>>>>> also added a comment for ArchivedWALFile.
>>>>>>>> I feel like the division of labor between get_archive_wal_entry() and
>>>>>>>> read_archive_wal_page() is odd. I noticed this in the last version,
>>>>>>>> too, and it still seems to be the case. get_archive_wal_entry() first
>>>>>>>> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
>>>>>>>> If it doesn't, it loops until an entry for the requested file shows up
>>>>>>>> and then returns it. Then control returns to read_archive_wal_page()
>>>>>>>> which loops some more until we have all the data we need for the
>>>>>>>> requested file. But it seems odd to me to have two separate loops
>>>>>>>> here. I think that the first loop is going to call read_archive_file()
>>>>>>>> until we find the beginning of the file that we care about and then
>>>>>>>> the second one is going to call read_archive_file() some more until we
>>>>>>>> have read enough of it to satisfy the request. It feels odd to me to
>>>>>>>> do it that way, as if we told somebody to first wait until 9 o'clock
>>>>>>>> and then wait another 30 minutes, instead of just telling them to wait
>>>>>>>> until 9:30. I realize it's not quite the same thing, because apart
>>>>>>>> from calling read_archive_file(), the two loops do different things,
>>>>>>>> but I still think it looks odd.
>>>>>>>>
>>>>>>>> + /*
>>>>>>>> + * Ignore if the timeline is different or the current segment is not
>>>>>>>> + * the desired one.
>>>>>>>> + */
>>>>>>>> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
>>>>>>>> + if (privateInfo->timeline != curSegTimeline ||
>>>>>>>> + privateInfo->startSegNo > curSegNo ||
>>>>>>>> + privateInfo->endSegNo < curSegNo ||
>>>>>>>> + segno > curSegNo)
>>>>>>>> + {
>>>>>>>> + free_archive_wal_entry(entry->fname, privateInfo);
>>>>>>>> + continue;
>>>>>>>> + }
>>>>>>>>
>>>>>>>> The comment doesn't match the code. If it did, the test would be
>>>>>>>> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
>>>>>>>> instead the segno test is > rather than !=, and the checks against
>>>>>>>> startSegNo and endSegNo aren't explained at all. I think I understand
>>>>>>>> why the segno test uses > rather than !=, but it's the point of the
>>>>>>>> comment to explain things like that, rather than leaving the reader to
>>>>>>>> guess. And I don't know why we also need to test startSegNo and
>>>>>>>> endSegNo.
>>>>>>>>
>>>>>>>> I also wonder what the point is of doing XLogFromFileName() on the
>>>>>>>> fname provided by the caller and then again on entry->fname. Couldn't
>>>>>>>> you just compare the strings?
>>>>>>>>
>>>>>>>> Again, the division of labor is really odd here. It's the job of
>>>>>>>> astreamer_waldump_content() to skip things that aren't WAL files at
>>>>>>>> all, but it's the job of get_archive_wal_entry() to skip things that
>>>>>>>> are WAL files but not the one we want. I disagree with putting those
>>>>>>>> checks in completely separate parts of the code.
>>>>>>>>
>>>>>>> Keeping the timeline and segment start-end range checks inside the
>>>>>>> archive streamer creates a circular dependency that cannot be resolved
>>>>>>> without a 'dirty hack'. We must read the first available WAL file page
>>>>>>> to determine the wal_segment_size before it can calculate the target
>>>>>>> segment range. Moving the checks inside the streamer would make it
>>>>>>> impossible to process that initial file, as the necessary filtering
>>>>>>> parameters --  would still be unknown which would need to be skipped
>>>>>>> for the first read somehow. What if later we realized that the first
>>>>>>> WAL file which was allowed to be streamed by skipping that check is
>>>>>>> irrelevant and doesn't fall under the start-end segment range?
>>>>>>>
>>>>>> Please have a look at the attached version, specifically patch 0005.
>>>>>> In astreamer_waldump_content(), I have moved the WAL file filtration
>>>>>> check from get_archive_wal_entry(). This check will be skipped during
>>>>>> the initial read in init_archive_reader(), which instead performs it
>>>>>> explicitly once it determines the WAL segment size and the start/end
>>>>>> segments.
>>>>>>
>>>>>> To access the WAL segment size inside astreamer_waldump_content(), I
>>>>>> have moved the WAL segment size variable into the XLogDumpPrivate
>>>>>> structure in the separate 0004 patch.
>>>>> Attached is an updated version including the aforesaid changes. It
>>>>> includes a new refactoring patch (0001) that moves the logic for
>>>>> identifying tar archives and their compression types from
>>>>> pg_basebackup and pg_verifybackup into a separate-reusable function,
>>>>> per a suggestion from Euler [1].  Additionally, I have added a test
>>>>> for the contrecord decoding to the main patch (now 0006).
>>>>>
>>>>> 1]http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
>>>>>
>>>> Rebased against the latest master, fixed typos in code comments, and
>>>> replaced palloc0 with palloc0_object.
>>>>
>>> Hi Amul.
>>>
>>>
>>> I think this looks in pretty good shape.
>>>
>> Thank you very much for looking at the patch.
>>
>>> Attached are patches for a few things I think could be fixed. They are
>>> mostly self-explanatory. The TAP test fix is the only sane way I could
>>> come up with stopping the skip code you had from reporting a wildly
>>> inaccurate number of tests skipped. The sane way to do this from a
>>> Test::More perspective is a subtest, but unfortunately meson does not
>>> like subtest output, which is why we don't use it elsewhere, so the only
>>> way I could come up with was to split this out into a separate test. Of
>>> course, we might just say we don't care about the misreport, in which
>>> case we could just live with things as they are.
>>>
>> I agree that the reported skip number was incorrect, and I have
>> corrected it in the attached patch. I haven't applied your patch for
>> the TAP test improvements yet because I wanted to double-check it
>> first with you; the patch as it stood created duplicate tests already
>> present in 001_basic.pl. To avoid this duplication, I have added a
>> loop that performs tests for both plain and tar WAL directory inputs,
>> similar to the approach used in pg_verifybackup for different
>> compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
>> don't have any objection to doing so if you feel the duplication is
>> acceptable, but I feel that using a loop for the tests in 001_basic.pl
>> is a bit tidier. Let me know your thoughts.
>
>
> I will take a look.
>

I'm ok, with doing it this way. It's just a bit fragile - if we add a 
test the number will be wrong. But maybe it's not worth worrying about.

Everything else looks fairly good. The attached fixes a few relatively 
minor issues in v15. The main one is that it stops allocating/freeing a 
buffer every time we call read_archive_file() and instead adds a 
reusable buffer. It also adds back wal-directory as an undocumented 
alias of wal-path, to avoid breaking legacy scripts unnecessarily, and 
adds constness to the fname argument of pg_tar_compress_algorithm, as 
well as fixing some indentation and grammar issues.

All in all I think we're in good shape.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  46. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-09T12:26:50Z

    On Sat, Mar 7, 2026 at 3:51 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2026-03-04 We 4:50 PM, Andrew Dunstan wrote:
> >
> >
> > On 2026-03-04 We 7:52 AM, Amul Sul wrote:
> >> On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan<andrew@dunslane.net> wrote:
> >>> On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
> >>>> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul<sulamul@gmail.com> wrote:
> >>>>> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul<sulamul@gmail.com> wrote:
> >>>>>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul<sulamul@gmail.com> wrote:
> >>>>>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas<robertmhaas@gmail.com> wrote:
> >>>>>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul<sulamul@gmail.com> wrote:
> >>>>>>>>> In the attached version, I am using the WAL segment name as the hash
> >>>>>>>>> key, which is much more straightforward. I have rewritten
> >>>>>>>>> read_archive_wal_page(), and it looks much cleaner than before. The
> >>>>>>>>> logic to discard irrelevant WAL files is still within
> >>>>>>>>> get_archive_wal_entry. I added an explanation for setting cur_wal to
> >>>>>>>>> NULL, which is now handled in the separate function I mentioned
> >>>>>>>>> previously.
> >>>>>>>>>
> >>>>>>>>> Kindly have a look at the attached version; let me know if you are
> >>>>>>>>> still not happy with the current approach for filtering/discarding
> >>>>>>>>> irrelevant WAL segments. It isn't much different from the previous
> >>>>>>>>> version, but I have tried to keep it in a separate routine for better
> >>>>>>>>> code readability, with comments to make it easier to understand. I
> >>>>>>>>> also added a comment for ArchivedWALFile.
> >>>>>>>> I feel like the division of labor between get_archive_wal_entry() and
> >>>>>>>> read_archive_wal_page() is odd. I noticed this in the last version,
> >>>>>>>> too, and it still seems to be the case. get_archive_wal_entry() first
> >>>>>>>> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> >>>>>>>> If it doesn't, it loops until an entry for the requested file shows up
> >>>>>>>> and then returns it. Then control returns to read_archive_wal_page()
> >>>>>>>> which loops some more until we have all the data we need for the
> >>>>>>>> requested file. But it seems odd to me to have two separate loops
> >>>>>>>> here. I think that the first loop is going to call read_archive_file()
> >>>>>>>> until we find the beginning of the file that we care about and then
> >>>>>>>> the second one is going to call read_archive_file() some more until we
> >>>>>>>> have read enough of it to satisfy the request. It feels odd to me to
> >>>>>>>> do it that way, as if we told somebody to first wait until 9 o'clock
> >>>>>>>> and then wait another 30 minutes, instead of just telling them to wait
> >>>>>>>> until 9:30. I realize it's not quite the same thing, because apart
> >>>>>>>> from calling read_archive_file(), the two loops do different things,
> >>>>>>>> but I still think it looks odd.
> >>>>>>>>
> >>>>>>>> + /*
> >>>>>>>> + * Ignore if the timeline is different or the current segment is not
> >>>>>>>> + * the desired one.
> >>>>>>>> + */
> >>>>>>>> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> >>>>>>>> + if (privateInfo->timeline != curSegTimeline ||
> >>>>>>>> + privateInfo->startSegNo > curSegNo ||
> >>>>>>>> + privateInfo->endSegNo < curSegNo ||
> >>>>>>>> + segno > curSegNo)
> >>>>>>>> + {
> >>>>>>>> + free_archive_wal_entry(entry->fname, privateInfo);
> >>>>>>>> + continue;
> >>>>>>>> + }
> >>>>>>>>
> >>>>>>>> The comment doesn't match the code. If it did, the test would be
> >>>>>>>> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> >>>>>>>> instead the segno test is > rather than !=, and the checks against
> >>>>>>>> startSegNo and endSegNo aren't explained at all. I think I understand
> >>>>>>>> why the segno test uses > rather than !=, but it's the point of the
> >>>>>>>> comment to explain things like that, rather than leaving the reader to
> >>>>>>>> guess. And I don't know why we also need to test startSegNo and
> >>>>>>>> endSegNo.
> >>>>>>>>
> >>>>>>>> I also wonder what the point is of doing XLogFromFileName() on the
> >>>>>>>> fname provided by the caller and then again on entry->fname. Couldn't
> >>>>>>>> you just compare the strings?
> >>>>>>>>
> >>>>>>>> Again, the division of labor is really odd here. It's the job of
> >>>>>>>> astreamer_waldump_content() to skip things that aren't WAL files at
> >>>>>>>> all, but it's the job of get_archive_wal_entry() to skip things that
> >>>>>>>> are WAL files but not the one we want. I disagree with putting those
> >>>>>>>> checks in completely separate parts of the code.
> >>>>>>>>
> >>>>>>> Keeping the timeline and segment start-end range checks inside the
> >>>>>>> archive streamer creates a circular dependency that cannot be resolved
> >>>>>>> without a 'dirty hack'. We must read the first available WAL file page
> >>>>>>> to determine the wal_segment_size before it can calculate the target
> >>>>>>> segment range. Moving the checks inside the streamer would make it
> >>>>>>> impossible to process that initial file, as the necessary filtering
> >>>>>>> parameters --  would still be unknown which would need to be skipped
> >>>>>>> for the first read somehow. What if later we realized that the first
> >>>>>>> WAL file which was allowed to be streamed by skipping that check is
> >>>>>>> irrelevant and doesn't fall under the start-end segment range?
> >>>>>>>
> >>>>>> Please have a look at the attached version, specifically patch 0005.
> >>>>>> In astreamer_waldump_content(), I have moved the WAL file filtration
> >>>>>> check from get_archive_wal_entry(). This check will be skipped during
> >>>>>> the initial read in init_archive_reader(), which instead performs it
> >>>>>> explicitly once it determines the WAL segment size and the start/end
> >>>>>> segments.
> >>>>>>
> >>>>>> To access the WAL segment size inside astreamer_waldump_content(), I
> >>>>>> have moved the WAL segment size variable into the XLogDumpPrivate
> >>>>>> structure in the separate 0004 patch.
> >>>>> Attached is an updated version including the aforesaid changes. It
> >>>>> includes a new refactoring patch (0001) that moves the logic for
> >>>>> identifying tar archives and their compression types from
> >>>>> pg_basebackup and pg_verifybackup into a separate-reusable function,
> >>>>> per a suggestion from Euler [1].  Additionally, I have added a test
> >>>>> for the contrecord decoding to the main patch (now 0006).
> >>>>>
> >>>>> 1]http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
> >>>>>
> >>>> Rebased against the latest master, fixed typos in code comments, and
> >>>> replaced palloc0 with palloc0_object.
> >>>>
> >>> Hi Amul.
> >>>
> >>>
> >>> I think this looks in pretty good shape.
> >>>
> >> Thank you very much for looking at the patch.
> >>
> >>> Attached are patches for a few things I think could be fixed. They are
> >>> mostly self-explanatory. The TAP test fix is the only sane way I could
> >>> come up with stopping the skip code you had from reporting a wildly
> >>> inaccurate number of tests skipped. The sane way to do this from a
> >>> Test::More perspective is a subtest, but unfortunately meson does not
> >>> like subtest output, which is why we don't use it elsewhere, so the only
> >>> way I could come up with was to split this out into a separate test. Of
> >>> course, we might just say we don't care about the misreport, in which
> >>> case we could just live with things as they are.
> >>>
> >> I agree that the reported skip number was incorrect, and I have
> >> corrected it in the attached patch. I haven't applied your patch for
> >> the TAP test improvements yet because I wanted to double-check it
> >> first with you; the patch as it stood created duplicate tests already
> >> present in 001_basic.pl. To avoid this duplication, I have added a
> >> loop that performs tests for both plain and tar WAL directory inputs,
> >> similar to the approach used in pg_verifybackup for different
> >> compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
> >> don't have any objection to doing so if you feel the duplication is
> >> acceptable, but I feel that using a loop for the tests in 001_basic.pl
> >> is a bit tidier. Let me know your thoughts.
> >
> >
> > I will take a look.
> >
>
> I'm ok, with doing it this way. It's just a bit fragile - if we add a
> test the number will be wrong. But maybe it's not worth worrying about.
>
> Everything else looks fairly good. The attached fixes a few relatively
> minor issues in v15. The main one is that it stops allocating/freeing a
> buffer every time we call read_archive_file() and instead adds a
> reusable buffer. It also adds back wal-directory as an undocumented
> alias of wal-path, to avoid breaking legacy scripts unnecessarily, and
> adds constness to the fname argument of pg_tar_compress_algorithm, as
> well as fixing some indentation and grammar issues.
>
> All in all I think we're in good shape.

Thanks for the review. I have incorporated your suggested changes,
with one exception: I have skipped the buffer reallocation code in
read_archive_file(). Since we only handle two specific read sizes --
XLOG_BLCKSZ and READ_CHUNK_SIZE (128 KB, we defined in
archive_waldump.c) -- dynamic reallocation seems unnecessary. Instead,
I moved the allocation to init_archive_reader(), which now initializes
a buffer at READ_CHUNK_SIZE. I also added an assertion in
read_archive_file() to ensure that no read request exceeds this
allocated capacity.

Kindly have a look at the attached version and let me know your thoughts.

Regards,
Amul

  47. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-11T17:08:04Z

    On 2026-03-09 Mo 8:26 AM, Amul Sul wrote:
> On Sat, Mar 7, 2026 at 3:51 AM Andrew Dunstan<andrew@dunslane.net> wrote:
>>
>> On 2026-03-04 We 4:50 PM, Andrew Dunstan wrote:
>>>
>>> On 2026-03-04 We 7:52 AM, Amul Sul wrote:
>>>> On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan<andrew@dunslane.net> wrote:
>>>>> On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
>>>>>> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>>> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>>>> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas<robertmhaas@gmail.com> wrote:
>>>>>>>>>> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul<sulamul@gmail.com> wrote:
>>>>>>>>>>> In the attached version, I am using the WAL segment name as the hash
>>>>>>>>>>> key, which is much more straightforward. I have rewritten
>>>>>>>>>>> read_archive_wal_page(), and it looks much cleaner than before. The
>>>>>>>>>>> logic to discard irrelevant WAL files is still within
>>>>>>>>>>> get_archive_wal_entry. I added an explanation for setting cur_wal to
>>>>>>>>>>> NULL, which is now handled in the separate function I mentioned
>>>>>>>>>>> previously.
>>>>>>>>>>>
>>>>>>>>>>> Kindly have a look at the attached version; let me know if you are
>>>>>>>>>>> still not happy with the current approach for filtering/discarding
>>>>>>>>>>> irrelevant WAL segments. It isn't much different from the previous
>>>>>>>>>>> version, but I have tried to keep it in a separate routine for better
>>>>>>>>>>> code readability, with comments to make it easier to understand. I
>>>>>>>>>>> also added a comment for ArchivedWALFile.
>>>>>>>>>> I feel like the division of labor between get_archive_wal_entry() and
>>>>>>>>>> read_archive_wal_page() is odd. I noticed this in the last version,
>>>>>>>>>> too, and it still seems to be the case. get_archive_wal_entry() first
>>>>>>>>>> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
>>>>>>>>>> If it doesn't, it loops until an entry for the requested file shows up
>>>>>>>>>> and then returns it. Then control returns to read_archive_wal_page()
>>>>>>>>>> which loops some more until we have all the data we need for the
>>>>>>>>>> requested file. But it seems odd to me to have two separate loops
>>>>>>>>>> here. I think that the first loop is going to call read_archive_file()
>>>>>>>>>> until we find the beginning of the file that we care about and then
>>>>>>>>>> the second one is going to call read_archive_file() some more until we
>>>>>>>>>> have read enough of it to satisfy the request. It feels odd to me to
>>>>>>>>>> do it that way, as if we told somebody to first wait until 9 o'clock
>>>>>>>>>> and then wait another 30 minutes, instead of just telling them to wait
>>>>>>>>>> until 9:30. I realize it's not quite the same thing, because apart
>>>>>>>>>> from calling read_archive_file(), the two loops do different things,
>>>>>>>>>> but I still think it looks odd.
>>>>>>>>>>
>>>>>>>>>> + /*
>>>>>>>>>> + * Ignore if the timeline is different or the current segment is not
>>>>>>>>>> + * the desired one.
>>>>>>>>>> + */
>>>>>>>>>> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
>>>>>>>>>> + if (privateInfo->timeline != curSegTimeline ||
>>>>>>>>>> + privateInfo->startSegNo > curSegNo ||
>>>>>>>>>> + privateInfo->endSegNo < curSegNo ||
>>>>>>>>>> + segno > curSegNo)
>>>>>>>>>> + {
>>>>>>>>>> + free_archive_wal_entry(entry->fname, privateInfo);
>>>>>>>>>> + continue;
>>>>>>>>>> + }
>>>>>>>>>>
>>>>>>>>>> The comment doesn't match the code. If it did, the test would be
>>>>>>>>>> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
>>>>>>>>>> instead the segno test is > rather than !=, and the checks against
>>>>>>>>>> startSegNo and endSegNo aren't explained at all. I think I understand
>>>>>>>>>> why the segno test uses > rather than !=, but it's the point of the
>>>>>>>>>> comment to explain things like that, rather than leaving the reader to
>>>>>>>>>> guess. And I don't know why we also need to test startSegNo and
>>>>>>>>>> endSegNo.
>>>>>>>>>>
>>>>>>>>>> I also wonder what the point is of doing XLogFromFileName() on the
>>>>>>>>>> fname provided by the caller and then again on entry->fname. Couldn't
>>>>>>>>>> you just compare the strings?
>>>>>>>>>>
>>>>>>>>>> Again, the division of labor is really odd here. It's the job of
>>>>>>>>>> astreamer_waldump_content() to skip things that aren't WAL files at
>>>>>>>>>> all, but it's the job of get_archive_wal_entry() to skip things that
>>>>>>>>>> are WAL files but not the one we want. I disagree with putting those
>>>>>>>>>> checks in completely separate parts of the code.
>>>>>>>>>>
>>>>>>>>> Keeping the timeline and segment start-end range checks inside the
>>>>>>>>> archive streamer creates a circular dependency that cannot be resolved
>>>>>>>>> without a 'dirty hack'. We must read the first available WAL file page
>>>>>>>>> to determine the wal_segment_size before it can calculate the target
>>>>>>>>> segment range. Moving the checks inside the streamer would make it
>>>>>>>>> impossible to process that initial file, as the necessary filtering
>>>>>>>>> parameters --  would still be unknown which would need to be skipped
>>>>>>>>> for the first read somehow. What if later we realized that the first
>>>>>>>>> WAL file which was allowed to be streamed by skipping that check is
>>>>>>>>> irrelevant and doesn't fall under the start-end segment range?
>>>>>>>>>
>>>>>>>> Please have a look at the attached version, specifically patch 0005.
>>>>>>>> In astreamer_waldump_content(), I have moved the WAL file filtration
>>>>>>>> check from get_archive_wal_entry(). This check will be skipped during
>>>>>>>> the initial read in init_archive_reader(), which instead performs it
>>>>>>>> explicitly once it determines the WAL segment size and the start/end
>>>>>>>> segments.
>>>>>>>>
>>>>>>>> To access the WAL segment size inside astreamer_waldump_content(), I
>>>>>>>> have moved the WAL segment size variable into the XLogDumpPrivate
>>>>>>>> structure in the separate 0004 patch.
>>>>>>> Attached is an updated version including the aforesaid changes. It
>>>>>>> includes a new refactoring patch (0001) that moves the logic for
>>>>>>> identifying tar archives and their compression types from
>>>>>>> pg_basebackup and pg_verifybackup into a separate-reusable function,
>>>>>>> per a suggestion from Euler [1].  Additionally, I have added a test
>>>>>>> for the contrecord decoding to the main patch (now 0006).
>>>>>>>
>>>>>>> 1]http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
>>>>>>>
>>>>>> Rebased against the latest master, fixed typos in code comments, and
>>>>>> replaced palloc0 with palloc0_object.
>>>>>>
>>>>> Hi Amul.
>>>>>
>>>>>
>>>>> I think this looks in pretty good shape.
>>>>>
>>>> Thank you very much for looking at the patch.
>>>>
>>>>> Attached are patches for a few things I think could be fixed. They are
>>>>> mostly self-explanatory. The TAP test fix is the only sane way I could
>>>>> come up with stopping the skip code you had from reporting a wildly
>>>>> inaccurate number of tests skipped. The sane way to do this from a
>>>>> Test::More perspective is a subtest, but unfortunately meson does not
>>>>> like subtest output, which is why we don't use it elsewhere, so the only
>>>>> way I could come up with was to split this out into a separate test. Of
>>>>> course, we might just say we don't care about the misreport, in which
>>>>> case we could just live with things as they are.
>>>>>
>>>> I agree that the reported skip number was incorrect, and I have
>>>> corrected it in the attached patch. I haven't applied your patch for
>>>> the TAP test improvements yet because I wanted to double-check it
>>>> first with you; the patch as it stood created duplicate tests already
>>>> present in 001_basic.pl. To avoid this duplication, I have added a
>>>> loop that performs tests for both plain and tar WAL directory inputs,
>>>> similar to the approach used in pg_verifybackup for different
>>>> compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
>>>> don't have any objection to doing so if you feel the duplication is
>>>> acceptable, but I feel that using a loop for the tests in 001_basic.pl
>>>> is a bit tidier. Let me know your thoughts.
>>>
>>> I will take a look.
>>>
>> I'm ok, with doing it this way. It's just a bit fragile - if we add a
>> test the number will be wrong. But maybe it's not worth worrying about.
>>
>> Everything else looks fairly good. The attached fixes a few relatively
>> minor issues in v15. The main one is that it stops allocating/freeing a
>> buffer every time we call read_archive_file() and instead adds a
>> reusable buffer. It also adds back wal-directory as an undocumented
>> alias of wal-path, to avoid breaking legacy scripts unnecessarily, and
>> adds constness to the fname argument of pg_tar_compress_algorithm, as
>> well as fixing some indentation and grammar issues.
>>
>> All in all I think we're in good shape.
> Thanks for the review. I have incorporated your suggested changes,
> with one exception: I have skipped the buffer reallocation code in
> read_archive_file(). Since we only handle two specific read sizes --
> XLOG_BLCKSZ and READ_CHUNK_SIZE (128 KB, we defined in
> archive_waldump.c) -- dynamic reallocation seems unnecessary. Instead,
> I moved the allocation to init_archive_reader(), which now initializes
> a buffer at READ_CHUNK_SIZE. I also added an assertion in
> read_archive_file() to ensure that no read request exceeds this
> allocated capacity.
>
> Kindly have a look at the attached version and let me know your thoughts.
>

Looks pretty good. I have squashed them into three patches I think are 
committable. Also attached is a diff showing what's changed - mainly this:

. --follow + tar archive rejected (pg_waldump.c) — new validation 
prevents a confusing pg_fatal when combining --follow with a tar archive
. error messages split (archive_waldump.c) — the single "could not read 
file" error is now two distinct messages: "WAL segment is too short" 
(truncated file) vs "unexpected end of archive" (archive EOF) - Fixes an 
issue raised in review
. hash table cleanup (archive_waldump.c) — free_archive_reader now 
iterates and frees all remaining hash entries and destroys the table

cheers

andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  48. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-18T11:45:47Z

    On Wed, Mar 11, 2026 at 10:38 PM Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2026-03-09 Mo 8:26 AM, Amul Sul wrote:
>
> On Sat, Mar 7, 2026 at 3:51 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>
> On 2026-03-04 We 4:50 PM, Andrew Dunstan wrote:
>
> On 2026-03-04 We 7:52 AM, Amul Sul wrote:
>
> On Wed, Mar 4, 2026 at 6:07 AM Andrew Dunstan<andrew@dunslane.net> wrote:
>
> On 2026-03-02 Mo 8:00 AM, Amul Sul wrote:
>
> On Wed, Feb 18, 2026 at 12:28 PM Amul Sul<sulamul@gmail.com> wrote:
>
> On Tue, Feb 10, 2026 at 3:06 PM Amul Sul<sulamul@gmail.com> wrote:
>
> On Wed, Feb 4, 2026 at 6:39 PM Amul Sul<sulamul@gmail.com> wrote:
>
> On Wed, Jan 28, 2026 at 2:41 AM Robert Haas<robertmhaas@gmail.com> wrote:
>
> On Tue, Jan 27, 2026 at 7:07 AM Amul Sul<sulamul@gmail.com> wrote:
>
> In the attached version, I am using the WAL segment name as the hash
> key, which is much more straightforward. I have rewritten
> read_archive_wal_page(), and it looks much cleaner than before. The
> logic to discard irrelevant WAL files is still within
> get_archive_wal_entry. I added an explanation for setting cur_wal to
> NULL, which is now handled in the separate function I mentioned
> previously.
>
> Kindly have a look at the attached version; let me know if you are
> still not happy with the current approach for filtering/discarding
> irrelevant WAL segments. It isn't much different from the previous
> version, but I have tried to keep it in a separate routine for better
> code readability, with comments to make it easier to understand. I
> also added a comment for ArchivedWALFile.
>
> I feel like the division of labor between get_archive_wal_entry() and
> read_archive_wal_page() is odd. I noticed this in the last version,
> too, and it still seems to be the case. get_archive_wal_entry() first
> calls ArchivedWAL_lookup(). If that finds an entry, it just returns.
> If it doesn't, it loops until an entry for the requested file shows up
> and then returns it. Then control returns to read_archive_wal_page()
> which loops some more until we have all the data we need for the
> requested file. But it seems odd to me to have two separate loops
> here. I think that the first loop is going to call read_archive_file()
> until we find the beginning of the file that we care about and then
> the second one is going to call read_archive_file() some more until we
> have read enough of it to satisfy the request. It feels odd to me to
> do it that way, as if we told somebody to first wait until 9 o'clock
> and then wait another 30 minutes, instead of just telling them to wait
> until 9:30. I realize it's not quite the same thing, because apart
> from calling read_archive_file(), the two loops do different things,
> but I still think it looks odd.
>
> + /*
> + * Ignore if the timeline is different or the current segment is not
> + * the desired one.
> + */
> + XLogFromFileName(entry->fname, &curSegTimeline, &curSegNo, WalSegSz);
> + if (privateInfo->timeline != curSegTimeline ||
> + privateInfo->startSegNo > curSegNo ||
> + privateInfo->endSegNo < curSegNo ||
> + segno > curSegNo)
> + {
> + free_archive_wal_entry(entry->fname, privateInfo);
> + continue;
> + }
>
> The comment doesn't match the code. If it did, the test would be
> (privateInfo->timeline != curSegTimeline || segno != curSegno). But
> instead the segno test is > rather than !=, and the checks against
> startSegNo and endSegNo aren't explained at all. I think I understand
> why the segno test uses > rather than !=, but it's the point of the
> comment to explain things like that, rather than leaving the reader to
> guess. And I don't know why we also need to test startSegNo and
> endSegNo.
>
> I also wonder what the point is of doing XLogFromFileName() on the
> fname provided by the caller and then again on entry->fname. Couldn't
> you just compare the strings?
>
> Again, the division of labor is really odd here. It's the job of
> astreamer_waldump_content() to skip things that aren't WAL files at
> all, but it's the job of get_archive_wal_entry() to skip things that
> are WAL files but not the one we want. I disagree with putting those
> checks in completely separate parts of the code.
>
> Keeping the timeline and segment start-end range checks inside the
> archive streamer creates a circular dependency that cannot be resolved
> without a 'dirty hack'. We must read the first available WAL file page
> to determine the wal_segment_size before it can calculate the target
> segment range. Moving the checks inside the streamer would make it
> impossible to process that initial file, as the necessary filtering
> parameters --  would still be unknown which would need to be skipped
> for the first read somehow. What if later we realized that the first
> WAL file which was allowed to be streamed by skipping that check is
> irrelevant and doesn't fall under the start-end segment range?
>
> Please have a look at the attached version, specifically patch 0005.
> In astreamer_waldump_content(), I have moved the WAL file filtration
> check from get_archive_wal_entry(). This check will be skipped during
> the initial read in init_archive_reader(), which instead performs it
> explicitly once it determines the WAL segment size and the start/end
> segments.
>
> To access the WAL segment size inside astreamer_waldump_content(), I
> have moved the WAL segment size variable into the XLogDumpPrivate
> structure in the separate 0004 patch.
>
> Attached is an updated version including the aforesaid changes. It
> includes a new refactoring patch (0001) that moves the logic for
> identifying tar archives and their compression types from
> pg_basebackup and pg_verifybackup into a separate-reusable function,
> per a suggestion from Euler [1].  Additionally, I have added a test
> for the contrecord decoding to the main patch (now 0006).
>
> 1]http://postgr.es/m/3c8e7b02-2152-495a-a0b6-e37cf9286a70@app.fastmail.com
>
> Rebased against the latest master, fixed typos in code comments, and
> replaced palloc0 with palloc0_object.
>
> Hi Amul.
>
>
> I think this looks in pretty good shape.
>
> Thank you very much for looking at the patch.
>
> Attached are patches for a few things I think could be fixed. They are
> mostly self-explanatory. The TAP test fix is the only sane way I could
> come up with stopping the skip code you had from reporting a wildly
> inaccurate number of tests skipped. The sane way to do this from a
> Test::More perspective is a subtest, but unfortunately meson does not
> like subtest output, which is why we don't use it elsewhere, so the only
> way I could come up with was to split this out into a separate test. Of
> course, we might just say we don't care about the misreport, in which
> case we could just live with things as they are.
>
> I agree that the reported skip number was incorrect, and I have
> corrected it in the attached patch. I haven't applied your patch for
> the TAP test improvements yet because I wanted to double-check it
> first with you; the patch as it stood created duplicate tests already
> present in 001_basic.pl. To avoid this duplication, I have added a
> loop that performs tests for both plain and tar WAL directory inputs,
> similar to the approach used in pg_verifybackup for different
> compression type tests (e.g., 008_untar.pl, 010_client_untar.pl). I
> don't have any objection to doing so if you feel the duplication is
> acceptable, but I feel that using a loop for the tests in 001_basic.pl
> is a bit tidier. Let me know your thoughts.
>
> I will take a look.
>
> I'm ok, with doing it this way. It's just a bit fragile - if we add a
> test the number will be wrong. But maybe it's not worth worrying about.
>
> Everything else looks fairly good. The attached fixes a few relatively
> minor issues in v15. The main one is that it stops allocating/freeing a
> buffer every time we call read_archive_file() and instead adds a
> reusable buffer. It also adds back wal-directory as an undocumented
> alias of wal-path, to avoid breaking legacy scripts unnecessarily, and
> adds constness to the fname argument of pg_tar_compress_algorithm, as
> well as fixing some indentation and grammar issues.
>
> All in all I think we're in good shape.
>
> Thanks for the review. I have incorporated your suggested changes,
> with one exception: I have skipped the buffer reallocation code in
> read_archive_file(). Since we only handle two specific read sizes --
> XLOG_BLCKSZ and READ_CHUNK_SIZE (128 KB, we defined in
> archive_waldump.c) -- dynamic reallocation seems unnecessary. Instead,
> I moved the allocation to init_archive_reader(), which now initializes
> a buffer at READ_CHUNK_SIZE. I also added an assertion in
> read_archive_file() to ensure that no read request exceeds this
> allocated capacity.
>
> Kindly have a look at the attached version and let me know your thoughts.
>
>
> Looks pretty good. I have squashed them into three patches I think are committable. Also attached is a diff showing what's changed - mainly this:
>
> . --follow + tar archive rejected (pg_waldump.c) — new validation prevents a confusing pg_fatal when combining --follow with a tar archive
> . error messages split (archive_waldump.c) — the single "could not read file" error is now two distinct messages: "WAL segment is too short" (truncated file) vs "unexpected end of archive" (archive EOF) - Fixes an issue raised in review
> . hash table cleanup (archive_waldump.c) — free_archive_reader now iterates and frees all remaining hash entries and destroys the table
>

The final squashed version looks good to me, thank you. But, I would
like to propose splitting the 0001 patch into two separate commits: a
preparatory refactoring of the pg_waldump code and a standalone commit
that moves the tar archive detection and compression logic to a common
location, as the latter is an independent improvement to the existing
codebase. Additionally, since the test file refactoring was only kept
separate to facilitate the review and has already been reviewed, I
suggest merging those changes into the main feature patch i.e. 0002.
All other elements should remain in a single preparatory refactoring
patch for pg_waldump.

Attached is the version that includes the proposed split. No
additional changes to 0002 and 0003 patches.

Regards,
Amul

  49. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-18T15:16:35Z

    On Wed, Mar 18, 2026 at 5:15 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Wed, Mar 11, 2026 at 10:38 PM Andrew Dunstan <andrew@dunslane.net> wrote:
> > [...]
> > Looks pretty good. I have squashed them into three patches I think are committable. Also attached is a diff showing what's changed - mainly this:
> >
> > . --follow + tar archive rejected (pg_waldump.c) — new validation prevents a confusing pg_fatal when combining --follow with a tar archive
> > . error messages split (archive_waldump.c) — the single "could not read file" error is now two distinct messages: "WAL segment is too short" (truncated file) vs "unexpected end of archive" (archive EOF) - Fixes an issue raised in review
> > . hash table cleanup (archive_waldump.c) — free_archive_reader now iterates and frees all remaining hash entries and destroys the table
> >
>
> The final squashed version looks good to me, thank you. But, I would
> like to propose splitting the 0001 patch into two separate commits: a
> preparatory refactoring of the pg_waldump code and a standalone commit
> that moves the tar archive detection and compression logic to a common
> location, as the latter is an independent improvement to the existing
> codebase. Additionally, since the test file refactoring was only kept
> separate to facilitate the review and has already been reviewed, I
> suggest merging those changes into the main feature patch i.e. 0002.
> All other elements should remain in a single preparatory refactoring
> patch for pg_waldump.
>
> Attached is the version that includes the proposed split. No
> additional changes to 0002 and 0003 patches.
>

Added the two missing 'Reviewed-by' lines to the credit section of the
commit message and did a minor optimization in get_archive_wal_entry.

Regards,
Amul

  50. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-19T10:20:59Z

    On Wed, Mar 18, 2026 at 8:46 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Wed, Mar 18, 2026 at 5:15 PM Amul Sul <sulamul@gmail.com> wrote:
> >
> > On Wed, Mar 11, 2026 at 10:38 PM Andrew Dunstan <andrew@dunslane.net> wrote:
> > > [...]
> > > Looks pretty good. I have squashed them into three patches I think are committable. Also attached is a diff showing what's changed - mainly this:
> > >
> > > . --follow + tar archive rejected (pg_waldump.c) — new validation prevents a confusing pg_fatal when combining --follow with a tar archive
> > > . error messages split (archive_waldump.c) — the single "could not read file" error is now two distinct messages: "WAL segment is too short" (truncated file) vs "unexpected end of archive" (archive EOF) - Fixes an issue raised in review
> > > . hash table cleanup (archive_waldump.c) — free_archive_reader now iterates and frees all remaining hash entries and destroys the table
> > >
> >
> > The final squashed version looks good to me, thank you. But, I would
> > like to propose splitting the 0001 patch into two separate commits: a
> > preparatory refactoring of the pg_waldump code and a standalone commit
> > that moves the tar archive detection and compression logic to a common
> > location, as the latter is an independent improvement to the existing
> > codebase. Additionally, since the test file refactoring was only kept
> > separate to facilitate the review and has already been reviewed, I
> > suggest merging those changes into the main feature patch i.e. 0002.
> > All other elements should remain in a single preparatory refactoring
> > patch for pg_waldump.
> >
> > Attached is the version that includes the proposed split. No
> > additional changes to 0002 and 0003 patches.
> >
>
> Added the two missing 'Reviewed-by' lines to the credit section of the
> commit message and did a minor optimization in get_archive_wal_entry.
>

Attaching an updated version. It includes some tweaks to code
comments, adds an assert inside get_archive_wal_entry(), moves the
archive_read_buf_size declaration and usage into an assert-enabled
check, and makes a minor change to precheck_tar_backup_file() to
assign out-variables only after successful validation.

Regards,
Amul

  51. Re: pg_waldump: support decoding of WAL inside tarfile

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-19T20:48:27Z

    Hello!

Path is ignored with a positional argument, I think this is a bug?

This fails:

pg_waldump --path /wal/dir 000000010000000000000001

And this works:

pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8

+{
+ int fname_len = strlen(fname);
+

Shouldn't this use size_t?

+ /*
+ * Setup temporary directory to store WAL segments and set up an exit
+ * callback to remove it upon completion.
+ */
+ setup_tmpwal_dir(waldir);

Maybe this could be deferred to be created only on first use? If I
understand correctly, in a typical scenario waldump won't use this
temporary directory, yet it always creates it.

  52. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-20T11:31:00Z

    On Fri, Mar 20, 2026 at 2:18 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
>
> Hello!
>
> Path is ignored with a positional argument, I think this is a bug?
>
> This fails:
>
> pg_waldump --path /wal/dir 000000010000000000000001
>
> And this works:
>
> pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8
>

Good catch! I've fixed this in the attached version and updated a test
case to cover this scenario.

> +{
> + int fname_len = strlen(fname);
> +
>
> Shouldn't this use size_t?
>

Okay, that can be used. I’ve done the same in the attached version.

> + /*
> + * Setup temporary directory to store WAL segments and set up an exit
> + * callback to remove it upon completion.
> + */
> + setup_tmpwal_dir(waldir);
>
> Maybe this could be deferred to be created only on first use? If I
> understand correctly, in a typical scenario waldump won't use this
> temporary directory, yet it always creates it.

Yeah, that optimization can be done, but passing the waldir  -- which
is only used once -- to the point where the first temp file is created
would require quite a bit of code refactoring that doesn't seem to
offer much gain, IMO.

Regards,
Amul

  53. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-20T13:26:42Z

    On Fri, Mar 20, 2026 at 5:01 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Fri, Mar 20, 2026 at 2:18 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
> >
> > Hello!
> >
> > Path is ignored with a positional argument, I think this is a bug?
> >
> > This fails:
> >
> > pg_waldump --path /wal/dir 000000010000000000000001
> >
> > And this works:
> >
> > pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8
> >
>
> Good catch! I've fixed this in the attached version and updated a test
> case to cover this scenario.
>
> > +{
> > + int fname_len = strlen(fname);
> > +
> >
> > Shouldn't this use size_t?
> >
>
> Okay, that can be used. I’ve done the same in the attached version.
>
> > + /*
> > + * Setup temporary directory to store WAL segments and set up an exit
> > + * callback to remove it upon completion.
> > + */
> > + setup_tmpwal_dir(waldir);
> >
> > Maybe this could be deferred to be created only on first use? If I
> > understand correctly, in a typical scenario waldump won't use this
> > temporary directory, yet it always creates it.
>
> Yeah, that optimization can be done, but passing the waldir  -- which
> is only used once -- to the point where the first temp file is created
> would require quite a bit of code refactoring that doesn't seem to
> offer much gain, IMO.
>

Since Andrew also leans toward creating the directory only when
needed, I have reconsidered the approach. I think we can pass waldir
(the archive directory) via XLogDumpPrivate, and I’ve implemented that
in the attached version.

Regards,
Amul

  54. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-20T19:33:18Z

    On 2026-03-20 Fr 9:26 AM, Amul Sul wrote:
> On Fri, Mar 20, 2026 at 5:01 PM Amul Sul <sulamul@gmail.com> wrote:
>> On Fri, Mar 20, 2026 at 2:18 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
>>> Hello!
>>>
>>> Path is ignored with a positional argument, I think this is a bug?
>>>
>>> This fails:
>>>
>>> pg_waldump --path /wal/dir 000000010000000000000001
>>>
>>> And this works:
>>>
>>> pg_waldump --path /wal/dir --start 0/01000028 --end 0/010020F8
>>>
>> Good catch! I've fixed this in the attached version and updated a test
>> case to cover this scenario.
>>
>>> +{
>>> + int fname_len = strlen(fname);
>>> +
>>>
>>> Shouldn't this use size_t?
>>>
>> Okay, that can be used. I’ve done the same in the attached version.
>>
>>> + /*
>>> + * Setup temporary directory to store WAL segments and set up an exit
>>> + * callback to remove it upon completion.
>>> + */
>>> + setup_tmpwal_dir(waldir);
>>>
>>> Maybe this could be deferred to be created only on first use? If I
>>> understand correctly, in a typical scenario waldump won't use this
>>> temporary directory, yet it always creates it.
>> Yeah, that optimization can be done, but passing the waldir  -- which
>> is only used once -- to the point where the first temp file is created
>> would require quite a bit of code refactoring that doesn't seem to
>> offer much gain, IMO.
>>
> Since Andrew also leans toward creating the directory only when
> needed, I have reconsidered the approach. I think we can pass waldir
> (the archive directory) via XLogDumpPrivate, and I’ve implemented that
> in the attached version.
>

Thanks, committed with very minor tweaks.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  55. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-21T03:49:02Z

    Andrew Dunstan <andrew@dunslane.net> writes:
> Thanks, committed with very minor tweaks.

Buildfarm members batta and hachi don't like this very much.
They fail the pg_verifybackup tests like so:

# Running: pg_verifybackup --exit-on-error /home/admin/batta/buildroot/HEAD/pgsql.build/src/bin/pg_verifybackup/tmp_check/t_008_untar_primary_data/backup/server-backup
pg_waldump: error: could not find WAL in archive "base.tar.zst"
pg_verifybackup: error: WAL parsing failed for timeline 1

Only the zstd-compression case fails.  I've spent several hours trying
to reproduce this, without any luck, although I can get a similar
failure in only the gzip case if I build with --with-wal-blocksize=64.
I do not have an explanation for the seeming cross-platform
difference.  However after adding a lot of debug tracing, I believe
I see the bug, or at least a related bug.  This bit in
archive_waldump.c's init_archive_reader is where the error comes from:

    /*
     * Read until we have at least one full WAL page (XLOG_BLCKSZ bytes) from
     * the first WAL segment in the archive so we can extract the WAL segment
     * size from the long page header.
     */
    while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
    {
        if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
            pg_fatal("could not find WAL in archive \"%s\"",
                     privateInfo->archive_name);

        entry = privateInfo->cur_file;
    }

That looks plausible but is in fact utterly broken when there's not a
lot of WAL data in the archive, as there is not in this test case.
There are at least two problems:

1. read_archive_file reads some data from the source WAL archive and
shoves it into the astreamer decompression pipeline.  However, once it
runs out of source data, it just returns zero and we fail immediately.
This does not account for the possibility --- nay, certainty --- that
there is data queued inside the decompression pipeline.  So this
doesn't work if the data we need has been compressed into less than
XLOG_BLCKSZ worth of compressed data.  (I suppose that the seeming
cross-platform differences have to do with the effectiveness of the
compression algorithm, but I don't really understand why it'd not be
the same everywhere.)  We need to do astreamer_finalize once we run
out of source data.  I think the cleanest place to handle that would
be inside read_archive_file, but its return convention will need some
rework if we want to put it there (because rc == 0 shouldn't cause an
immediate failure if we were able to finalize some more data).  As an
ugly experiment I put an astreamer_finalize call into the rc == 0 path
of the above loop, but it still didn't work, because:

2. If the decompression pipeline reaches the end of the WAL file that
we want, the ASTREAMER_MEMBER_TRAILER case in
astreamer_waldump_content instantly resets privateInfo->cur_file to
NULL.  Then the loop in init_archive_reader cannot exit successfully,
and it will just read till the end of the archive and fail.

I see that of the three callers of read_archive_file, only
get_archive_wal_entry is aware of this possibility; but
init_archive_reader certainly needs to deal with it and I bet
read_archive_wal_page does too.  Moreover, get_archive_wal_entry's
solution looks to me like a fragile kluge that probably doesn't work
reliably either, the reason being that privateInfo->cur_file can
change multiple times during a single call to read_archive_file,
if the WAL data has been compressed sufficiently.  That whole API
seems to need some rethinking, not to mention better documentation
than the zero it has now.

While I'm bitching: this error message "could not find WAL in archive
\"%s\"" seems to me to be completely misleading and off-point.

			regards, tom lane

  56. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-21T06:19:28Z

    On Sat, Mar 21, 2026 at 9:19 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Andrew Dunstan <andrew@dunslane.net> writes:
> > Thanks, committed with very minor tweaks.
>
> Buildfarm members batta and hachi don't like this very much.
> They fail the pg_verifybackup tests like so:
>
> # Running: pg_verifybackup --exit-on-error /home/admin/batta/buildroot/HEAD/pgsql.build/src/bin/pg_verifybackup/tmp_check/t_008_untar_primary_data/backup/server-backup
> pg_waldump: error: could not find WAL in archive "base.tar.zst"
> pg_verifybackup: error: WAL parsing failed for timeline 1
>
> Only the zstd-compression case fails.  I've spent several hours trying
> to reproduce this, without any luck, although I can get a similar
> failure in only the gzip case if I build with --with-wal-blocksize=64.
> I do not have an explanation for the seeming cross-platform
> difference.  However after adding a lot of debug tracing, I believe
> I see the bug, or at least a related bug.  This bit in
> archive_waldump.c's init_archive_reader is where the error comes from:
>
>     /*
>      * Read until we have at least one full WAL page (XLOG_BLCKSZ bytes) from
>      * the first WAL segment in the archive so we can extract the WAL segment
>      * size from the long page header.
>      */
>     while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
>     {
>         if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
>             pg_fatal("could not find WAL in archive \"%s\"",
>                      privateInfo->archive_name);
>
>         entry = privateInfo->cur_file;
>     }
>
> That looks plausible but is in fact utterly broken when there's not a
> lot of WAL data in the archive, as there is not in this test case.
> There are at least two problems:
>

Thanks for the detailed debugging. I noticed the failure this morning
and had started investigating the issue, but in the meantime, I got
your helpful reply, which saved me a bunch of time and energy.

> 1. read_archive_file reads some data from the source WAL archive and
> shoves it into the astreamer decompression pipeline.  However, once it
> runs out of source data, it just returns zero and we fail immediately.
> This does not account for the possibility --- nay, certainty --- that
> there is data queued inside the decompression pipeline.  So this
> doesn't work if the data we need has been compressed into less than
> XLOG_BLCKSZ worth of compressed data.  (I suppose that the seeming
> cross-platform differences have to do with the effectiveness of the
> compression algorithm, but I don't really understand why it'd not be
> the same everywhere.)  We need to do astreamer_finalize once we run
> out of source data.  I think the cleanest place to handle that would
> be inside read_archive_file, but its return convention will need some
> rework if we want to put it there (because rc == 0 shouldn't cause an
> immediate failure if we were able to finalize some more data).  As an
> ugly experiment I put an astreamer_finalize call into the rc == 0 path
> of the above loop, but it still didn't work, because:
>
> 2. If the decompression pipeline reaches the end of the WAL file that
> we want, the ASTREAMER_MEMBER_TRAILER case in
> astreamer_waldump_content instantly resets privateInfo->cur_file to
> NULL.  Then the loop in init_archive_reader cannot exit successfully,
> and it will just read till the end of the archive and fail.
>
> I see that of the three callers of read_archive_file, only
> get_archive_wal_entry is aware of this possibility; but
> init_archive_reader certainly needs to deal with it and I bet
> read_archive_wal_page does too.  Moreover, get_archive_wal_entry's
> solution looks to me like a fragile kluge that probably doesn't work
> reliably either, the reason being that privateInfo->cur_file can
> change multiple times during a single call to read_archive_file,
> if the WAL data has been compressed sufficiently.  That whole API
> seems to need some rethinking, not to mention better documentation
> than the zero it has now.
>

I agree; init_archive_reader needs that handling, but
read_archive_wal_page doesn't need any fix. Since it only deals with
the current entry and already holds a reference to it, there is no
need to fetch it from the hash table again.

init_archive_reader has to scan the hash table because it doesn't
already have the specific WAL filename it is looking for, unlike
get_archive_wal_entry. Please have a look at the attached patch, which
tries to fix that.

> While I'm bitching: this error message "could not find WAL in archive
> \"%s\"" seems to me to be completely misleading and off-point.
>

I tried to improve that in the attached version.

regards,
Amul

  57. Re: pg_waldump: support decoding of WAL inside tarfile

    Michael Paquier <michael@paquier.xyz> — 2026-03-21T06:23:41Z

    On Fri, Mar 20, 2026 at 11:49:02PM -0400, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
> > Thanks, committed with very minor tweaks.
> 
> Buildfarm members batta and hachi don't like this very much.
> They fail the pg_verifybackup tests like so:
> 
> # Running: pg_verifybackup --exit-on-error /home/admin/batta/buildroot/HEAD/pgsql.build/src/bin/pg_verifybackup/tmp_check/t_008_untar_primary_data/backup/server-backup
> pg_waldump: error: could not find WAL in archive "base.tar.zst"
> pg_verifybackup: error: WAL parsing failed for timeline 1

I did not look at what's happening on the host, but it seems like a
safe bet to assume that we are not seeing many failures in the
buildfarm because we don't have many animals that have the idea to add
--with-zstd to their build configuration, like these two ones.
--
Michael

  58. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-21T06:34:00Z

    Michael Paquier <michael@paquier.xyz> writes:
> On Fri, Mar 20, 2026 at 11:49:02PM -0400, Tom Lane wrote:
>> Buildfarm members batta and hachi don't like this very much.

> I did not look at what's happening on the host, but it seems like a
> safe bet to assume that we are not seeing many failures in the
> buildfarm because we don't have many animals that have the idea to add
> --with-zstd to their build configuration, like these two ones.

That may be part of the story, but only part.  I spent a good deal of
time trying to reproduce batta & hachi's configurations locally, on
several different platforms, but still couldn't duplicate what they
are showing.

			regards, tom lane

  59. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-21T12:21:49Z

    On 2026-03-21 Sa 2:34 AM, Tom Lane wrote:
> Michael Paquier<michael@paquier.xyz> writes:
>> On Fri, Mar 20, 2026 at 11:49:02PM -0400, Tom Lane wrote:
>>> Buildfarm members batta and hachi don't like this very much.
>> I did not look at what's happening on the host, but it seems like a
>> safe bet to assume that we are not seeing many failures in the
>> buildfarm because we don't have many animals that have the idea to add
>> --with-zstd to their build configuration, like these two ones.
> That may be part of the story, but only part.  I spent a good deal of
> time trying to reproduce batta & hachi's configurations locally, on
> several different platforms, but still couldn't duplicate what they
> are showing.
>
> 			



Yeah, I haven't been able to reproduce it either. But while 
investigating I found a couple of issues. We neglected to add one of the 
tests to meson.build, and we neglected to close some files, causing 
errors on windows.

I also noticed a possible bug in astreamer, where the decompressor 
finalize functions send bbs_buffer.maxlen bytes to the next streamer 
when flushing remaining data at end-of-stream. This seems wrong because 
the buffer may only be partially filled with valid decompressed data. 
Possible patch for that attached. (But I don't think it's related to 
these failures).


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  60. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-21T14:45:46Z

    Andrew Dunstan <andrew@dunslane.net> writes:
> I also noticed a possible bug in astreamer, where the decompressor 
> finalize functions send bbs_buffer.maxlen bytes to the next streamer 
> when flushing remaining data at end-of-stream. This seems wrong because 
> the buffer may only be partially filled with valid decompressed data. 
> Possible patch for that attached. (But I don't think it's related to 
> these failures).

Ugh.  That's surely very broken, but how did we not notice?
Do all the consumers know enough to ignore garbage trailing data?

			regards, tom lane

  61. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-21T15:35:50Z

    On Sat, Mar 21, 2026 at 5:51 PM Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2026-03-21 Sa 2:34 AM, Tom Lane wrote:
>
> Michael Paquier <michael@paquier.xyz> writes:
>
> On Fri, Mar 20, 2026 at 11:49:02PM -0400, Tom Lane wrote:
>
> Buildfarm members batta and hachi don't like this very much.
>
> I did not look at what's happening on the host, but it seems like a
> safe bet to assume that we are not seeing many failures in the
> buildfarm because we don't have many animals that have the idea to add
> --with-zstd to their build configuration, like these two ones.
>
> That may be part of the story, but only part.  I spent a good deal of
> time trying to reproduce batta & hachi's configurations locally, on
> several different platforms, but still couldn't duplicate what they
> are showing.
>
>
>
>
>
> Yeah, I haven't been able to reproduce it either. But while investigating I found a couple of issues. We neglected to add one of the tests to meson.build, and we neglected to close some files, causing errors on windows.
>

While the proposed fix of closing the file pointer before returning is
correct, we also need to ensure the file is reopened in the next call
to spill any remaining buffered data. I’ve made a small update to
Andrew's 0001 patch to handle this. Also, changes to meson.build don't
seem to be needed as we haven't committed that file yet (unless I am
missing something).

I’ve also reattached the other patches so they don't get lost: v2-0002
is Andrew's patch for the archive streamer, and v2-0003 is the patch I
posted previously [1].


Regards,
Amul

1] http://postgr.es/m/CAAJ_b95L5J7bjRNDjRj6WgqFcQeaBD+JX3sAuxPA4uopqEThxA@mail.gmail.com

  62. Re: pg_waldump: support decoding of WAL inside tarfile

    amul sul <sulamul@gmail.com> — 2026-03-21T17:26:38Z

    On Sat, Mar 21, 2026 at 9:05 PM Amul Sul <sulamul@gmail.com> wrote:
>
> On Sat, Mar 21, 2026 at 5:51 PM Andrew Dunstan <andrew@dunslane.net> wrote:
> >
> >
> > On 2026-03-21 Sa 2:34 AM, Tom Lane wrote:
> >
> > Michael Paquier <michael@paquier.xyz> writes:
> >
> > On Fri, Mar 20, 2026 at 11:49:02PM -0400, Tom Lane wrote:
> >
> > Buildfarm members batta and hachi don't like this very much.
> >
> > I did not look at what's happening on the host, but it seems like a
> > safe bet to assume that we are not seeing many failures in the
> > buildfarm because we don't have many animals that have the idea to add
> > --with-zstd to their build configuration, like these two ones.
> >
> > That may be part of the story, but only part.  I spent a good deal of
> > time trying to reproduce batta & hachi's configurations locally, on
> > several different platforms, but still couldn't duplicate what they
> > are showing.
> >
> >
> >
> >
> >
> > Yeah, I haven't been able to reproduce it either. But while investigating I found a couple of issues. We neglected to add one of the tests to meson.build, and we neglected to close some files, causing errors on windows.
> >
>
> While the proposed fix of closing the file pointer before returning is
> correct, we also need to ensure the file is reopened in the next call
> to spill any remaining buffered data. I’ve made a small update to
> Andrew's 0001 patch to handle this. Also, changes to meson.build don't
> seem to be needed as we haven't committed that file yet (unless I am
> missing something).
>
> I’ve also reattached the other patches so they don't get lost: v2-0002
> is Andrew's patch for the archive streamer, and v2-0003 is the patch I
> posted previously [1].
>
>

On further thought, I don't think v2-0001 is the right patch. Consider
the case where we write a temporary file partially: if the next
segment required for decoding is that same segment,
TarWALDumpReadPage() will find the physical file present and continue
decoding, potentially triggering an error later due to the shorter
file.

I have attached the v3-0001 patch, which ensures that once we start
writing a temporary file, it should be finished before performing the
lookup. This ensures we don't leave a partial file on disk.

Updated patches are attached; 0002 and 0003 remain the same as before.


Regards,
Amul

  63. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-21T18:23:55Z

    I have made some progress on the question of how to reproduce
these failures.  If I do this:

diff --git a/src/bin/pg_waldump/archive_waldump.c b/src/bin/pg_waldump/archive_waldump.c
index b078c2d6960..c389a227be5 100644
--- a/src/bin/pg_waldump/archive_waldump.c
+++ b/src/bin/pg_waldump/archive_waldump.c
@@ -178,7 +178,7 @@ init_archive_reader(XLogDumpPrivate *privateInfo,
     */
    while (entry == NULL || entry->buf->len < XLOG_BLCKSZ)
    {
-       if (read_archive_file(privateInfo, XLOG_BLCKSZ) == 0)
+       if (read_archive_file(privateInfo, READ_CHUNK_SIZE) == 0)
            pg_fatal("could not find WAL in archive \"%s\"",
                     privateInfo->archive_name);

then I get the "could not find WAL in archive" failures in
pg_verifybackup's gzip and lz4 tests, but not zstd.  This happens
reproducibly even without any special hacks on XLOG_BLCKSZ or
wal_compression settings.  Of course, this is not exactly what's
happening on batta/hachi, because they fail on zstd and not the
other two.  But I think it confirms my theory that the problem
is essentially poor handling of EOF boundary conditions.

(Per discussion, there are other bugs here too; I don't mean
to minimize that aspect.)

			regards, tom lane

  64. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-21T19:31:10Z

    I don't like the v3 patches too much: in particular, they do nothing
for the failure-to-finalize bug I identified yesterday.  v3-0003
is on the right track but seems overcomplicated.  Here is my own
set of proposed patches, which I think will fix what we are seeing
in the buildfarm:

v4-0001 is Andrew's fix for incorrect decompressor finalization.

v4-0002 fixes read_archive_file() to do the missing finalize step.

v4-0003 fixes init_archive_reader() to not depend on cur_file.
This is closely allied to v3-0003 but simpler.  I also added
some commentary to pg_waldump.h about what it's safe to do with
cur_file.

get_archive_wal_entry() violates that advice and is pretty much
utterly broken IMO, because it still believes that it can use cur_file
in an incorrect way.  However, the impact of that is that it may fail
to flush some hashtable entries out to temp files (in case a single
read_archive_file() step reads more than one WAL file, which is
entirely possible with compression).  That is a performance issue but
it's not causing our buildfarm problems, so I left it untouched here.
But I don't think any of the patches proposed so far fix it properly.
What it should do should look more like the revised version of
init_archive_reader's loop: call read_archive_file(), then scan the
hash table for WAL entries we need to flush to files, then finally
return the desired WAL entry if it's present, else loop around.

			regards, tom lane

  65. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-21T22:24:07Z

    I wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> I also noticed a possible bug in astreamer, where the decompressor 
>> finalize functions send bbs_buffer.maxlen bytes to the next streamer 
>> when flushing remaining data at end-of-stream. This seems wrong because 
>> the buffer may only be partially filled with valid decompressed data. 
>> Possible patch for that attached. (But I don't think it's related to 
>> these failures).

> Ugh.  That's surely very broken, but how did we not notice?
> Do all the consumers know enough to ignore garbage trailing data?

I poked into this question and found that all of our existing uses
of the decompression astreamers are in front of astreamer_tar.c's
astreamer_tar_parser_ops.  Since a tar file has a trailer (two
zero blocks), it is not surprising that garbage after the trailer
will be ignored.  It might appear that astreamer_tar_parser_content
itself will complain about such extra data:

            case ASTREAMER_ARCHIVE_TRAILER:

                /*
                 * We've seen an end-of-archive indicator, so anything more is
                 * buffered and sent as part of the archive trailer. But we
                 * don't expect more than 2 blocks.
                 */
                astreamer_buffer_bytes(streamer, &data, &len, len);
                if (len > 2 * TAR_BLOCK_SIZE)
                    pg_fatal("tar file trailer exceeds 2 blocks");
                return;

But this code is itself buggy!  The astreamer_buffer_bytes call
is guaranteed to reduce "len" to zero, therefore that pg_fatal
call can never fire, no matter how much garbage we just stuffed
into the astreamer's buffer.

We might want to do

-               if (len > 2 * TAR_BLOCK_SIZE)
+               if (streamer->bbs_buffer.len > 2 * TAR_BLOCK_SIZE)

Unsurprisingly, applying this change to unmodified master results
in the pg_waldump and pg_verifybackup tests falling over.  More
surprisingly, they still fall over after applying your fix to the
decompressors, so there's some other source of garbage trailing
data.  I haven't figured out what.

However, I'm a bit loath to make that change even after fixing the
decompressors, because this discovery suggests that we may currently
be building tar files that contain garbage after the trailer.
And it seems to me that such tar files are perfectly legal anyway.
The POSIX spec for "ustar" format (found under pax(1)) says

    At the end of the archive file there shall be two 512-octet
    logical records filled with binary zeros, interpreted as an
    end-of-archive indicator.

    The logical records may be grouped for physical I/O operations, as
    described under the -bblocksize and -x ustar options. Each group
    of logical records may be written with a single operation
    equivalent to the write() function. On magnetic tape, the result
    of this write shall be a single tape physical block. The last
    physical block shall always be the full size, so logical records
    after the two zero logical records may contain undefined data.

That last sentence seems to say that you can put arbitrary garbage
after the trailer without invalidating the file.  Sure, it's a
holdover from magtape days, but that's how the spec is written.

Therefore, I think we might be best off to just remove this
nonfunctional error check, rather than "fix" it.  Another
idea which might be a bit better is to drop any data after
the standard trailer on the floor, instead of passing it on
as the code now does.

Thoughts?

			regards, tom lane

  66. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-22T04:24:47Z

    I wrote:
> Unsurprisingly, applying this change to unmodified master results
> in the pg_waldump and pg_verifybackup tests falling over.  More
> surprisingly, they still fall over after applying your fix to the
> decompressors, so there's some other source of garbage trailing
> data.  I haven't figured out what.

In the learn-something-new-every-day dept.: good ol' GNU tar itself
does that.  By default, it zero-pads its output to a multiple of 10kB
after it's written the required terminator.  Moreover, this behavior
is actually specified by POSIX:

  -x format
    Specify the output archive format. The pax utility shall support
    the following formats:
    ...
    ustar
      The tar interchange format; see the EXTENDED DESCRIPTION
      section. The default blocksize for this format for character
      special archive files shall be 10240. Implementations shall
      support all blocksize values less than or equal to 32256 that
      are multiples of 512.

So, astreamer_tar_parser_content's idea that it should disallow more
than 1024 bytes of trailer is completely wrong, which we would have
figured out long ago if the code attempting to enforce that weren't
completely broken.

You could argue that this means the tar files our existing utilities
create aren't POSIX-compliant.  I think it's all right though: we
can just say that we write these files with blocksize 1024 not
blocksize 10240, and tar-file readers are required to accept that
per the above spec text.

However, this discourages me from editorializing on the file trailer
emitted by whatever wrote the tar file we are reading.  I think
emitting it as-is is the most appropriate thing.  So we should just
get rid of astreamer_tar_parser_content's nonfunctional error check
and not change its behavior otherwise.

			regards, tom lane

  67. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-22T11:24:56Z

    On Sun, Mar 22, 2026 at 12:24 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

> I wrote:
> > Unsurprisingly, applying this change to unmodified master results
> > in the pg_waldump and pg_verifybackup tests falling over.  More
> > surprisingly, they still fall over after applying your fix to the
> > decompressors, so there's some other source of garbage trailing
> > data.  I haven't figured out what.
>
> In the learn-something-new-every-day dept.: good ol' GNU tar itself
> does that.  By default, it zero-pads its output to a multiple of 10kB
> after it's written the required terminator.  Moreover, this behavior
> is actually specified by POSIX:
>
>   -x format
>     Specify the output archive format. The pax utility shall support
>     the following formats:
>     ...
>     ustar
>       The tar interchange format; see the EXTENDED DESCRIPTION
>       section. The default blocksize for this format for character
>       special archive files shall be 10240. Implementations shall
>       support all blocksize values less than or equal to 32256 that
>       are multiples of 512.
>
> So, astreamer_tar_parser_content's idea that it should disallow more
> than 1024 bytes of trailer is completely wrong, which we would have
> figured out long ago if the code attempting to enforce that weren't
> completely broken.
>
> You could argue that this means the tar files our existing utilities
> create aren't POSIX-compliant.  I think it's all right though: we
> can just say that we write these files with blocksize 1024 not
> blocksize 10240, and tar-file readers are required to accept that
> per the above spec text.
>
> However, this discourages me from editorializing on the file trailer
> emitted by whatever wrote the tar file we are reading.  I think
> emitting it as-is is the most appropriate thing.  So we should just
> get rid of astreamer_tar_parser_content's nonfunctional error check
> and not change its behavior otherwise.
>
>
>
OK, patch 5 of this set does that. I reworked your previous patches 2 and 3
slightly - mostly additional comments, and fixing a bug in use
of sizeof(XLogLongPageHeader). Patch 4 here tries to fix the wrong use of
cur_file in get_archive_wal_entry()

cheers

andrew

  68. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-22T17:29:44Z

    Andrew Dunstan <andrew@dunslane.net> writes:
> OK, patch 5 of this set does that. I reworked your previous patches 2 and 3
> slightly - mostly additional comments, and fixing a bug in use
> of sizeof(XLogLongPageHeader). Patch 4 here tries to fix the wrong use of
> cur_file in get_archive_wal_entry()

A few nits:

0001: I think we need to rewrite the commit message now that we've
run the unknowns to ground.  Perhaps like

Send the correct amount of data to the next astreamer, not the
whole allocated buffer size.  This bug escaped detection because
in present uses the next astreamer is always a tar-file parser
which is insensitive to trailing garbage.  But that may not
always be true.

Also, I think we ought to back-patch 0001, just in case we have
misunderstood the scope of usage or somebody tries to make use of
back-branch code for a new purpose.  This is a bit tedious because
before f80b09bac/3c9056981 the code was somewhere else and used
different names.  But the bugs are clearly there, back to v15.

0002, 0003, 0005: LGTM.

0004: Don't like this at all.  It fails to respond to the point
I was trying to make: the code will only consider spilling hash
entries that are the cur_file at some read_archive_file boundary.
Also, it seems to be doubling down on the already over-complicated
and probably under-correct goal of spilling entries that are still
in progress of being read.  We can make this function far simpler
and more obviously correct if we just accept that we'll read a
WAL file completely before spilling it.  See my proposed
alternative to 0004, attached.

			regards, tom lane

  69. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-22T18:17:53Z

    I wrote:
> ... We can make this function far simpler
> and more obviously correct if we just accept that we'll read a
> WAL file completely before spilling it.  See my proposed
> alternative to 0004, attached.

Actually, we can make that better yet by not expecting
get_archive_wal_entry to clean up after init_archive's
failure to free all irrelevant hashtable entries.
Better version attached.

			regards, tom lane

  70. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-22T21:19:09Z

    On Sun, Mar 22, 2026 at 2:17 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

> I wrote:
> > ... We can make this function far simpler
> > and more obviously correct if we just accept that we'll read a
> > WAL file completely before spilling it.  See my proposed
> > alternative to 0004, attached.
>
> Actually, we can make that better yet by not expecting
> get_archive_wal_entry to clean up after init_archive's
> failure to free all irrelevant hashtable entries.
> Better version attached.
>
>
>

Yeah, this looks good.  I know we also still need to do something about
rmtree trying to remove files we haven't closed. But what we have so far in
this set LGTM. If you want to push this I'm good, otherwise I'll look at it
tomorrow or Tuesday.

cheers

andrew

  71. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-22T21:39:26Z

    Andrew Dunstan <andrew@dunslane.net> writes:
> Yeah, this looks good.  I know we also still need to do something about
> rmtree trying to remove files we haven't closed. But what we have so far in
> this set LGTM. If you want to push this I'm good, otherwise I'll look at it
> tomorrow or Tuesday.

WFM.  I'd like to get the buildfarm green again, so I'll work on
pushing this stuff now.

			regards, tom lane

  72. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-22T23:13:11Z

    Everything's pushed, although it looks like it'll be a few hours
before we see results from batta or hachi.

In the meantime, I have one more change that I'd like to propose in
archive_waldump.c: I think we should drop read_archive_file's "count"
parameter and just always try to fill archive_read_buf completely,
as attached.  I don't believe that using a variable count is buying
anything.  I get that the idea was to not load more data than we have
to before init_archive_reader can identify the segment size and start
filtering data we don't need.  But I doubt that that helps a whole
lot: I think most of the cost in this pipeline is going into
decompression and tar format parsing, not into copying data into the
hashtable.  Also, I noticed while I was investigating the bugs we
just fixed that the tar file very likely contains a ton of other
data before the first WAL file.  In the regression test scenarios
it seems to reliably have all of the cluster's actual table data
ahead of the WAL directory.  By using a small read count, we are
slowing down scanning of all that data.  Maybe not by much, I've
not tried to measure it.  But I think the case for using a variable
count is very weak.

Even if you don't like that change, I think we should apply the
parts of the attached that change archive_read_buf_size into an
always-present, always-filled field of XLogDumpPrivate.  Having
its presence be dependent on USE_ASSERT_CHECKING is just asking
for trouble, ie different .c files having a different opinion
of whether it's there.

			regards, tom lane

  73. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-23T03:02:20Z

    I wrote:
> Everything's pushed, although it looks like it'll be a few hours
> before we see results from batta or hachi.

batta just went green, so I'm feeling optimistic that we've correctly
diagnosed and fixed the problem.

Meanwhile, I noticed one more thing that could use fixing: while
astreamer_zstd.c is careful to use a reasonably-sized output
bbs_buffer (about 256kB looks like), the decompressors in
astreamer_gzip.c and astreamer_lz4.c just leave bbs_buffer's size
at the default 1kB.  This means we're pushing decompressed data
to the next astreamer in very small units in those cases.

I suspect strongly that this difference in buffer sizes has a lot
to do with the observed results that only zstd failed on batta/hachi
while only the other two failed in the test cases that I've been
able to reproduce here.  I'm too lazy to try to run that to ground
though, especially since it's mostly moot now.  But I think that
on performance grounds, we ought to use a reasonable buffer size
for all three decompressors.

I also noticed that astreamer_zstd.c was using ZSTD_DStreamOutSize()
as the target buffer size for both compression and decompression,
although the libzstd API provides a separate function
ZSTD_CStreamOutSize() for the compression case.  Those two functions
produce the same result (256K) on my machine, so this seems to be just
a cosmetic issue, but it still seems pretty tin-eared to not use the
functions as specified.

Proposed patch attached.  There might be an argument for using some
other size than 256K for the other two decompressors, but my
inclination is to try to make all three use roughly the same block
size.  (See also 66ec01dc4.)

			regards, tom lane

  74. Re: pg_waldump: support decoding of WAL inside tarfile

    Michael Paquier <michael@paquier.xyz> — 2026-03-24T03:11:44Z

    On Sun, Mar 22, 2026 at 11:02:20PM -0400, Tom Lane wrote:
> Proposed patch attached.  There might be an argument for using some
> other size than 256K for the other two decompressors, but my
> inclination is to try to make all three use roughly the same block
> size.  (See also 66ec01dc4.)

The buildfarm has switched mostly to green, except on this one:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42
--
Michael

  75. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-25T17:25:21Z

    Michael Paquier <michael@paquier.xyz> writes:
> The buildfarm has switched mostly to green, except on this one:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42

Yeah, there are some other weird failures on other machines too.
There's also the problem we already knew about of FD leakage breaking
cleanup of the temp file directory on Windows.

I wrote a patch to fix the FD leakage problem (v8-0001 attached).
I don't have Windows at hand, but I tested it by dint of having
the atexit callback invoke "lsof" to see if there were any open
files in the temp directory.

I also occasionally saw some of the weird errors mentioned above.
After much debugging, I believe that the issue is that
archive_waldump.c is unaware that inserting or deleting entries
in a simplehash.h hash table can cause other entries to move.
That can break privateInfo->cur_file, and it can also break
read_archive_wal_page which thought it could just re-use its
entry pointer after calling read_archive_file.  v8-0002 attached
fixes that, and I'm not seeing weird failures anymore.

Two additional thoughts:

1. The amount of data that pg_waldump's TAP tests use is not
sufficient to trigger these problems with any degree of reliability.
I'm hesitant to make the tests run longer, but clearly we do not
have adequate coverage now.

2. I didn't do it here, but I urgently think we should rip out
read_archive_wal_page's stanza that truncates the entry's
"buf" string (the "if (privateInfo->decoding_started)" part).
My faith in this code in general is at rock bottom, and my faith in
the extent to which we've tested it is somewhere below ground level.
I don't think we need rickety optimizations that serve only to
keep the active hashtable entry to less than 16MB, when we're going
to reclaim that space altogether as soon as we've finished dumping
that segment.  This truncation scares me because it adds a whole
'nother level of poorly-documented complexity to the invariants
around what is in entry->buf.  Also, while we theoretically should
not need to spill the entry after this point, if we did we would
write a corrupted spill file.

			regards, tom lane

  76. Re: pg_waldump: support decoding of WAL inside tarfile

    Andres Freund <andres@anarazel.de> — 2026-03-25T17:28:26Z

    Hi,

On 2026-03-24 12:11:44 +0900, Michael Paquier wrote:
> On Sun, Mar 22, 2026 at 11:02:20PM -0400, Tom Lane wrote:
> > Proposed patch attached.  There might be an argument for using some
> > other size than 256K for the other two decompressors, but my
> > inclination is to try to make all three use roughly the same block
> > size.  (See also 66ec01dc4.)
>
> The buildfarm has switched mostly to green, except on this one:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42

I think there's a few more failues. Fairywren regularly fails, including in a
run from today.

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2026-03-25%2003%3A48%3A06

There also are a lot of CI failures. E.g.

https://cirrus-ci.com/task/6153854431002624
https://api.cirrus-ci.com/v1/artifact/task/6153854431002624/testrun/build/testrun/pg_waldump/001_basic/log/regress_log_001_basic
# Running: pg_waldump --path C:\msys64\tmp\tNfU5IfQ4a/pg_wal.tar.gz --start 0/01806F48 --end 0/03093BD8
[22:46:25.358](3.991s) not ok 160 - runs with path option and start and end locations: exit code 0
[22:46:25.363](0.005s) #   Failed test 'runs with path option and start and end locations: exit code 0'
#   at C:/cirrus/src/bin/pg_waldump/t/001_basic.pl line 399.
[22:46:25.364](0.001s) ok 161 - runs with path option and start and end locations: no stderr
[22:46:25.365](0.001s) not ok 162 - runs with path option and start and end locations: matches
[22:46:25.365](0.000s) #   Failed test 'runs with path option and start and end locations: matches'
#   at C:/cirrus/src/bin/pg_waldump/t/001_basic.pl line 399.
[22:46:25.366](0.000s) #                   ''
#     doesn't match '(?^:.)'


I was first suspecting that this is due to

commit 1c162c965a1
Author: Fujii Masao <fujii@postgresql.org>
Date:   2026-03-24 22:33:09 +0900

    Report detailed errors from XLogFindNextRecord() failures.

but there are afaict failures from before that:
https://cirrus-ci.com/task/5501609960013824
which is for 4019f725f5d, preceding 1c162c965a1
and
https://cirrus-ci.com/task/5317196043255808


It does feel however the failure frequency has increased substantially:
https://cirrus-ci.com/github/postgres/postgres/master

Greetings,

Andres Freund

  77. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-03-28T21:36:06Z

    On Thu, Mar 26, 2026 at 6:28 AM Andres Freund <andres@anarazel.de> wrote:
> On 2026-03-24 12:11:44 +0900, Michael Paquier wrote:
> > On Sun, Mar 22, 2026 at 11:02:20PM -0400, Tom Lane wrote:
> > > Proposed patch attached.  There might be an argument for using some
> > > other size than 256K for the other two decompressors, but my
> > > inclination is to try to make all three use roughly the same block
> > > size.  (See also 66ec01dc4.)
> >
> > The buildfarm has switched mostly to green, except on this one:
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hoatzin&dt=2026-03-23%2006%3A00%3A42
>
> I think there's a few more failues. Fairywren regularly fails, including in a
> run from today.

This fails 100% of the time on my machine, even after e9d72348 and ff84efe4, eg:

# Running: pg_waldump --path /tmp/D8WG1Sv2HE/pg_wal.tar --start
0/017A2610 --end 0/02093848
[09:43:29.288](0.148s) not ok 104 - runs with path option and start
and end locations: exit code 0
[09:43:29.289](0.001s) #   Failed test 'runs with path option and
start and end locations: exit code 0'
#   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
line 402.
[09:43:29.290](0.001s) not ok 105 - runs with path option and start
and end locations: no stderr
[09:43:29.291](0.001s) #   Failed test 'runs with path option and
start and end locations: no stderr'
#   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
line 402.
[09:43:29.291](0.000s) #          got: 'pg_waldump: error: could not
find WAL "000000010000000000000002" in archive "pg_wal.tar"
# '

I can see that it is wrong about the contents of the tar file:

$ pg_waldump --path _tmp_H_1gv81G1L_pg_wal.tar --start 0/017A2610
--end 0/020934F8 2>&1 | tail -3
rmgr: Hash        len (rec/tot):     72/    72, tx:        720, lsn:
0/01FFC1B8, prev 0/01FFC178, desc: INSERT off 40, blkref #0: rel
1663/5/16397 blk 2, blkref #1: rel
1663/5/16397 blk 0
rmgr: Transaction len (rec/tot):     46/    46, tx:        720, lsn:
0/01FFC200, prev 0/01FFC1B8, desc: COMMIT 2026-03-29 10:15:24.112967
NZDT
pg_waldump: error: could not find WAL "000000010000000000000002" in
archive "_tmp_H_1gv81G1L_pg_wal.tar"

$ tar tvf _tmp_H_1gv81G1L_pg_wal.tar
drwx------  0 tmunro tmunro      0 Mar 29 10:15 archive_status/
-rw-------  0 tmunro tmunro      0 Mar 29 10:15
archive_status/000000010000000000000002.ready
-rw-------  0 tmunro tmunro      0 Mar 29 10:15
archive_status/000000010000000000000001.ready
drwx------  0 tmunro tmunro      0 Mar 29 10:08 summaries/
-rw-------  0 tmunro tmunro 16777216 Mar 29 10:15 000000010000000000000002
-rw-------  0 tmunro tmunro 16777216 Mar 29 10:15 000000010000000000000001
-rw-------  0 tmunro tmunro 16777216 Mar 29 10:15 000000010000000000000003

It seems like the place we'd be looking for the file is in
astreamer_tar_header(), so I added in some caveman debugging:

    /*
     * Parse key fields out of the header.
     */
fprintf(stderr, "XXXX [%s] XXXX\n", &buffer[TAR_OFFSET_NAME]);
    strlcpy(member->pathname, &buffer[TAR_OFFSET_NAME], MAXPGPATH);
    if (member->pathname[0] == '\0')
        pg_fatal("tar member has empty name");

Now I see:

XXXX [archive_status/] XXXX
XXXX [archive_status/000000010000000000000002.ready] XXXX
XXXX [archive_status/000000010000000000000001.ready] XXXX
XXXX [summaries/] XXXX
XXXX [PaxHeader/000000010000000000000002] XXXX
XXXX [GNUSparseFile.0/000000010000000000000002] XXXX
XXXX [000000010000000000000001] XXXX
rmgr: XLOG        len (rec/tot):     30/    30, tx:          0, lsn:
0/017A2610, prev 0/017A25F0, desc: NEXTOID 24576
rmgr: Standby     len (rec/tot):     42/    42, tx:        692, lsn:
0/017A2630, prev 0/017A2610, desc: LOCK xid 692 db 5 rel 16384
rmgr: Storage     len (rec/tot):     42/    42, tx:        692, lsn:
0/017A2660, prev 0/017A2630, desc: CREATE base/5/16384
... lots more normal output ...
rmgr: Hash        len (rec/tot):     72/    72, tx:        720, lsn:
0/01FFBED8, prev 0/01FFBE98, desc: INSERT off 97, blkref #0: rel
1663/5/16397 blk 2, blkref #1: rel
1663/5/16397 blk 0
rmgr: Heap        len (rec/tot):    575/   575, tx:        720, lsn:
0/01FFBF20, prev 0/01FFBED8, desc: INSERT off: 12, flags: 0x08, blkref
#0: rel 1663/5/16393 blk 52
rmgr: Btree       len (rec/tot):     64/    64, tx:        720,
lsn:XXXX [PaxHeader/000000010000000000000003] XXXX
XXXX [GNUSparseFile.0/000000010000000000000003] XXXX
 0/01FFC178, prev 0/01FFBF20, desc: INSERT_LEAF off: 344, blkref #0:
rel 1663/5/16396 blk 2
rmgr: Hash        len (rec/tot):     72/    72, tx:        720, lsn:
0/01FFC1B8, prev 0/01FFC178, desc: INSERT off 40, blkref #0: rel
1663/5/16397 blk 2, blkref #1: rel
1663/5/16397 blk 0
rmgr: Transaction len (rec/tot):     46/    46, tx:        720, lsn:
0/01FFC200, prev 0/01FFC1B8, desc: COMMIT 2026-03-29 10:15:24.112967
NZDT
pg_waldump: error: could not find WAL "000000010000000000000002" in
archive "_tmp_H_1gv81G1L_pg_wal.tar"

Seems like it already stepped over 000000010000000000000002 earlier?
Could it be a table-of-contents order dependency bug or something like
that?

  78. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-28T21:49:29Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> This fails 100% of the time on my machine, even after e9d72348 and ff84efe4, eg:

> # Running: pg_waldump --path /tmp/D8WG1Sv2HE/pg_wal.tar --start
> 0/017A2610 --end 0/02093848
> [09:43:29.288](0.148s) not ok 104 - runs with path option and start
> and end locations: exit code 0
> [09:43:29.289](0.001s) #   Failed test 'runs with path option and
> start and end locations: exit code 0'
> #   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
> line 402.
> [09:43:29.290](0.001s) not ok 105 - runs with path option and start
> and end locations: no stderr
> [09:43:29.291](0.001s) #   Failed test 'runs with path option and
> start and end locations: no stderr'
> #   at /home/tmunro/projects/postgresql/src/bin/pg_waldump/t/001_basic.pl
> line 402.
> [09:43:29.291](0.000s) #          got: 'pg_waldump: error: could not
> find WAL "000000010000000000000002" in archive "pg_wal.tar"
> # '

Huh.  What is your platform exactly?  Maybe more to the point,
what is the tar program you're using?

> Seems like it already stepped over 000000010000000000000002 earlier?
> Could it be a table-of-contents order dependency bug or something like
> that?

If you look at the TAP script, you'll see that it tries to randomize
the order of the entries in the tar file (see sub generate_archive).
So if that's the problem, it shouldn't reproduce 100%, and also we
should be seeing lots of freckles on the buildfarm.  We're not, so
there must be something off-the-beaten-track about your test
environment.

I'm certainly more than prepared to believe there's still bugs
lurking here, but we need to figure out why you're seeing it.

			regards, tom lane

  79. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-03-28T22:08:43Z

    On Sun, Mar 29, 2026 at 10:49 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Huh.  What is your platform exactly?  Maybe more to the point,
> what is the tar program you're using?

FreeBSD tar 3.8.2.

> > Seems like it already stepped over 000000010000000000000002 earlier?
> > Could it be a table-of-contents order dependency bug or something like
> > that?
>
> If you look at the TAP script, you'll see that it tries to randomize
> the order of the entries in the tar file (see sub generate_archive).
> So if that's the problem, it shouldn't reproduce 100%, and also we
> should be seeing lots of freckles on the buildfarm.  We're not, so
> there must be something off-the-beaten-track about your test
> environment.

Right, I see now.  There is something different about
000000010000000000000002 though: it doesn't seem to have a normal
(non-PAX, non-GNU) TOC entry, unlike000000010000000000000001.  Trying
to figure out why...

  80. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-28T22:18:03Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> It seems like the place we'd be looking for the file is in
> astreamer_tar_header(), so I added in some caveman debugging:

> XXXX [PaxHeader/000000010000000000000002] XXXX
> XXXX [GNUSparseFile.0/000000010000000000000002] XXXX

After a bit of research, I see the problem: your tar has decided to
store 000000010000000000000002 as a sparse file, and evidently it
inserts "GNUSparseFile.0/" into the file name when it does that.
Our tar-parsing logic will not recognize that name as being one
of the WAL files it wants; and even if it did, it doesn't know
how to de-sparse the file contents, so it would get the wrong
output.

I'm pretty disinclined to try to support sparse files here,
especially with ~10 days till feature freeze.

However ... I do not find any indication in the GNU tar docs
that it produces sparse files by default.  It looks like you
need to say -S/--sparse to make that happen.  Maybe you have
a version that's been hacked to make that the default?

			regards, tom lane

  81. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-28T22:36:57Z

    I wrote:
> However ... I do not find any indication in the GNU tar docs
> that it produces sparse files by default.  It looks like you
> need to say -S/--sparse to make that happen.  Maybe you have
> a version that's been hacked to make that the default?

Bleah.  Digging in the man pages at freebsd.org, I read

    --read-sparse
	       (c, r, u	modes only) Read sparse	file  information  from	 disk.
	       This  is	the reverse of --no-read-sparse	and the	default	behav-
	       ior.

It's apparently been there and been default since FreeBSD 13.1.
This leads one to wonder how come BF member dikkop is managing
to run this test successfully.  I speculate that it's using a
filesystem type that doesn't do sparse files (cc'ing Vondra
for confirmation on that).

It looks like to make this test stable on modern FreeBSD,
we need to see if tar accepts --no-read-sparse and use that
switch if so.

			regards, tom lane

  82. Re: pg_waldump: support decoding of WAL inside tarfile

    Tomas Vondra <tomas@vondra.me> — 2026-03-28T22:57:37Z

    On 3/28/26 23:36, Tom Lane wrote:
> I wrote:
>> However ... I do not find any indication in the GNU tar docs
>> that it produces sparse files by default.  It looks like you
>> need to say -S/--sparse to make that happen.  Maybe you have
>> a version that's been hacked to make that the default?
> 
> Bleah.  Digging in the man pages at freebsd.org, I read
> 
>     --read-sparse
> 	       (c, r, u	modes only) Read sparse	file  information  from	 disk.
> 	       This  is	the reverse of --no-read-sparse	and the	default	behav-
> 	       ior.
> 
> It's apparently been there and been default since FreeBSD 13.1.
> This leads one to wonder how come BF member dikkop is managing
> to run this test successfully.  I speculate that it's using a
> filesystem type that doesn't do sparse files (cc'ing Vondra
> for confirmation on that).
> 

It's running on ufs. But I think the explanation is very simple. We had
a short power outage on Thursday, and the FreeBSD machine failed to boot
properly after the power was restored. IIUC this test is new, right?

I fixed the machine, it'll start running the tests in a couple minutes.


regards

-- 
Tomas Vondra

  83. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-28T23:12:11Z

    Tomas Vondra <tomas@vondra.me> writes:
> On 3/28/26 23:36, Tom Lane wrote:
>> It's apparently been there and been default since FreeBSD 13.1.
>> This leads one to wonder how come BF member dikkop is managing
>> to run this test successfully.  I speculate that it's using a
>> filesystem type that doesn't do sparse files (cc'ing Vondra
>> for confirmation on that).

> It's running on ufs. But I think the explanation is very simple. We had
> a short power outage on Thursday, and the FreeBSD machine failed to boot
> properly after the power was restored. IIUC this test is new, right?

Not that new, it dates to b15c15139, about a week ago.

I've reproduced Thomas' failure on a local FreeBSD 15.0 image
using zfs, and confirmed that this cowboy hack fixes it:

diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 8bb8fa225f6..2904f4ef118 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -346,7 +346,7 @@ sub generate_archive
        # move into the WAL directory before archiving files
        my $cwd = getcwd;
        chdir($directory) || die "chdir: $!";
-       command_ok([$tar, $compression_flags, $archive, @files]);
+       command_ok([$tar, $compression_flags, $archive, '--no-read-sparse', @files]);
        chdir($cwd) || die "chdir: $!";
 }

Of course we can't commit that, but maybe somebody wants to
product-ize it.  (Not me, at least not today.)

			regards, tom lane

  84. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-03-28T23:15:39Z

    On Sun, Mar 29, 2026 at 11:37 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I wrote:
> > However ... I do not find any indication in the GNU tar docs
> > that it produces sparse files by default.  It looks like you
> > need to say -S/--sparse to make that happen.  Maybe you have
> > a version that's been hacked to make that the default?
>
> Bleah.  Digging in the man pages at freebsd.org, I read
>
>     --read-sparse
>                (c, r, u modes only) Read sparse file  information  from  disk.
>                This  is the reverse of --no-read-sparse and the default behav-
>                ior.
>
> It's apparently been there and been default since FreeBSD 13.1.
> This leads one to wonder how come BF member dikkop is managing
> to run this test successfully.  I speculate that it's using a
> filesystem type that doesn't do sparse files (cc'ing Vondra
> for confirmation on that).
>
> It looks like to make this test stable on modern FreeBSD,
> we need to see if tar accepts --no-read-sparse and use that
> switch if so.

Yeah.  Here's my attempt at perl.

I think your Mac probably has a similar tar program BTW... but apfs
probably doesn't go around making holes visible to lseek()
automatically or at least as eagerly as my ZFS system.

  85. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-28T23:34:35Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Sun, Mar 29, 2026 at 11:37 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It looks like to make this test stable on modern FreeBSD,
>> we need to see if tar accepts --no-read-sparse and use that
>> switch if so.

> Yeah.  Here's my attempt at perl.

Andrew might have some stylistic suggestions, but this looks
plausible to me.

> I think your Mac probably has a similar tar program BTW... but apfs
> probably doesn't go around making holes visible to lseek()
> automatically or at least as eagerly as my ZFS system.

Interesting ... on my Tahoe laptop, I see

$ tar --version
bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8 

There is nothing in "tar --help" about sparse files, but
experimentation shows that it accepts --[no-]read-sparse.
Can't easily tell if the switch actually does anything.

			regards, tom lane

  86. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-03-29T01:59:21Z

    

> On Mar 28, 2026, at 7:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Thomas Munro <thomas.munro@gmail.com> writes:
>>> On Sun, Mar 29, 2026 at 11:37 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> It looks like to make this test stable on modern FreeBSD,
>>> we need to see if tar accepts --no-read-sparse and use that
>>> switch if so.
> 
>> Yeah.  Here's my attempt at perl.
> 
> Andrew might have some stylistic suggestions, but this looks
> plausible to me.

Looks basically ok to me, although I wouldn’t make the variable name all caps.

Cheers

Andrew

  87. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-03-29T02:01:24Z

    On Sun, Mar 29, 2026 at 2:59 PM Andrew Dunstan <andrew@dunslane.net> wrote:
> Looks basically ok to me, although I wouldn’t make the variable name all caps.

Will fix and push.  Thanks!

  88. Re: pg_waldump: support decoding of WAL inside tarfile

    Tomas Vondra <tomas@vondra.me> — 2026-03-29T13:33:10Z

    
On 3/29/26 00:12, Tom Lane wrote:
> Tomas Vondra <tomas@vondra.me> writes:
>> On 3/28/26 23:36, Tom Lane wrote:
>>> It's apparently been there and been default since FreeBSD 13.1.
>>> This leads one to wonder how come BF member dikkop is managing
>>> to run this test successfully.  I speculate that it's using a
>>> filesystem type that doesn't do sparse files (cc'ing Vondra
>>> for confirmation on that).
> 
>> It's running on ufs. But I think the explanation is very simple. We had
>> a short power outage on Thursday, and the FreeBSD machine failed to boot
>> properly after the power was restored. IIUC this test is new, right?
> 
> Not that new, it dates to b15c15139, about a week ago.
> 
> I've reproduced Thomas' failure on a local FreeBSD 15.0 image
> using zfs, and confirmed that this cowboy hack fixes it:
> 

Interesting. Then I guess it has to be due to some difference in ufs vs.
zfs, when handling sparse files. It might be useful to add a bit more
variation here, and switch some of the animals to non-default
filesystems (not just the FreeBSD ones, which we seem to have only two
that run reasonably often). I'd bet most of the linux systems run on
ext4/xfs, few on btrfs/zfs.

regards

-- 
Tomas Vondra

  89. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-03-29T22:11:50Z

    On Mon, Mar 30, 2026 at 2:33 AM Tomas Vondra <tomas@vondra.me> wrote:
> On 3/29/26 00:12, Tom Lane wrote:
> > I've reproduced Thomas' failure on a local FreeBSD 15.0 image
> > using zfs, and confirmed that this cowboy hack fixes it:
> >
>
> Interesting. Then I guess it has to be due to some difference in ufs vs.
> zfs, when handling sparse files. It might be useful to add a bit more
> variation here, and switch some of the animals to non-default
> filesystems (not just the FreeBSD ones, which we seem to have only two
> that run reasonably often). I'd bet most of the linux systems run on
> ext4/xfs, few on btrfs/zfs.

UFS does have sparse files (its ancestor invented them some time
around (time_t) 0), it just doesn't make them unless you tell it to.
PostgreSQL only does that if you set wal_init_zero=false.

ZFS is different because it creates holes automagically when you write
zeroes, at least if compression is enabled so it has to scan all your
bytes anyway.

I was curious to know if BTRFS does that too, or hides
zero-compression at some lower invisible level:

$ echo "hello" > 1MB-sparse.dat
$ truncate -s 512KB 1MB-sparse.dat
$ echo "world" >> 1MB-sparse.dat
$ truncate -s 1MB 1MB-sparse.dat
$ ls -l 1MB-sparse.dat
-rw-rw-r-- 1 tmunro tmunro 1000000 Mar 30 10:11 1MB-sparse.dat
$ du -hs 1MB-sparse.dat
8.0K    1MB-sparse.dat
$ strace tar -S -cf foo.tar 1MB-sparse.dat 2>&1 | grep seek
lseek(4, 0, SEEK_DATA)                  = 0
lseek(4, 0, SEEK_HOLE)                  = 4096
lseek(4, 4096, SEEK_DATA)               = 512000
lseek(4, 512000, SEEK_HOLE)             = 516096
lseek(4, 516096, SEEK_DATA)             = -1 ENXIO (No such device or address)

... so that's a yes, lseek sees holes that we didn't ask it to make,
just like on ZFS, but the rest of this trace of GNU tar -S -cf is
interesting:

lseek(5, 0, SEEK_SET)                   = 0
lseek(5, 0, SEEK_SET)                   = 0
lseek(4, 0, SEEK_SET)                   = 0
lseek(4, 512000, SEEK_SET)              = 512000
lseek(4, 1000000, SEEK_SET)             = 1000000

It didn't write out PAX format!  Instead it replicated the holes into
the tar file itself with SEEK_SET.

$ strings foo.tar | grep Sparse

You have to add --format=posix to enable the GNU behaviour that BSD
tar is emulating by default:

$ tar --format=posix -S -cf foo.tar 1MB-sparse.dat
$ strings foo.tar | grep Sparse
./GNUSparseFile.4190/1MB-sparse.dat

I expected GNU tar to be forced to do that if writing to non-seekable
output, eg "tar -S -c 1MB-sparse.dat | cat > foo.tar", but somehow it
manages to write out only ~10KB of plain ustar format that it is able
to restore to the full 1MB apparent size using some other trick, but
... ENOTIME, I dunno how it's doing that.  Might be interesting to see
if pg_waldump can read it though, 'cause the bytes aren't all there.

BTW I confirmed that Apple tar does have -S by default too, it's just
that APFS doesn't make holes magically, so this test would presumably
have broken on a Mac if wal_init_zero had been forced to zero (not
tested).

Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
unlikely to hit this in the wild, and the symptom is a confusing error
in a maintenance tool, not corruption, so I don't think this is a big
deal.  I might still try teaching the astreamer code to understand PAX
1.0 when it sees it in the next cycle though, for the benefit of
FreeBSD users.  A quick and dirty version could probably just unmangle
the name and skip the first block of data, since any valid WAL file
will not begin with a hole and valid WAL data will end at the first
hole and fail our verification, but of course a real implementation
should read the map properly[1]...

[1] https://www.gnu.org/software/tar/manual/html_node/PAX-1.html

  90. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-03-29T22:20:24Z

    On Mon, Mar 30, 2026 at 11:11 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> ... so that's a yes, lseek sees holes that we didn't ask it to make,

Oops, sorry, I wrote that email too fast and got my examples mixed up,
BTFS actually *doesnt* do that automatically, that was of course a
trace showing a file with explicitly made holes.  So this is probably
be a ZFS-only issue unless you're using wal_init_zero=0, and then any
file system could result in PAX-sparse-format tarballs, but even then
only if you use non-default switches that in practice no one will use
with GNU tar, or if you use BSD tar.  So in practice this is a
FreeBSD-only issue.

  91. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-29T22:23:31Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
> unlikely to hit this in the wild, and the symptom is a confusing error
> in a maintenance tool, not corruption, so I don't think this is a big
> deal.  I might still try teaching the astreamer code to understand PAX
> 1.0 when it sees it in the next cycle though, for the benefit of
> FreeBSD users.

I agree that this isn't too critical if the effects are confined to
pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
astreamer_tar.c, but it's not clear to me if they'd ever be asked to
parse files made by tar(1) and not by our own sparseness-ignorant
tar-writing code.  If they can be, that'd be a higher-priority reason
to fill in this gap.

			regards, tom lane

  92. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-01T02:05:53Z

    On Mon, Mar 30, 2026 at 11:23 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
> > unlikely to hit this in the wild, and the symptom is a confusing error
> > in a maintenance tool, not corruption, so I don't think this is a big
> > deal.  I might still try teaching the astreamer code to understand PAX
> > 1.0 when it sees it in the next cycle though, for the benefit of
> > FreeBSD users.
>
> I agree that this isn't too critical if the effects are confined to
> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
> parse files made by tar(1) and not by our own sparseness-ignorant
> tar-writing code.  If they can be, that'd be a higher-priority reason
> to fill in this gap.

I pushed the workaround for the test.

Yeah I can't see any reason why pg_verifybackup --wal-path=foo.tar
won't suffer the same problem in the wild.  Again, it's not the end of
the world because it'll just fail and you'll probably eventually
figure out why.  So perhaps we should just improve our detection of
archives that we can't handle?  Straw man algorithm:

If you can't find $NAME in the archive, then check if PaxHeaders/$NAME
exists, and if so, fail with 'unsupported TAR format for WAL file "%s"
in archive "%s"' instead.  That'd probably work well enough in
practice, because astreamer_tar.c treats PAX extended header
pseudo-files as regular files (they're not, they have type 'x'), and
both GNU and BSD tar happen to use that.

POSIX doesn't require that naming, so it would in theory be more
correct to teach astreamer_tar.c to recognise PAX extended headers and
fish out enough information and link it to the following archive
member, but a simple test to improve error messaging seems like the
right level of effort here.

Here's a test patch that shows the problem on any system with GNU tar
or BSD tar and a file system that supports sparse files.  The test
succeeds because it looks for "error: could not find WAL" but the idea
would be to change it to look for a new error message like that.  My
motivation was to make this reproducible on any system, in case that's
helpful for Amul and Andrew if they're interested in trying to improve
this edge case in time for the release.  Otherwise I'll come back to
it, but probably not in time...

  93. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-04-01T10:39:05Z

    On 2026-03-31 Tu 10:05 PM, Thomas Munro wrote:
> On Mon, Mar 30, 2026 at 11:23 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> Thomas Munro<thomas.munro@gmail.com> writes:
>>> Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
>>> unlikely to hit this in the wild, and the symptom is a confusing error
>>> in a maintenance tool, not corruption, so I don't think this is a big
>>> deal.  I might still try teaching the astreamer code to understand PAX
>>> 1.0 when it sees it in the next cycle though, for the benefit of
>>> FreeBSD users.
>> I agree that this isn't too critical if the effects are confined to
>> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
>> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
>> parse files made by tar(1) and not by our own sparseness-ignorant
>> tar-writing code.  If they can be, that'd be a higher-priority reason
>> to fill in this gap.
> I pushed the workaround for the test.


It occurred to me this morning that we probably shouldn't run this test 
on Windows, and if we do we shouldn't be using /dev/null (the Windows 
equivalent of which is just "nul"). The simplest fix would just be to 
add a "!$windows_os" to the if test.


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  94. Re: pg_waldump: support decoding of WAL inside tarfile

    Andres Freund <andres@anarazel.de> — 2026-04-01T13:26:00Z

    Hi,

On 2026-04-01 06:39:05 -0400, Andrew Dunstan wrote:
> 
> On 2026-03-31 Tu 10:05 PM, Thomas Munro wrote:
> > On Mon, Mar 30, 2026 at 11:23 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
> > > Thomas Munro<thomas.munro@gmail.com> writes:
> > > > Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
> > > > unlikely to hit this in the wild, and the symptom is a confusing error
> > > > in a maintenance tool, not corruption, so I don't think this is a big
> > > > deal.  I might still try teaching the astreamer code to understand PAX
> > > > 1.0 when it sees it in the next cycle though, for the benefit of
> > > > FreeBSD users.
> > > I agree that this isn't too critical if the effects are confined to
> > > pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
> > > astreamer_tar.c, but it's not clear to me if they'd ever be asked to
> > > parse files made by tar(1) and not by our own sparseness-ignorant
> > > tar-writing code.  If they can be, that'd be a higher-priority reason
> > > to fill in this gap.
> > I pushed the workaround for the test.
> 
> 
> It occurred to me this morning that we probably shouldn't run this test on
> Windows, and if we do we shouldn't be using /dev/null (the Windows
> equivalent of which is just "nul"). The simplest fix would just be to add a
> "!$windows_os" to the if test.

Why should we skip this test on windows?

I think we have historically been way too liberal about sprinkling
!$windows_os test disablements around. More than once there were actual bugs
that we just swept under the rug by disabling the tests that detected them.
Either we support windows or we don't.

Greetings,

Andres Freund

  95. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-04-01T14:19:32Z

    On 2026-04-01 We 9:26 AM, Andres Freund wrote:
> Hi,
>
> On 2026-04-01 06:39:05 -0400, Andrew Dunstan wrote:
>> On 2026-03-31 Tu 10:05 PM, Thomas Munro wrote:
>>> On Mon, Mar 30, 2026 at 11:23 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>>>> Thomas Munro<thomas.munro@gmail.com> writes:
>>>>> Anyway, given the defaults, GNU tar + ZFS/BTRFS users must be pretty
>>>>> unlikely to hit this in the wild, and the symptom is a confusing error
>>>>> in a maintenance tool, not corruption, so I don't think this is a big
>>>>> deal.  I might still try teaching the astreamer code to understand PAX
>>>>> 1.0 when it sees it in the next cycle though, for the benefit of
>>>>> FreeBSD users.
>>>> I agree that this isn't too critical if the effects are confined to
>>>> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
>>>> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
>>>> parse files made by tar(1) and not by our own sparseness-ignorant
>>>> tar-writing code.  If they can be, that'd be a higher-priority reason
>>>> to fill in this gap.
>>> I pushed the workaround for the test.
>>
>> It occurred to me this morning that we probably shouldn't run this test on
>> Windows, and if we do we shouldn't be using /dev/null (the Windows
>> equivalent of which is just "nul"). The simplest fix would just be to add a
>> "!$windows_os" to the if test.
> Why should we skip this test on windows?
>
> I think we have historically been way too liberal about sprinkling
> !$windows_os test disablements around. More than once there were actual bugs
> that we just swept under the rug by disabling the tests that detected them.
> Either we support windows or we don't.
>

Maybe I misunderstood, but I didn't think this was going to be an issue 
on NTFS.

In general I agree with you, though. I try to avoid skipping things on 
Windows.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  96. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-01T18:25:40Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Mon, Mar 30, 2026 at 11:23 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I agree that this isn't too critical if the effects are confined to
>> pg_waldump.  I believe that pg_basebackup and pg_verifybackup also use
>> astreamer_tar.c, but it's not clear to me if they'd ever be asked to
>> parse files made by tar(1) and not by our own sparseness-ignorant
>> tar-writing code.  If they can be, that'd be a higher-priority reason
>> to fill in this gap.

> Yeah I can't see any reason why pg_verifybackup --wal-path=foo.tar
> won't suffer the same problem in the wild.  Again, it's not the end of
> the world because it'll just fail and you'll probably eventually
> figure out why.  So perhaps we should just improve our detection of
> archives that we can't handle?

After reading the POSIX spec for pax format (in the pax(1) man page),
I think it's absolutely essential that we reject files that contain
pax extension headers.  Those can change the interpretation of the
following file header(s) in nearly arbitrary ways, so we have plenty
of problems besides this sparse-file issue if we just ignore them.

(Of course, later we can consider improving the code to handle them
correctly, but that ain't happening in time for v19.)

Also, if we are admitting the possibility that what we are reading
was made by a platform-supplied tar and not our own code, I think
it verges on lunacy to behave as though unsupported typeflags are
regular files.

So I think we need something more or less like the attached.

			regards, tom lane

  97. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-02T00:16:17Z

    On Thu, Apr 2, 2026 at 7:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Also, if we are admitting the possibility that what we are reading
> was made by a platform-supplied tar and not our own code, I think
> it verges on lunacy to behave as though unsupported typeflags are
> regular files.

Yeah, if this is the first time we parse files we didn't make then
that makes total sense.  I was a bit unsure of that question when I
suggested we reject pax only after we've failed to find a file, in
case there are scenarios that work today with harmless ignorable pax
headers that don't change the file name.

> So I think we need something more or less like the attached.

LGTM.  Tested with both tars here.  I updated that little test patch
for this.  Not sure if you think it's worth a test though, now that
it's so simple.

@Andrew: I tried using File::Spec->devnull() this time.  Are you able
to check if this works OK on Windows, applied on top of Tom's patch?
AFAIK should be able to run this new test and pass, not skip it.  But
it could be that the shell invocation needs tweaking.  It's hard to
tell from CI.  (Huh, apparently Windows ships a copy of BSD tar as
C:\Windows\System32\tar.exe these days.)

  98. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T01:22:44Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Thu, Apr 2, 2026 at 7:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Also, if we are admitting the possibility that what we are reading
>> was made by a platform-supplied tar and not our own code, I think
>> it verges on lunacy to behave as though unsupported typeflags are
>> regular files.

> Yeah, if this is the first time we parse files we didn't make then
> that makes total sense.  I was a bit unsure of that question when I
> suggested we reject pax only after we've failed to find a file, in
> case there are scenarios that work today with harmless ignorable pax
> headers that don't change the file name.

Of course this is all new so far as pg_waldump is concerned.
I'm a bit unclear about whether pg_verifybackup's exposure
is large enough to warrant back-patching any of this.

Looking again at astreamer_tar.c, I suddenly realized that it doesn't
do any meaningful input validation.  So if you feed it junk input,
you get garbage errors that aren't even predictable:

$ head -c 32768 /dev/urandom >junk.tar
$ pg_waldump --path junk.tar --start 1/0x10000000
pg_waldump: error: COPY stream ended before last file was finished
$ head -c 32768 /dev/urandom >junk.tar
$ pg_waldump --path junk.tar --start 1/0x10000000
pg_waldump: error: tar member has empty name
$ head -c 32768 /dev/urandom >junk.tar
$ pg_waldump --path junk.tar --start 1/0x10000000
pg_waldump: error: COPY stream ended before last file was finished
$ head -c 32768 /dev/urandom >junk.tar
$ pg_waldump --path junk.tar --start 1/0x10000000
pg_waldump: error: could not find WAL in archive "junk.tar"

tar itself is considerably saner:

$ tar tf junk.tar 
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

So I think we need something like the attached, in addition
to what I sent before.  This just makes astreamer_tar.c use
the isValidTarHeader function that pg_dump already had.
(I decided to const-ify isValidTarHeader's argument while
moving it to a shared location, which in turn requires
const-ifying tarChecksum.)

			regards, tom lane

  99. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-02T02:14:09Z

    On Thu, Apr 2, 2026 at 2:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Looking again at astreamer_tar.c, I suddenly realized that it doesn't
> do any meaningful input validation.  So if you feed it junk input,
> you get garbage errors that aren't even predictable:

Wow.

> So I think we need something like the attached, in addition
> to what I sent before.  This just makes astreamer_tar.c use
> the isValidTarHeader function that pg_dump already had.
> (I decided to const-ify isValidTarHeader's argument while
> moving it to a shared location, which in turn requires
> const-ifying tarChecksum.)

LGTM.

$ echo -n x | dd of=foo.tar bs=1 seek=257 count=1 conv=notrunc
$ strings foo.tar | grep tar | head -1
xstar
$ pg_waldump --path=foo.tar -s 0/1 -e 0/100
pg_waldump: error: input file does not appear to be a valid tar archive

$ echo -n u | dd of=foo.tar bs=1 seek=257 count=1 conv=notrunc
$ strings foo.tar | grep tar | head -1
ustar
$ pg_waldump --path=foo.tar -s 0/1 -e 0/100
... other output...

  100. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-04-02T11:36:25Z

    On 2026-04-01 We 8:16 PM, Thomas Munro wrote:
> On Thu, Apr 2, 2026 at 7:25 AM Tom Lane<tgl@sss.pgh.pa.us> wrote:
>> Also, if we are admitting the possibility that what we are reading
>> was made by a platform-supplied tar and not our own code, I think
>> it verges on lunacy to behave as though unsupported typeflags are
>> regular files.
> Yeah, if this is the first time we parse files we didn't make then
> that makes total sense.  I was a bit unsure of that question when I
> suggested we reject pax only after we've failed to find a file, in
> case there are scenarios that work today with harmless ignorable pax
> headers that don't change the file name.
>
>> So I think we need something more or less like the attached.
> LGTM.  Tested with both tars here.  I updated that little test patch
> for this.  Not sure if you think it's worth a test though, now that
> it's so simple.
>
> @Andrew: I tried usingFile::Spec->devnull() this time.  Are you able
> to check if this works OK on Windows, applied on top of Tom's patch?
> AFAIK should be able to run this new test and pass, not skip it.  But
> it could be that the shell invocation needs tweaking.  It's hard to
> tell from CI.  (Huh, apparently Windows ships a copy of BSD tar as
> C:\Windows\System32\tar.exe these days.)


Yes, that appears to work. I would put a "2>&1" at the end - we don't 
care about the output, just whether or not it succeeds:


C:\Windows\system32>perl -MFile::Spec -e "print File::Spec->devnull();"
nul

C:\Windows\system32>tar --no-read-sparse -c - nul > nul 2>&1 && echo hello

C:\Windows\system32>tar --no-read-sparse -c - nul > nul 2>&1 || echo hello
hello


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  101. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T16:29:52Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Thu, Apr 2, 2026 at 2:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> So I think we need something like the attached, in addition
>> to what I sent before.  This just makes astreamer_tar.c use
>> the isValidTarHeader function that pg_dump already had.

> LGTM.

Pushed, thanks for reviewing!  In the event I decided to back-patch to
v18, where these fixes could protect pg_verifybackup against tar files
it can't handle.  In older branches the astreamer (nee bbstreamer)
logic only exists inside pg_basebackup, and IIUC that's not really
exposed to tar files that weren't made by our own code.  So I did
not bother with back-patching further, though it'd be possible to
do that if someone knows a scenario where it'd matter.

			regards, tom lane

  102. Re: pg_waldump: support decoding of WAL inside tarfile

    Tomas Vondra <tomas@vondra.me> — 2026-04-02T17:13:08Z

    On 4/2/26 18:29, Tom Lane wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
>> On Thu, Apr 2, 2026 at 2:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> So I think we need something like the attached, in addition
>>> to what I sent before.  This just makes astreamer_tar.c use
>>> the isValidTarHeader function that pg_dump already had.
> 
>> LGTM.
> 
> Pushed, thanks for reviewing!  In the event I decided to back-patch to
> v18, where these fixes could protect pg_verifybackup against tar files
> it can't handle.  In older branches the astreamer (nee bbstreamer)
> logic only exists inside pg_basebackup, and IIUC that's not really
> exposed to tar files that weren't made by our own code.  So I did
> not bother with back-patching further, though it'd be possible to
> do that if someone knows a scenario where it'd matter.
> 

It seems jay/hippopotamus failed on this, for some reason. Those two
animals are on the same host, just using different compilers. I did
update+reboot the machine this afternoon (before the test), but I don't
see why that would cause the failure.

Maybe there's something special about OpenSUSE?

-- 
Tomas Vondra

  103. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T17:43:36Z

    Tomas Vondra <tomas@vondra.me> writes:
> On 4/2/26 18:29, Tom Lane wrote:
>> Pushed, thanks for reviewing!  In the event I decided to back-patch to
>> v18, where these fixes could protect pg_verifybackup against tar files
>> it can't handle.

> It seems jay/hippopotamus failed on this, for some reason.

Hmm:

#   Failed test 'corrupt backup fails verification: extra_file: matches'
#   at t/003_corruption.pl line 198.
#                   'pg_verifybackup: error: pax extensions to tar format are not supported
# '
#     doesn't match '(?^:extra_file.*present (on disk|in archive "[^"]+") but not in the manifest)'

> Maybe there's something special about OpenSUSE?

Apparently its version of "tar" will produce pax-extended files
at the drop of a hat.  I have an OpenSUSE image around here
somewhere, will see if I can reproduce this.  But while I'm
asking, what filesystem are those animals running on top of?

			regards, tom lane

  104. Re: pg_waldump: support decoding of WAL inside tarfile

    Tomas Vondra <tomas@vondra.me> — 2026-04-02T18:08:18Z

    
On 4/2/26 19:43, Tom Lane wrote:
> Tomas Vondra <tomas@vondra.me> writes:
>> On 4/2/26 18:29, Tom Lane wrote:
>>> Pushed, thanks for reviewing!  In the event I decided to back-patch to
>>> v18, where these fixes could protect pg_verifybackup against tar files
>>> it can't handle.
> 
>> It seems jay/hippopotamus failed on this, for some reason.
> 
> Hmm:
> 
> #   Failed test 'corrupt backup fails verification: extra_file: matches'
> #   at t/003_corruption.pl line 198.
> #                   'pg_verifybackup: error: pax extensions to tar format are not supported
> # '
> #     doesn't match '(?^:extra_file.*present (on disk|in archive "[^"]+") but not in the manifest)'
> 
>> Maybe there's something special about OpenSUSE?
> 
> Apparently its version of "tar" will produce pax-extended files
> at the drop of a hat.  I have an OpenSUSE image around here
> somewhere, will see if I can reproduce this.  But while I'm
> asking, what filesystem are those animals running on top of?
> 

btrfs

-- 
Tomas Vondra

  105. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T18:47:52Z

    Tomas Vondra <tomas@vondra.me> writes:
> On 4/2/26 19:43, Tom Lane wrote:
>> Tomas Vondra <tomas@vondra.me> writes:
>>> Maybe there's something special about OpenSUSE?

>> Apparently its version of "tar" will produce pax-extended files
>> at the drop of a hat.  I have an OpenSUSE image around here
>> somewhere, will see if I can reproduce this.  But while I'm
>> asking, what filesystem are those animals running on top of?

> btrfs

Yup, so capable of making sparse WAL files.  I can reproduce the
problem here, and what I see is

> tar --version
tar (GNU tar) 1.34
Copyright (C) 2021 Free Software Foundation, Inc.
...

> tar -?
...
  -H, --format=FORMAT        create archive of the given format

 FORMAT is one of the following:
    gnu                      GNU tar 1.13.x format
    oldgnu                   GNU format as per tar <= 1.12
    pax                      POSIX 1003.1-2001 (pax) format
    posix                    same as pax
    ustar                    POSIX 1003.1-1988 (ustar) format
    v7                       old V7 tar format
...
*This* tar defaults to:
--format=posix -f- -b20 --quoting-style=escape --rmt-command=/usr/bin/rmt
--rsh-command=/usr/bin/ssh

So there you have it: pax format by default.  This is unlike what
I see on RHEL or Fedora:

...
*This* tar defaults to:
--format=gnu -f- -b20 --quoting-style=escape --rmt-command=/etc/rmt
--rsh-command=/usr/bin/ssh

So it looks like we need a switch hack similar to what we did for
BSD tar, but injecting "--format=gnu" (or perhaps "--format=ustar"?)
if the tar program will take that.

Interestingly, pg_verifybackup's t/003_corruption.pl test also fails
with the same issue, so apparently this platform is even more
aggressive about sparse-ifying files than Thomas' FreeBSD box.
I wonder how come we managed to pass that test case before on
these machines.

I'm inclined to push the logic for selecting these tar options
into some common subroutine in Test::Utils, rather than having
two copies (and maybe more later).

			regards, tom lane

  106. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T19:15:36Z

    I wrote:
> Interestingly, pg_verifybackup's t/003_corruption.pl test also fails
> with the same issue, so apparently this platform is even more
> aggressive about sparse-ifying files than Thomas' FreeBSD box.
> I wonder how come we managed to pass that test case before on
> these machines.

The answer to that seems to be that the test scripts for
pg_verifybackup simply fail to detect when it's mishandling
sparse tar entries.  We only check failing cases not successful
cases, and in each case the error checked for is independent of
whether we would have extracted WAL data correctly.  Grumble.

			regards, tom lane

  107. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-02T22:41:33Z

    On Fri, Apr 3, 2026 at 7:48 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> *This* tar defaults to:
> --format=posix -f- -b20 --quoting-style=escape --rmt-command=/usr/bin/rmt
> --rsh-command=/usr/bin/ssh

From some quick googling, OpenSUSE is the only distro I could find
that does this, but the fine manual says they plan to make pax the
default upstream so eventually it might be everywhere:

https://www.gnu.org/software/tar/manual/html_section/Formats.html

> Interestingly, pg_verifybackup's t/003_corruption.pl test also fails
> with the same issue, so apparently this platform is even more
> aggressive about sparse-ifying files than Thomas' FreeBSD box.

Looks like sparse files and BTRFS might be a red herring then, if it's
simply been told to put a pax header on every file?

I think my system might be defaulting to "restricted pax", meaning
that pax headers are added to individual files only if necessary,
path-too-long etc, magic sparse format triggered, etc.

https://man.freebsd.org/cgi/man.cgi?query=libarchive-formats

> I'm inclined to push the logic for selecting these tar options
> into some common subroutine in Test::Utils, rather than having
> two copies (and maybe more later).

How about using --format=ustar, instead of that sparse control stuff?
That solves the BSD tar + ZFS problem and the OpenSUSE GNU tar default
format problem in one fell swoop, and actually says what we want
explicitly.

Should we have a <note> near the --path/--wal-path documentation to
state explicitly that usta format is required, and that pax is not
[yet] supported, and that --format=ustar might be required on some
systems?

  108. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T22:50:50Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Fri, Apr 3, 2026 at 7:48 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Interestingly, pg_verifybackup's t/003_corruption.pl test also fails
>> with the same issue, so apparently this platform is even more
>> aggressive about sparse-ifying files than Thomas' FreeBSD box.

> Looks like sparse files and BTRFS might be a red herring then, if it's
> simply been told to put a pax header on every file?

It looked to me that it was only putting a pax header on sparse-ified
WAL files.

> How about using --format=ustar, instead of that sparse control stuff?

I did it that way for GNU tar, but did not research whether bsdtar
will take that option.  Feel free to hack on ebba64c08 some more.

(It seems though that the two tars' locutions for "write to stdout"
are different, so we might have to have separate tests even if they
end up pushing the same option.)

			regards, tom lane

  109. Re: pg_waldump: support decoding of WAL inside tarfile

    Sami Imseih <samimseih@gmail.com> — 2026-04-02T23:43:33Z

    > >> So I think we need something like the attached, in addition
> >> to what I sent before.  This just makes astreamer_tar.c use
> >> the isValidTarHeader function that pg_dump already had.
>
> > LGTM.
>
> Pushed, thanks for reviewing!  In the event I decided to back-patch to
> v18, where these fixes could protect pg_verifybackup against tar files
> it can't handle.

Hi,

I just encountered a regression test failure for pg_waldump due to ebba64c08d9.

````
――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀
――――――――――――――――――――――――――――――――――――――――――――――――――――――
Listing only the last 100 lines from a long log.
#   at /local/home/simseih/pgdev/installations/worktrees/dev/src/bin/pg_waldump/t/001_basic.pl
line 432.
#          got: 'pg_waldump: error: could not find WAL in archive
"pg_wal.tar.gz"
# '
#     expected: ''
```

and regress_log_001_basic shows this:

```
# Running: /usr/bin/tar --format=ustar -cf /tmp/ja26rXZOnb/pg_wal.tar
archive_status 000000010000000000000002 000000010000000000000001
summaries 000000010000000000000003
[22:25:00.525](0.008s) not ok 101
[22:25:00.525](0.000s) #   Failed test at
/local/home/simseih/pgdev/installations/worktrees/dev/src/bin/pg_waldump/t/001_basic.pl
line 350.
[22:25:00.525](0.000s) # ---------- command failed ----------
[22:25:00.526](0.000s) # /usr/bin/tar --format=ustar -cf
/tmp/ja26rXZOnb/pg_wal.tar archive_status 000000010000000000000002
000000010000000000000001 summaries 000000010000000000000003
[22:25:00.526](0.000s) # -------------- stderr --------------
[22:25:00.526](0.000s) # /usr/bin/tar: value 10012663 out of uid_t
range 0..2097151
```

The --format=ustar has a limit of 2^21 (2097151) for UID/GID [1]
and on my machine the UID is 10012663.

So I found that one way to deal with this is to run the tar command with
--owner=0 --group=0. As far as I can tell, the owner and group IDs don't
matter for these tests, so maybe that is OK.

@@ -1333,6 +1333,10 @@ sub tar_portability_options
                == 0)
        {
                push(@tar_p_flags, "--format=ustar");
+               # ustar format supports UIDs only up to 2^21 (2097151).
+               # Override owner/group to avoid failures on systems where
+               # the running user's UID/GID exceeds that limit.
+               push(@tar_p_flags, "--owner=0", "--group=0");
        }

While this fixes the test, I am now not sure what the broader implications are
for --format=ustar for pg_waldump in the broader discussion?

[1] [https://www.gnu.org/software/tar/manual/html_section/Formats.html]

--
Sami Imseih
Amazon Web Services (AWS)

  110. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-02T23:49:21Z

    On Fri, Apr 3, 2026 at 11:50 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > How about using --format=ustar, instead of that sparse control stuff?
>
> I did it that way for GNU tar, but did not research whether bsdtar
> will take that option.  Feel free to hack on ebba64c08 some more.
>
> (It seems though that the two tars' locutions for "write to stdout"
> are different, so we might have to have separate tests even if they
> end up pushing the same option.)

I have:

$ tar --version
bsdtar 3.8.2 - libarchive 3.8.2 zlib/1.3.1 liblzma/5.8.1 libzstd/1.5.2
openssl/3.5.4 libb2/bundled
$ gtar --version
tar (GNU tar) 1.35
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

This seems to work for both:

$ tar --format=ustar -c /dev/null  > /dev/null
tar: Removing leading '/' from member names
$ gtar --format=ustar -c /dev/null  > /dev/null
gtar: Removing leading `/' from member names

The attached passes with both, and regress_log_001_basic looks like:

# Running: /usr/bin/tar --format=ustar -cf /tmp/J_ifbfUOSd/pg_wal.tar
archive_status 000000010000000000000001 000000010000000000000003
000000010000000000000002 summaries
[12:12:24.301](0.072s) ok 101

# Running: /usr/local/bin/gtar --format=ustar -cf
/tmp/pbdsHdrAdw/pg_wal.tar 000000010000000000000002 archive_status
000000010000000000000003 summaries 000000010000000000000001
[12:18:14.739](0.050s) ok 101

I think a Windows system could be using either.  BSD tar comes
pre-installed by Microsoft and people often install GNU tools.  So I
think we should use File::Spec->devnull() instead of /dev/null, and
Andrew showed that working.  I doubt Windows is capable of making
sparse files (except perhaps with ReFS?), but it's nice to use the
same code everywhere and future-proof in case GNU carries out its
thread to switch to pax by default.  Windows probably has file
attributes that ustar can't represent (?), so I guess that might
motivate it to use pax headers if they are indeed added only when
needed.

Longer term I think we need to tolerate but ignore pax headers.  If I
understand the spirit of this long evolution, pax archives are
intended to be acceptable to pre-pax implementations, which implies
that they can't really change the meaning of the bits of the file
contents.  That's why GNU's --sparse hides funky file encodings from
old tars by renaming them to GNUSparseFile.%p/%f, and that leads back
to my original suggestion that we should figure out how to detect and
reject pax only if we failed to find the file under the expected name.
(Or of course we could just implement support for that, and I have a
half-baked trial patch for that but now is not the time.)

  111. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-03T00:07:40Z

    On Fri, Apr 3, 2026 at 12:43 PM Sami Imseih <samimseih@gmail.com> wrote:
> The --format=ustar has a limit of 2^21 (2097151) for UID/GID [1]
> and on my machine the UID is 10012663.
>
> So I found that one way to deal with this is to run the tar command with
> --owner=0 --group=0. As far as I can tell, the owner and group IDs don't
> matter for these tests, so maybe that is OK.
>
> @@ -1333,6 +1333,10 @@ sub tar_portability_options
>                 == 0)
>         {
>                 push(@tar_p_flags, "--format=ustar");
> +               # ustar format supports UIDs only up to 2^21 (2097151).
> +               # Override owner/group to avoid failures on systems where
> +               # the running user's UID/GID exceeds that limit.
> +               push(@tar_p_flags, "--owner=0", "--group=0");

Interesting.  BSD tar accepts those too, so here's an update to my
previous patch.

> While this fixes the test, I am now not sure what the broader implications are
> for --format=ustar for pg_waldump in the broader discussion?

I think users who have their own tar scripts will mostly be
unaffected, but a small minority will see the new error if they try to
use pg_verifybackup or pg_waldump, and they'll find their way to
--format=ustar, and then they might see the UID/GID error in their own
.tar production scripts, and find their way to adding those switches
too.  Seems OK?  Especially with a documentation note once we've
settle all of this.

  112. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-03T00:11:37Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Fri, Apr 3, 2026 at 11:50 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> How about using --format=ustar, instead of that sparse control stuff?

>> I did it that way for GNU tar, but did not research whether bsdtar
>> will take that option.  Feel free to hack on ebba64c08 some more.

> This seems to work for both:

> $ tar --format=ustar -c /dev/null  > /dev/null
> tar: Removing leading '/' from member names
> $ gtar --format=ustar -c /dev/null  > /dev/null
> gtar: Removing leading `/' from member names

Cool.  LGTM.

> I think a Windows system could be using either.  BSD tar comes
> pre-installed by Microsoft and people often install GNU tools.  So I
> think we should use File::Spec->devnull() instead of /dev/null, and
> Andrew showed that working.

Agreed.

> Longer term I think we need to tolerate but ignore pax headers.  If I
> understand the spirit of this long evolution, pax archives are
> intended to be acceptable to pre-pax implementations, which implies
> that they can't really change the meaning of the bits of the file
> contents.

I don't buy that.  For example, POSIX specifies these allowed
fields in an extended header:

    linkpath
        The pathname of a link being created to another file, of any
        type, previously archived. This record shall override the
        linkname field in the following ustar header block(s).

    path
        The pathname of the following file(s). This record shall
        override the name and prefix fields in the following header
        block(s).

    size
        The size of the file in octets, expressed as a decimal number
        using digits from the ISO/IEC 646:1991 standard. This record
        shall override the size field in the following header
        block(s).

GNU tar seems to try hard to ensure that a non-pax-aware tar can
extract *something* from a tar file, but it's not guaranteed that the
something contains the right data or is located at the right pathname.
It looks like the goal is to allow post-processing to pick up the
pieces.

In any case, this is all completely moot if we don't write code to
de-sparse a sparse entry: we will not be able to validate WAL data
if the WAL file is missing some pages.  So I see little point in
having code that tolerates pax headers if it doesn't also do that.

			regards, tom lane

  113. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-03T00:47:28Z

    On Fri, Apr 3, 2026 at 1:11 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> In any case, this is all completely moot if we don't write code to
> de-sparse a sparse entry: we will not be able to validate WAL data
> if the WAL file is missing some pages.  So I see little point in
> having code that tolerates pax headers if it doesn't also do that.

Yeah.  FWIW I spent a few hours hacking on that the other day and
could decode many files, but I now realise that the task was made more
difficult by a problem you fixed: without header validation, small
mistakes resulted in corruption or went bananas.  With that now
addressed, I hope I can get it into shape and propose it for the next
cycle...

For what it's worth, I was just speculating about how one might
reasonably handle unrecognised *non-standard* header names, not the
POSIX-standardised ones which, you're right, we'd probably need to
grok properly.  If we assumed reasonable engineering decisions
following (what I understood to be) the spirit of pax, maybe we could
assume that new non-standard headers either don't affect file contents
and thus could be ignored (think: GNU.windows.permissions=...), or do
affect file contents but have measures in place to prevent unknown
encodings from being exposed to unsuspecting software (think:
deathstation.byte=9bit).  That's a position we could choose to take,
anyway, in the absence of a crystal ball...  Fortunately there aren't
really many implementations of POSIX left, so it's not like we're
dealing with the Fermi Paradox here...

  114. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-03T00:59:16Z

    On Fri, Apr 3, 2026 at 1:11 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Cool.  LGTM.

Thanks.  I'll go ahead and push v2 shortly, which includes Sami's
change (with credit), unless you have any better ideas for that bit.

  115. Re: pg_waldump: support decoding of WAL inside tarfile

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-03T01:11:22Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> Thanks.  I'll go ahead and push v2 shortly, which includes Sami's
> change (with credit), unless you have any better ideas for that bit.

Not really.  I wondered for a bit if it was smart to create tar files
whose contents would appear root-owned, but it could only matter if
somebody extracted such a file as root, and I don't see a plausible
pathway for that to happen, since these are merely transient test
files.

We could dodge that hazard by using, say, "--owner=1" but I'm
not sure whether that'd introduce its own problems.  In any case,
walmethods.c's tar_open_for_write is already filling in zeroes
there.

			regards, tom lane

  116. Re: pg_waldump: support decoding of WAL inside tarfile

    Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-04-03T11:37:47Z

    Hi,

On Thu, 2 Apr 2026 at 21:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>  FORMAT is one of the following:
>     gnu                      GNU tar 1.13.x format
>     oldgnu                   GNU format as per tar <= 1.12
>     pax                      POSIX 1003.1-2001 (pax) format
>     posix                    same as pax
>     ustar                    POSIX 1003.1-1988 (ustar) format
>     v7                       old V7 tar format
> ...
> *This* tar defaults to:
> --format=posix -f- -b20 --quoting-style=escape --rmt-command=/usr/bin/rmt
> --rsh-command=/usr/bin/ssh
>
> So there you have it: pax format by default.  This is unlike what
> I see on RHEL or Fedora:

It seems that the problem also applies to OpenBSD [1]:

-F format
    Specify the output archive format, with the default format being
pax. tar currently supports the following formats:

OpenBSD CI tasks started to fail [2] after bc30c704ad with the errors:

```
Listing only the last 100 lines from a long log.
#   at /home/postgres/postgres/src/bin/pg_waldump/t/001_basic.pl line 440.
#          got: 'pg_waldump: error: pax extensions to tar format are
not supported

#   Failed test 'corrupt backup fails verification: extra_file: matches'
#   at /home/postgres/postgres/src/bin/pg_verifybackup/t/003_corruption.pl
line 198.
#                   'pg_verifybackup: error: pax extensions to tar
format are not supported

Summary of Failures:
239/381 postgresql:pg_waldump / pg_waldump/001_basic
                 ERROR            18.58s   exit status 84
225/381 postgresql:pg_verifybackup / pg_verifybackup/003_corruption
                 ERROR            45.12s   exit status 8
```

I also tried Thomas'
"v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
didn't fix the problem on OpenBSD [4].

[1] https://man.openbsd.org/tar#F
[2] https://cirrus-ci.com/task/5439721360326656
[3] https://postgr.es/m/CA%2BhUKGLMkv_fnGXzVRO8qbx5uHs-qMn151GTJYCfn9w1ZamGNg%40mail.gmail.com
[4] https://cirrus-ci.com/task/5602126958690304

-- 
Regards,
Nazir Bilal Yavuz
Microsoft

  117. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-03T14:37:26Z

    On Sat, Apr 4, 2026 at 12:38 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> I also tried Thomas'
> "v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
> didn't fix the problem on OpenBSD [4].

Apparently it wants -F ustar, like this.  Funny that it passed on the
build farm animals though.  Oh, it looks like they changed the default
fairly recently.

https://undeadly.org/cgi?action=article;sid=20240417053301

  118. Re: pg_waldump: support decoding of WAL inside tarfile

    Sami Imseih <samimseih@gmail.com> — 2026-04-03T14:59:19Z

    Hi,

> On Sat, Apr 4, 2026 at 12:38 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > I also tried Thomas'
> > "v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
> > didn't fix the problem on OpenBSD [4].
>
> Apparently it wants -F ustar, like this.  Funny that it passed on the
> build farm animals though.  Oh, it looks like they changed the default
> fairly recently.

LGTM with just a correction of my earlier comment.

< +    # ustar format supports UIDs only up to 2^21 (2097151).  Override
---
> +    # ustar format supports UIDs only up to 2^21 - 1 (2097151).  Override

--
Sami

  119. Re: pg_waldump: support decoding of WAL inside tarfile

    Nazir Bilal Yavuz <byavuz81@gmail.com> — 2026-04-03T15:30:01Z

    Hi,

On Fri, 3 Apr 2026 at 17:59, Sami Imseih <samimseih@gmail.com> wrote:
>
> Hi,
>
> > On Sat, Apr 4, 2026 at 12:38 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > > I also tried Thomas'
> > > "v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
> > > didn't fix the problem on OpenBSD [4].
> >
> > Apparently it wants -F ustar, like this.  Funny that it passed on the
> > build farm animals though.  Oh, it looks like they changed the default
> > fairly recently.
>
> LGTM with just a correction of my earlier comment.

Thanks for the patches! I confirm that both v3 and v4 fix the problem
for OpenBSD CI.


-- 
Regards,
Nazir Bilal Yavuz
Microsoft

  120. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-04T01:07:04Z

    On Sat, Apr 4, 2026 at 4:30 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> On Fri, 3 Apr 2026 at 17:59, Sami Imseih <samimseih@gmail.com> wrote:
> > > On Sat, Apr 4, 2026 at 12:38 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > > > I also tried Thomas'
> > > > "v2-0001-Improve-tar-portability-logic-from-ebba64c0" [3] but it
> > > > didn't fix the problem on OpenBSD [4].
> > >
> > > Apparently it wants -F ustar, like this.  Funny that it passed on the
> > > build farm animals though.  Oh, it looks like they changed the default
> > > fairly recently.
> >
> > LGTM with just a correction of my earlier comment.
>
> Thanks for the patches! I confirm that both v3 and v4 fix the problem
> for OpenBSD CI.

Pushed, after testing on an OpenBSD VM and making some corrections:

* I'd screwed up the test command line in a way that worked by coincidence
** OpenBSD tar writes to a tape device by default, so use -f /dev/null
** I'd forgotten == 0, so the result was inverted, hiding that screwup
* -f /dev/null is a better form for all of them because the default
destination is a build option
* needed elsif instead of if, or BSD tar finished up getting both
--format=ustar and -F ustar
* ran perltidy, keeping only the hunks due to this patch

CI passes and shows "212 subtests passed" for all five Unixen +
Windows/mingw, but only "156 subtests passed" for Windows/MSVC.
.cirrus.tasks.yml appears to use the same $TAR for both, namely the
system tar, so I think we can say that *this* thing is working, but
something else might be wrong with our scripting glue somewhere?

The other OSes on our list are AIX and Solaris.  From a quick look at
their manuals, I don't foresee issues with pax or large UIDs.

Hopefully that covers everything!

  121. Re: pg_waldump: support decoding of WAL inside tarfile

    Thomas Munro <thomas.munro@gmail.com> — 2026-04-10T07:57:56Z

    Nitpicking code review for commit b15c1513:

+read_archive_wal_page(XLogDumpPrivate *privateInfo, XLogRecPtr targetPagePtr,
+                     Size count, char *readBuff)

I thought we agreed to stop using Size for new code?  size_t has been
around since C89.

+               pg_fatal("WAL segment \"%s\" in archive \"%s\" is too short: rea
d %lld of %lld bytes",
+                        fname, privateInfo->archive_name,
+                        (long long int) (count - nbytes),
+                        (long long int) count);

Why cast to long long int?  That's the sort of thing we used to have
to do for int64 (but no longer), but here it's size_t anyway.  %zu has
been around since C99.

  122. Re: pg_waldump: support decoding of WAL inside tarfile

    Andrew Dunstan <andrew@dunslane.net> — 2026-04-10T12:57:14Z

    On 2026-04-10 Fr 3:57 AM, Thomas Munro wrote:
> Nitpicking code review for commit b15c1513:
>
> +read_archive_wal_page(XLogDumpPrivate *privateInfo, XLogRecPtr targetPagePtr,
> +                     Size count, char *readBuff)
>
> I thought we agreed to stop using Size for new code?  size_t has been
> around since C89.


Must have missed the memo :-(


>
> +               pg_fatal("WAL segment \"%s\" in archive \"%s\" is too short: rea
> d %lld of %lld bytes",
> +                        fname, privateInfo->archive_name,
> +                        (long long int) (count - nbytes),
> +                        (long long int) count);
>
> Why cast to long long int?  That's the sort of thing we used to have
> to do for int64 (but no longer), but here it's size_t anyway.  %zu has
> been around since C99.


will fix. Thanks for looking.


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

  123. Documenting coding style

    Andres Freund <andres@anarazel.de> — 2026-04-10T14:17:44Z

    Hi, 

On April 10, 2026 3:57:56 AM EDT, Thomas Munro <thomas.munro@gmail.com> wrote:
>Nitpicking code review for commit b15c1513:
>
>+read_archive_wal_page(XLogDumpPrivate *privateInfo, XLogRecPtr targetPagePtr,
>+                     Size count, char *readBuff)
>
>I thought we agreed to stop using Size for new code?  size_t has been
>around since C89.

We really need to start documenting some of this stuff somewhere. Deciding something a few years ago, deep in a thread, won't actually help anyone but the participants (and maybe not even them) to know about it. 

I wonder if we should move the coding style section out of sgml into a top-level CODING_STYLE.md or something like that.

And then obviously add things like Size being deprecated. 

Greetings, 

Andres 


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

  124. Re: Documenting coding style

    Andrew Dunstan <andrew@dunslane.net> — 2026-04-10T14:45:44Z

    On 2026-04-10 Fr 10:17 AM, Andres Freund wrote:
> Hi,
>
> On April 10, 2026 3:57:56 AM EDT, Thomas Munro <thomas.munro@gmail.com> wrote:
>> Nitpicking code review for commit b15c1513:
>>
>> +read_archive_wal_page(XLogDumpPrivate *privateInfo, XLogRecPtr targetPagePtr,
>> +                     Size count, char *readBuff)
>>
>> I thought we agreed to stop using Size for new code?  size_t has been
>> around since C89.
> We really need to start documenting some of this stuff somewhere. Deciding something a few years ago, deep in a thread, won't actually help anyone but the participants (and maybe not even them) to know about it.
>
> I wonder if we should move the coding style section out of sgml into a top-level CODING_STYLE.md or something like that.
>
> And then obviously add things like Size being deprecated.
>

+many. Great idea.


cheers


andew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  125. Re: Documenting coding style

    Nathan Bossart <nathandbossart@gmail.com> — 2026-04-10T14:49:42Z

    On Fri, Apr 10, 2026 at 10:17:44AM -0400, Andres Freund wrote:
> On April 10, 2026 3:57:56 AM EDT, Thomas Munro <thomas.munro@gmail.com> wrote:
>> I thought we agreed to stop using Size for new code?  size_t has been
>> around since C89.
> 
> We really need to start documenting some of this stuff somewhere.
> Deciding something a few years ago, deep in a thread, won't actually help
> anyone but the participants (and maybe not even them) to know about it..

I certainly didn't know this.  There's no comment in c.h, either.

> I wonder if we should move the coding style section out of sgml into a
> top-level CODING_STYLE.md or something like that.
> 
> And then obviously add things like Size being deprecated. 

Unless we're going to actually remove the typedef in the near future, I'm
not sure I'd support even marking it deprecated.  If we're going to keep it
around indefinitely, that's just going to become another source of nitpicks
when new contributors inevitably copy/paste some code from the aughts.  A
style page makes the situation a little better, but it's yet another thing
that folks have to remember.

To be clear, if someone proposed a patch that completely removed all traces
of Size, I'd likely support it.  There is indeed no reason not to use
size_t.  (I see that Size has been an alias for size_t since 1998 [0].)
But it's also quite heavily used, so I'd be fine with leaving it around and
considering it fully supported, too.

[0] https://postgr.es/c/0ad5d2a3a8

-- 
nathan

  126. Re: Documenting coding style

    Peter Eisentraut <peter@eisentraut.org> — 2026-04-15T18:10:59Z

    On 10.04.26 16:49, Nathan Bossart wrote:
>> I wonder if we should move the coding style section out of sgml into a
>> top-level CODING_STYLE.md or something like that.
>>
>> And then obviously add things like Size being deprecated.
> Unless we're going to actually remove the typedef in the near future, I'm
> not sure I'd support even marking it deprecated.  If we're going to keep it
> around indefinitely, that's just going to become another source of nitpicks
> when new contributors inevitably copy/paste some code from the aughts.  A
> style page makes the situation a little better, but it's yet another thing
> that folks have to remember.

In this case, if one wanted to do something, one should at least enhance 
the code comment for Size, which is currently very bare and doesn't 
explain why the type exists.

As some counterexamples, some of which I wrote:

/*
  * Pointer
  *      Variable holding address of any memory resident object.
  *      (obsolescent; use void * or char *)
  */
typedef void *Pointer;

/* Historical names for types in <stdint.h>. */
typedef int8_t int8;
...

  * We require C11 and C++11, so static_assert() is expected to be there.
  * StaticAssertDecl() was previously used for portability, but it's now 
just a
  * plain wrapper and doesn't need to be used in new code.