Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Don't set the truncation block length greater than RELSEG_SIZE.

  1. [BUG] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Oleg Tkachenko <oatkachenko@gmail.com> — 2025-11-14T19:43:31Z

    Hello PostgreSQL developers,

I’ve encountered a bug in the incremental backup feature that prevents restoration of backups containing relations larger than 1 GB that were vacuum-truncated.



Problem Description

When taking incremental backups of relations that span multiple segments, if the relation is truncated during VACUUM (after the base backup but before the incremental one), pg_combinebackup fails with:

```

file "%s" has truncation block length %u in excess of segment size %u

```

pg_basebackup itself completes without errors, but the resulting incremental backup cannot be restored.



Root Cause

In segmented relations, a VACUUM that truncates blocks sets a limit_block in the WAL summary. The incremental restore logic miscalculates truncation_block_length when processing segment 0…N, because it compares the segment-local size with a relation-wide limit.

In src/backend/backup/basebackup_incremental.c:

```

*truncation_block_length = size / BLCKSZ;

if (BlockNumberIsValid(limit_block))

{

    unsigned relative_limit = limit_block - segno * RELSEG_SIZE;

    if (*truncation_block_length < relative_limit)   /* ← problematic */

        *truncation_block_length = relative_limit;

}

```

For example, if limit_block lies in segment 10, then relative_limit will be roughly 9 * RELSEG_SIZE while processing segment 0. This forces truncation_block_length far beyond the actual segment size, leading to a segment length larger than RELSEG_SIZE and eventually the restore error.



Reproduction Steps

Create a table larger than 1 GB (multiple segments).

Take a full base backup.

Delete rows that occupy the end of the relation.

Run VACUUM (VERBOSE, TRUNCATE) to ensure blocks are removed.

(optional) Confirm that the WAL summary includes a limit entry for the relation.

Take an incremental backup with pg_basebackup.

Attempt to restore using pg_combinebackup.

Observe the truncation block length error.



Patch

A patch correcting this logic is attached, and I’m happy to provide additional details or revisions if helpful.



Best regards,

Oleg Tkachenko



  2. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Oleg Tkachenko <oatkachenko@gmail.com> — 2025-12-15T11:09:03Z

    Hello,

I am following up on my report with a patch, which did not receive any responses. I suspect the issue only manifests under specific conditions, so I am sending additional details along with a reliable reproducer.



For context:

Incremental backups cannot be restored when a relation larger than 1 GB (multiple segments) is vacuum-truncated between the base backup and the incremental backup (pg_basebackup itself completes successfully, but the resulting incremental backup is not restorable)

For segmented relations, the WAL summarizer records limit_block in the WAL summary. During incremental backup, the truncation length is computed incorrectly because a relation-wide limit is compared against a segment-local size.



Reproducer

I am attaching a bash script that reliably reproduces the issue on my system. The script:

Creates a table large enough to span multiple segments.

Takes a full base backup.

Deletes rows at the end of the relation.

Runs VACUUM (TRUNCATE) to remove blocks.

Takes an incremental backup.

Fails during pg_combinebackup.

The script is fully automated and intended to be run as-is.

A patch itself in the previous message. 

I would appreciate feedback on the approach and am happy to revise it if needed.



Best regards,

Oleg Tkachenko



> On Nov 14, 2025, at 20:43, Oleg Tkachenko <oatkachenko@gmail.com> wrote:
> 
> Hello PostgreSQL developers,
> 
> I’ve encountered a bug in the incremental backup feature that prevents restoration of backups containing relations larger than 1 GB that were vacuum-truncated.
> 
> 
> 
> Problem Description
> 
> When taking incremental backups of relations that span multiple segments, if the relation is truncated during VACUUM (after the base backup but before the incremental one), pg_combinebackup fails with:
> 
> ```
> 
> file "%s" has truncation block length %u in excess of segment size %u
> 
> ```
> 
> pg_basebackup itself completes without errors, but the resulting incremental backup cannot be restored.
> 
> 
> 
> Root Cause
> 
> In segmented relations, a VACUUM that truncates blocks sets a limit_block in the WAL summary. The incremental restore logic miscalculates truncation_block_length when processing segment 0…N, because it compares the segment-local size with a relation-wide limit.
> 
> In src/backend/backup/basebackup_incremental.c:
> 
> ```
> 
> *truncation_block_length = size / BLCKSZ;
> 
> if (BlockNumberIsValid(limit_block))
> 
> {
> 
>     unsigned relative_limit = limit_block - segno * RELSEG_SIZE;
> 
>     if (*truncation_block_length < relative_limit)   /* ← problematic */
> 
>         *truncation_block_length = relative_limit;
> 
> }
> 
> ```
> 
> For example, if limit_block lies in segment 10, then relative_limit will be roughly 9 * RELSEG_SIZE while processing segment 0. This forces truncation_block_length far beyond the actual segment size, leading to a segment length larger than RELSEG_SIZE and eventually the restore error.
> 
> 
> 
> Reproduction Steps
> 
> 
> Create a table larger than 1 GB (multiple segments).
> 
> Take a full base backup.
> 
> Delete rows that occupy the end of the relation.
> 
> Run VACUUM (VERBOSE, TRUNCATE) to ensure blocks are removed.
> 
> (optional) Confirm that the WAL summary includes a limit entry for the relation.
> 
> Take an incremental backup with pg_basebackup.
> 
> Attempt to restore using pg_combinebackup.
> 
> Observe the truncation block length error.
> 
> 
> 
> 
> Patch
> 
> A patch correcting this logic is attached, and I’m happy to provide additional details or revisions if helpful.
> 
> 
> 
> Best regards,
> 
> Oleg Tkachenko
> 
> 
> <bug_truncation_block_length.patch>
> 
> 
> 
> 
>

  3. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    amul sul <sulamul@gmail.com> — 2025-12-15T14:01:05Z

    On Mon, Dec 15, 2025 at 4:39 PM Oleg Tkachenko <oatkachenko@gmail.com> wrote:
>
>  [....]
>
> A patch correcting this logic is attached, and I’m happy to provide additional details or revisions if helpful.
>

Thanks for the reproducer; I can see the reported issue, but I am not
quite sure the proposed fix is correct and might break other cases (I
haven't tried constructed that case yet) but there is a comment
detailing that case just before the point where you are planning to do
the changes:

    /*
     * The truncation block length is the minimum length of the reconstructed
     * file. Any block numbers below this threshold that are not present in
     * the backup need to be fetched from the prior backup. At or above this
     * threshold, blocks should only be included in the result if they are
     * present in the backup. (This may require inserting zero blocks if the
     * blocks included in the backup are non-consecutive.)
     */

IIUC, we might need the original assignment logic as it is. But we
need to ensure that truncation_block_length is not set to a value that
exceeds RELSEG_SIZE.

Regards,
Amul

  4. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Robert Haas <robertmhaas@gmail.com> — 2025-12-15T16:35:25Z

    [ sorry for not noticing this thread sooner; thanks to Andres for
pointing me to it ]

On Mon, Dec 15, 2025 at 9:01 AM Amul Sul <sulamul@gmail.com> wrote:
> Thanks for the reproducer; I can see the reported issue, but I am not
> quite sure the proposed fix is correct and might break other cases (I
> haven't tried constructed that case yet) but there is a comment
> detailing that case just before the point where you are planning to do
> the changes:
>
>     /*
>      * The truncation block length is the minimum length of the reconstructed
>      * file. Any block numbers below this threshold that are not present in
>      * the backup need to be fetched from the prior backup. At or above this
>      * threshold, blocks should only be included in the result if they are
>      * present in the backup. (This may require inserting zero blocks if the
>      * blocks included in the backup are non-consecutive.)
>      */
>
> IIUC, we might need the original assignment logic as it is. But we
> need to ensure that truncation_block_length is not set to a value that
> exceeds RELSEG_SIZE.

I think you're right. By way of example, let's say that the current
length of the file is 200 blocks, but the limit block is 100 blocks
into the current segment. That means that the only blocks that we can
get from any previous backup are blocks 0-99. Blocks 100-199 of the
current segment are either mentioned in the WAL summaries we're using
for this backup, or they're all zeroes. We can't set the
truncation_block_length to a value greater than 100, or we'll go
looking for the contents of any zero-filled blocks in previous
backups, will will either fail or produce the wrong answer. But Oleg
is correct that we also shouldn't set it to a value greater than
RELSEG_SIZE. So my guess is that the correct fix might be something
like the attached (untested, for discussion).

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  5. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Oleg Tkachenko <oatkachenko@gmail.com> — 2025-12-15T18:46:23Z

    Hello Robert,

Thank you for the explanation.

At first, I also thought about clamping truncation_block_length to RELSEG_SIZE. But I hesitated because I thought the reconstructed relation file couldn’t be larger than relative_limit.

After reading the reconstruction code and the comments on top of the discussed block of code (many times), I finally understood that truncation_block_length is the minimum length of the reconstructed file, not just a safety limit. It determines which blocks must be fetched from older backups. So a simple clamp could change how reconstruction works if some blocks are included in incremental backups.

I’ve tested the version with the limit enforced to RELSEG_SIZE, and it works correctly.

Also, I’ve attached a patch based on your guidance. The changes are effectively the same as your suggested approach, but I would be happy to be listed as a contributor.

Regards,
Oleg Tkachenko


> On Dec 15, 2025, at 17:35, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> [ sorry for not noticing this thread sooner; thanks to Andres for
> pointing me to it ]
> 
> On Mon, Dec 15, 2025 at 9:01 AM Amul Sul <sulamul@gmail.com> wrote:
>> Thanks for the reproducer; I can see the reported issue, but I am not
>> quite sure the proposed fix is correct and might break other cases (I
>> haven't tried constructed that case yet) but there is a comment
>> detailing that case just before the point where you are planning to do
>> the changes:
>> 
>>    /*
>>     * The truncation block length is the minimum length of the reconstructed
>>     * file. Any block numbers below this threshold that are not present in
>>     * the backup need to be fetched from the prior backup. At or above this
>>     * threshold, blocks should only be included in the result if they are
>>     * present in the backup. (This may require inserting zero blocks if the
>>     * blocks included in the backup are non-consecutive.)
>>     */
>> 
>> IIUC, we might need the original assignment logic as it is. But we
>> need to ensure that truncation_block_length is not set to a value that
>> exceeds RELSEG_SIZE.
> 
> I think you're right. By way of example, let's say that the current
> length of the file is 200 blocks, but the limit block is 100 blocks
> into the current segment. That means that the only blocks that we can
> get from any previous backup are blocks 0-99. Blocks 100-199 of the
> current segment are either mentioned in the WAL summaries we're using
> for this backup, or they're all zeroes. We can't set the
> truncation_block_length to a value greater than 100, or we'll go
> looking for the contents of any zero-filled blocks in previous
> backups, will will either fail or produce the wrong answer. But Oleg
> is correct that we also shouldn't set it to a value greater than
> RELSEG_SIZE. So my guess is that the correct fix might be something
> like the attached (untested, for discussion).
> 
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com


  6. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Robert Haas <robertmhaas@gmail.com> — 2025-12-15T20:13:31Z

    On Mon, Dec 15, 2025 at 1:46 PM Oleg Tkachenko <oatkachenko@gmail.com> wrote:
> Also, I’ve attached a patch based on your guidance. The changes are effectively the same as your suggested approach, but I would be happy to be listed as a contributor.

You'll certain be listed as the reporter for this issue when a fix is
committed. If you want to be listed as a co-author of the patch, I
think it is fair to say that it will need to contain some code written
by you. For example, maybe you would like to try writing a TAP test
case that fails without this fix and passes with it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  7. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Chao Li <li.evan.chao@gmail.com> — 2025-12-16T08:05:56Z

    
> On Dec 16, 2025, at 00:35, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> [ sorry for not noticing this thread sooner; thanks to Andres for
> pointing me to it ]
> 
> On Mon, Dec 15, 2025 at 9:01 AM Amul Sul <sulamul@gmail.com> wrote:
>> Thanks for the reproducer; I can see the reported issue, but I am not
>> quite sure the proposed fix is correct and might break other cases (I
>> haven't tried constructed that case yet) but there is a comment
>> detailing that case just before the point where you are planning to do
>> the changes:
>> 
>>    /*
>>     * The truncation block length is the minimum length of the reconstructed
>>     * file. Any block numbers below this threshold that are not present in
>>     * the backup need to be fetched from the prior backup. At or above this
>>     * threshold, blocks should only be included in the result if they are
>>     * present in the backup. (This may require inserting zero blocks if the
>>     * blocks included in the backup are non-consecutive.)
>>     */
>> 
>> IIUC, we might need the original assignment logic as it is. But we
>> need to ensure that truncation_block_length is not set to a value that
>> exceeds RELSEG_SIZE.
> 
> I think you're right. By way of example, let's say that the current
> length of the file is 200 blocks, but the limit block is 100 blocks
> into the current segment. That means that the only blocks that we can
> get from any previous backup are blocks 0-99. Blocks 100-199 of the
> current segment are either mentioned in the WAL summaries we're using
> for this backup, or they're all zeroes. We can't set the
> truncation_block_length to a value greater than 100, or we'll go
> looking for the contents of any zero-filled blocks in previous
> backups, will will either fail or produce the wrong answer. But Oleg
> is correct that we also shouldn't set it to a value greater than
> RELSEG_SIZE. So my guess is that the correct fix might be something
> like the attached (untested, for discussion).
> 
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com
> <v1-0001-Don-t-set-the-truncation_block_length-rather-than.patch>


The change looks good to me. Only nitpick is:

```
Subject: [PATCH v1] Don't set the truncation_block_length rather than
 RELSEG_SIZE.
```

I guess you meant to say “larger (or greater) than” instead of “rather than”.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

  8. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Robert Haas <robertmhaas@gmail.com> — 2025-12-16T18:43:31Z

    On Tue, Dec 16, 2025 at 3:06 AM Chao Li <li.evan.chao@gmail.com> wrote:
> I guess you meant to say “larger (or greater) than” instead of “rather than”.

Yes, thanks.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  9. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Oleg Tkachenko <oatkachenko@gmail.com> — 2025-12-16T21:55:02Z

    Hello, Robert

I’ve created a small test that reproduces the issue. With the proposed fix applied, the test passes, and the reconstruction behaves as expected.

I’m attaching the test for review. Please let me know if this looks OK or if you would like it changed.



Regards,

Oleg



> On Dec 15, 2025, at 21:13, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Mon, Dec 15, 2025 at 1:46 PM Oleg Tkachenko <oatkachenko@gmail.com> wrote:
>> Also, I’ve attached a patch based on your guidance. The changes are effectively the same as your suggested approach, but I would be happy to be listed as a contributor.
> 
> You'll certain be listed as the reporter for this issue when a fix is
> committed. If you want to be listed as a co-author of the patch, I
> think it is fair to say that it will need to contain some code written
> by you. For example, maybe you would like to try writing a TAP test
> case that fails without this fix and passes with it.
> 
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com

  10. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    amul sul <sulamul@gmail.com> — 2025-12-18T06:05:11Z

    On Wed, Dec 17, 2025 at 3:25 AM Oleg Tkachenko <oatkachenko@gmail.com> wrote:
>
> Hello, Robert
>
> I’ve created a small test that reproduces the issue. With the proposed fix applied, the test passes, and the reconstruction behaves as expected.
>
> I’m attaching the test for review. Please let me know if this looks OK or if you would like it changed.
>

Test looks good to me, but I have three suggestions as follow:

1. To minimize repetition in insert: use fillfactor 10, which is the
minimal we can set for a table, so that we can minimize tuples per
page. Use a longer string and lower count in repeat(), which I believe
helps the test become a bit faster.

2. I think we could add this test to the existing pg_combinebackup's
test file instead of creating a new file with a single-test. See the
attached version; it’s a bit smaller than your original patch, but
since I haven't copied all of your comments yet, I’ve marked it as
WIP.

3. Kindly combine the code fix and tests together into a single patch.

Regards,
Amul

  11. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Robert Haas <robertmhaas@gmail.com> — 2025-12-18T14:26:31Z

    On Thu, Dec 18, 2025 at 1:05 AM Amul Sul <sulamul@gmail.com> wrote:
> Test looks good to me, but I have three suggestions as follow:
>
> 1. To minimize repetition in insert: use fillfactor 10, which is the
> minimal we can set for a table, so that we can minimize tuples per
> page. Use a longer string and lower count in repeat(), which I believe
> helps the test become a bit faster.

I haven't checked how big a relation the test case creates, but it's
worth keeping in mind that the CI tests run on one platform with the
segment size set to six blocks. I think we should design the test case
with that in mind i.e. don't worry about catching the bug when the
segment size is 1GB, but make sure the test fails in CI without the
bug fix. Let's not rely on fillfactor -- the cost here is the disk
space and the time to write the blocks, not how many tuples they
actually contain.

> 2. I think we could add this test to the existing pg_combinebackup's
> test file instead of creating a new file with a single-test. See the
> attached version; it’s a bit smaller than your original patch, but
> since I haven't copied all of your comments yet, I’ve marked it as
> WIP.

-1. This kind of thing tends to make the tests harder to understand.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  12. Re: [BUG] [PATCH] pg_basebackup produces wrong incremental files after relation truncation in segmented tables

    Oleg Tkachenko <oatkachenko@gmail.com> — 2025-12-18T17:23:53Z

    Hi,

Here is a refactored test.

Now, it creates data depending on the relation block size, so it works even if the segment size is not standard. I tested it locally with segment_size_blocks = 6, and it works correctly.



I would be happy to hear your comments or suggestions.

Regards,

Oleg


> On Dec 18, 2025, at 15:26, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Dec 18, 2025 at 1:05 AM Amul Sul <sulamul@gmail.com> wrote:
>> Test looks good to me, but I have three suggestions as follow:
>> 
>> 1. To minimize repetition in insert: use fillfactor 10, which is the
>> minimal we can set for a table, so that we can minimize tuples per
>> page. Use a longer string and lower count in repeat(), which I believe
>> helps the test become a bit faster.
> 
> I haven't checked how big a relation the test case creates, but it's
> worth keeping in mind that the CI tests run on one platform with the
> segment size set to six blocks. I think we should design the test case
> with that in mind i.e. don't worry about catching the bug when the
> segment size is 1GB, but make sure the test fails in CI without the
> bug fix. Let's not rely on fillfactor -- the cost here is the disk
> space and the time to write the blocks, not how many tuples they
> actually contain.
> 
>> 2. I think we could add this test to the existing pg_combinebackup's
>> test file instead of creating a new file with a single-test. See the
>> attached version; it’s a bit smaller than your original patch, but
>> since I haven't copied all of your comments yet, I’ve marked it as
>> WIP.
> 
> -1. This kind of thing tends to make the tests harder to understand.
> 
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com