Horton

Thread

Commits

  1. Fix out-of-bounds access in autoprewarm worker

  2. Refactor autoprewarm_database_main() in preparation for read stream

  1. Autoprewarm workers terminated due to a segmentation fault

    Glauber Batista <glauberrbatista@gmail.com> — 2026-06-09T18:37:24Z

    Hello,

I have an issue with the autoprewarm workers segfaulting during the service
restart. Sometimes, it successfully restarts after a few tries, but usually
I need to remove the autoprewarm.blocks file. My setup consists of a
primary server with two replicas and all of them present the same issue. I
have been using this setup for several years with no issues, but since I
upgraded to Postgres 18 I'm having it. This is a production database.

Details:

Postgres Version: PostgreSQL 18.3 on aarch64-unknown-linux-gnu, compiled by
gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit

This is the related settings I'm using in postgresql.conf
```
shared_preload_libraries = 'pg_stat_statements,pg_prewarm'
pg_prewarm.autoprewarm = True
pg_prewarm.autoprewarm_interval = 300s
```

I'm using systemd to manage Postgres, but it also happens if I start
postgres using `pg_ctl`. So I ruled out a systemd issue.

This is the error message I'm seeing.
```
LOG:  restored log file "000000010000079200000015" from archive
LOG:  consistent recovery state reached at 792/15A586C8
LOG:  database system is ready to accept read-only connections
LOG:  restored log file "000000010000079200000016" from archive
LOG:  restored log file "000000010000079200000017" from archive
LOG:  background worker "autoprewarm worker" (PID 2350) was terminated by
signal 11: Segmentation fault
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted while in recovery at log time
2026-06-09 17:34:19 UTC
HINT:  If this has occurred more than once some data might be corrupted and
you might need to choose an earlier recovery target.
LOG:  restored log file "00000001000007920000000D" from archive
LOG:  restored log file "0000000100000791000000F8" from archive
LOG:  entering standby mode
LOG:  redo starts at 791/F804CB38
LOG:  database system is ready to accept read-only connections
LOG:  restored log file "0000000100000791000000F9" from archive
LOG:  restored log file "0000000100000791000000FA" from archive
LOG:  restored log file "0000000100000791000000FB" from archive
LOG:  background worker "autoprewarm worker" (PID 2522) was terminated by
signal 11: Segmentation fault
LOG:  terminating any other active server processes
FATAL:  could not restore file "0000000100000791000000FC" from archive:
child process was terminated by signal 3: Quit
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted while in recovery at log time
2026-06-09 17:34:19 UTC
HINT:  If this has occurred more than once some data might be corrupted and
you might need to choose an earlier recovery target.
LOG:  restored log file "00000001000007920000000D" from archive
LOG:  restored log file "0000000100000791000000F8" from archive
LOG:  entering standby mode
```
Changing the debug level to DEBUG1 didn't show anything useful, so I'm not
pasting it here.

At first, I thought it could be somehow related to this bug:
https://www.datadoghq.com/blog/engineering/unraveling-a-postgres-segfault/,
but investigating a bit it does not seem to be the case. Either way, I
compiled Postgres again using CFLAGS="-O0 -g -fno-strict-aliasing" to check
if it was something related to the code optimization for ARM64, but the
issue persisted.

Then, I used gdb to get the core file. Since it has been quite some time
since I didn't debug anything written in C/C++, I used Claude to guide me.
Here's some info:
```
(gdb) frame 0
#0  0x0000f5a0c6003854 in autoprewarm_database_main (main_arg=0) at
autoprewarm.c:649
649                blk = block_info[i];
(gdb) list
644
645                read_stream_end(stream);
646
647                /* Advance i past all the blocks just prewarmed. */
648                i = p.pos;
649                blk = block_info[i];
650            }
651
652            relation_close(rel, AccessShareLock);
653            CommitTransactionCommand();
(gdb) p *block_info
$1 = {database = 0, tablespace = 0, filenumber = 0, forknum = MAIN_FORKNUM,
blocknum = 0}
(gdb) p p
$2 = {block_info = 0xf5a07f600000, pos = 131072, tablespace = 1663,
filenumber = 28197, forknum = MAIN_FORKNUM, nblocks = 65329}
(gdb) p stream
$3 = (ReadStream *) 0xb82a946671f0
(gdb) p *(ReadStream *) stream
$4 = {max_ios = 0, io_combine_limit = 0, ios_in_progress = 0, queue_size =
0, max_pinned_buffers = 225, forwarded_buffers = 0, pinned_buffers = 0,
distance = 1,
  initialized_buffers = 9, read_buffers_flags = 0, sync_mode = false,
batch_mode = true, advice_enabled = false, temporary = false,
buffered_blocknum = 4294967295,
  callback = 0xf5a0c600330c <apw_read_stream_next_block>,
callback_private_data = 0xfffff758cea8, seq_blocknum = 2403,
seq_until_processed = 4294967295,
  pending_read_blocknum = 2403, pending_read_nblocks = 0,
per_buffer_data_size = 0, per_buffer_data = 0x0, ios = 0xb82a94667618,
oldest_io_index = 8, next_io_index = 8,
  fast_path = false, oldest_buffer_index = 9, next_buffer_index = 9,
buffers = 0xb82a94667254}
(gdb) p p.pos
$5 = 131072
(gdb) p p.nblocks
$6 = 65329
(gdb) p apw_state->prewarm_stop_idx
$7 = 0
(gdb) p apw_state->prewarm_start_idx
$8 = 0
(gdb) p block_info[131072]
Cannot access memory at address 0xf5a07f880000
(gdb) p &block_info[131072]
$9 = (BlockInfoRecord *) 0xf5a07f880000
(gdb) p block_info[131071]
$10 = {database = 0, tablespace = 0, filenumber = 0, forknum =
MAIN_FORKNUM, blocknum = 0}
(gdb) p block_info[131070]
$11 = {database = 0, tablespace = 0, filenumber = 0, forknum =
MAIN_FORKNUM, blocknum = 0}
(gdb) p block_info[1]
$12 = {database = 0, tablespace = 0, filenumber = 0, forknum =
MAIN_FORKNUM, blocknum = 0}
(gdb) p apw_state->prewarmed_blocks
$13 = 0
(gdb) p *apw_state
$14 = {lock = {tranche = 0, state = {value = 0}, waiters = {head = 0, tail
= 0}}, bgworker_pid = 0, pid_using_dumpfile = 0, block_info_handle = 0,
database = 0,
  prewarm_start_idx = 0, prewarm_stop_idx = 0, prewarmed_blocks = 0}
(gdb) info line autoprewarm.c:649
Line 649 of "autoprewarm.c" starts at address 0xf5a0c600382c
<autoprewarm_database_main+936> and ends at 0xf5a0c600384c
<autoprewarm_database_main+968>.
(gdb) list autoprewarm.c:640,660
640                {
641                    apw_state->prewarmed_blocks++;
642                    ReleaseBuffer(buf);
643                }
644
645                read_stream_end(stream);
646
647                /* Advance i past all the blocks just prewarmed. */
648                i = p.pos;
649                blk = block_info[i];
650            }
651
652            relation_close(rel, AccessShareLock);
653            CommitTransactionCommand();
654        }
655
656        dsm_detach(seg);
657    }
658
659    /*
660     * Dump information on blocks in shared buffers.  We use a text
format here
```

I found out that some parts of the autoprewarm were re-written recently
here:
https://www.postgresql.org/message-id/flat/CAN55FZ3n8Gd%2BhajbL%3D5UkGzu_aHGRqnn%2BxktXq2fuds%3D1AOR6Q%40mail.gmail.com
and I think it could be related, given the data present in the dump.

Also, I inspected my data to ensure it was not the culprit. I got the
database and filenumber from `bt full` and run the following queries.
```
blk = {database = 23583, tablespace = 1663, filenumber = 28197, forknum =
MAIN_FORKNUM, blocknum = 49}
```
So I queried the database (23583) and the filenumber (28197) using
```
SELECT datname FROM pg_database WHERE oid = 23583;
\c <datname>
SELECT relname, relkind, relpages, pg_relation_filenode(oid) AS filenode
FROM pg_class
WHERE pg_relation_filenode(oid) = 28197;
```
and it returned
```
   relname    | relkind | relpages | filenode
--------------+---------+----------+----------
 lccss_cc_idx | i       |    64805 |    28197
```
So I think it's not stale data or shrunk relation caused by VACUUM,
TRUNCATE, etc.

I'm only reporting with the latest data I collected, but this issue has
been happening since April, when I upgraded the database.

All that said, it seems there's a missing guard-clause at line 649. I
didn't spend much time reading the code, but it's clearly accessing a
position in the array that is not allocated.

Let me know if any extra information is needed.


Best,
Glauber Cassiano Batista

  2. Re: Autoprewarm workers terminated due to a segmentation fault

    Matheus Alcantara <matheusssilv97@gmail.com> — 2026-06-09T21:06:09Z

    Hi,

On Tue Jun 9, 2026 at 3:37 PM -03, Glauber Batista wrote:
> I have an issue with the autoprewarm workers segfaulting during the service
> restart. Sometimes, it successfully restarts after a few tries, but usually
> I need to remove the autoprewarm.blocks file. My setup consists of a
> primary server with two replicas and all of them present the same issue. I
> have been using this setup for several years with no issues, but since I
> upgraded to Postgres 18 I'm having it. This is a production database.
>
> Details:
>
> [ ... ] 
>
> All that said, it seems there's a missing guard-clause at line 649. I
> didn't spend much time reading the code, but it's clearly accessing a
> position in the array that is not allocated.
>

Thank you for the report!

I've managed to reproduce the issue with the following:

create table test_warm (id int, data text);
-- insert enough rows to generate too many pages on an index
insert into test_warm select g, repeat('a', 100) from generate_series(1, 5000000) g;
create index warm_idx on test_warm(id);

-- force read the index entirely into shared_buffers
select count(*) from test_warm where id > 0;

Then pg_ctl stop and pg_ctl start will start failing with the following logs:

2026-06-09 17:47:40.924 -03 [23025] LOG:  shutting down
2026-06-09 17:47:40.925 -03 [23025] LOG:  checkpoint starting: shutdown fast
2026-06-09 17:47:41.033 -03 [23022] LOG:  database system is shut down
2026-06-09 17:47:49.830 -03 [23172] LOG:  starting PostgreSQL 19beta1 on aarch64-darwin, compiled by clang-17.0.0, 64-bit
2026-06-09 17:47:49.842 -03 [23172] LOG:  database system is ready to accept connections
2026-06-09 17:47:49.917 -03 [23172] LOG:  background worker "autoprewarm worker" (PID 23182) was terminated by signal 11: Segmentation fault: 11
2026-06-09 17:47:49.917 -03 [23172] LOG:  terminating any other active server processes
2026-06-09 17:47:49.918 -03 [23172] LOG:  all server processes terminated; reinitializing

Wondering if the following would be enough?

  /* Advance i past all the blocks just prewarmed. */
  i = p.pos;
+ if (i >= apw_state->prewarm_stop_idx)
+         break;
+
  blk = block_info[i];


With this change on a fresh start I got very similar shared hit buffers
on the first and second execution, so I think that the
autoprewarm.blocks was reloaded successfully?

postgres=# explain(analyze, costs off, timing off, summary off) select count(*) from test_warm where id > 0;
                                    QUERY PLAN
-----------------------------------------------------------------------------------
 Finalize Aggregate (actual rows=1.00 loops=1)
   Buffers: shared hit=15965 read=70242
   ->  Gather (actual rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=15965 read=70242
         ->  Partial Aggregate (actual rows=1.00 loops=3)
               Buffers: shared hit=15965 read=70242
               ->  Parallel Seq Scan on test_warm (actual rows=1666666.67 loops=3)
                     Filter: (id > 0)
                     Buffers: shared hit=15965 read=70242

postgres=# explain(analyze, costs off, timing off, summary off) SELECT count(*) FROM test_warm WHERE id > 0;
                                    QUERY PLAN
-----------------------------------------------------------------------------------
 Finalize Aggregate (actual rows=1.00 loops=1)
   Buffers: shared hit=15970 read=70237
   ->  Gather (actual rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=15970 read=70237
         ->  Partial Aggregate (actual rows=1.00 loops=3)
               Buffers: shared hit=15970 read=70237
               ->  Parallel Seq Scan on test_warm (actual rows=1666666.67 loops=3)
                     Filter: (id > 0)
                     Buffers: shared hit=15970 read=70237


Thoughts?

--
Matheus Alcantara
EDB: https://www.enterprisedb.com

  3. Re: Autoprewarm workers terminated due to a segmentation fault

    Tomas Vondra <tomas@vondra.me> — 2026-06-09T21:44:28Z

    
On 6/9/26 23:06, Matheus Alcantara wrote:
> Hi,
> 
> On Tue Jun 9, 2026 at 3:37 PM -03, Glauber Batista wrote:
>> I have an issue with the autoprewarm workers segfaulting during the service
>> restart. Sometimes, it successfully restarts after a few tries, but usually
>> I need to remove the autoprewarm.blocks file. My setup consists of a
>> primary server with two replicas and all of them present the same issue. I
>> have been using this setup for several years with no issues, but since I
>> upgraded to Postgres 18 I'm having it. This is a production database.
>>
>> Details:
>>
>> [ ... ] 
>>
>> All that said, it seems there's a missing guard-clause at line 649. I
>> didn't spend much time reading the code, but it's clearly accessing a
>> position in the array that is not allocated.
>>
> 
> Thank you for the report!
> 
> I've managed to reproduce the issue with the following:
> 
> create table test_warm (id int, data text);
> -- insert enough rows to generate too many pages on an index
> insert into test_warm select g, repeat('a', 100) from generate_series(1, 5000000) g;
> create index warm_idx on test_warm(id);
> 
> -- force read the index entirely into shared_buffers
> select count(*) from test_warm where id > 0;
> 
> Then pg_ctl stop and pg_ctl start will start failing with the following logs:
> 
> 2026-06-09 17:47:40.924 -03 [23025] LOG:  shutting down
> 2026-06-09 17:47:40.925 -03 [23025] LOG:  checkpoint starting: shutdown fast
> 2026-06-09 17:47:41.033 -03 [23022] LOG:  database system is shut down
> 2026-06-09 17:47:49.830 -03 [23172] LOG:  starting PostgreSQL 19beta1 on aarch64-darwin, compiled by clang-17.0.0, 64-bit
> 2026-06-09 17:47:49.842 -03 [23172] LOG:  database system is ready to accept connections
> 2026-06-09 17:47:49.917 -03 [23172] LOG:  background worker "autoprewarm worker" (PID 23182) was terminated by signal 11: Segmentation fault: 11
> 2026-06-09 17:47:49.917 -03 [23172] LOG:  terminating any other active server processes
> 2026-06-09 17:47:49.918 -03 [23172] LOG:  all server processes terminated; reinitializing
> 
> Wondering if the following would be enough?
> 
>   /* Advance i past all the blocks just prewarmed. */
>   i = p.pos;
> + if (i >= apw_state->prewarm_stop_idx)
> +         break;
> +
>   blk = block_info[i];
> 
So how does it get advanced past the prewarm_stop_idx? I've been unable
to reproduce it locally, maybe it's platform-specific. The original
report was from ARM, are you on ARM too, Matheus?

But AFAIK the code may not account for read stream callback updating the
pos to prewarm_stop_idx? The callback may end with (p->pos =
apw_state->prewarm_stop_idx), and that seems to be past the end of the
array.

That'd mean the proposed check is generally the correct way to fix this.
TBH it's not clear to me why this needs to set the *next* entry at the
end of the loop. Well, it does that so that the loop condition can use
'blk', but that seems a bit fragile / confusing, and no one noticed the
issue.

Maybe this would be a better way to write the while loop?

    while (i < apw_state->prewarm_stop_idx)
    {
        blk = block_info[i];

        if (blk.tablespace != tablespace ||
            blk.filenumber != filenumber)
            break;

        ...
    }


regards

-- 
Tomas Vondra

  4. Re: Autoprewarm workers terminated due to a segmentation fault

    Matheus Alcantara <matheusssilv97@gmail.com> — 2026-06-09T22:25:55Z

    On Tue Jun 9, 2026 at 6:44 PM -03, Tomas Vondra wrote:
> So how does it get advanced past the prewarm_stop_idx? I've been unable
> to reproduce it locally, maybe it's platform-specific. The original
> report was from ARM, are you on ARM too, Matheus?
>

Yes, I'm also on ARM. I also set pg_prewarm.autoprewarm_interval=10s on
postgresql.conf, not sure if it make more easier to reproduce.

> But AFAIK the code may not account for read stream callback updating the
> pos to prewarm_stop_idx? The callback may end with (p->pos =
> apw_state->prewarm_stop_idx), and that seems to be past the end of the
> array.
>

Yes, this is my understanding.

> That'd mean the proposed check is generally the correct way to fix this.
> TBH it's not clear to me why this needs to set the *next* entry at the
> end of the loop. Well, it does that so that the loop condition can use
> 'blk', but that seems a bit fragile / confusing, and no one noticed the
> issue.
>
> Maybe this would be a better way to write the while loop?
>
>     while (i < apw_state->prewarm_stop_idx)
>     {
>         blk = block_info[i];
>
>         if (blk.tablespace != tablespace ||
>             blk.filenumber != filenumber)
>             break;
>
>         ...
>     }
>
>

Is attached patch what are you sugesting? If yes, I agree that looks
better, it's more safe and easier to understand.

--
Matheus Alcantara
EDB: https://www.enterprisedb.com

  5. Re: Autoprewarm workers terminated due to a segmentation fault

    Tomas Vondra <tomas@vondra.me> — 2026-06-11T14:40:37Z

    On 6/10/26 00:25, Matheus Alcantara wrote:
> On Tue Jun 9, 2026 at 6:44 PM -03, Tomas Vondra wrote:
>> So how does it get advanced past the prewarm_stop_idx? I've been unable
>> to reproduce it locally, maybe it's platform-specific. The original
>> report was from ARM, are you on ARM too, Matheus?
>>
> 
> Yes, I'm also on ARM. I also set pg_prewarm.autoprewarm_interval=10s on
> postgresql.conf, not sure if it make more easier to reproduce.
> 
>> But AFAIK the code may not account for read stream callback updating the
>> pos to prewarm_stop_idx? The callback may end with (p->pos =
>> apw_state->prewarm_stop_idx), and that seems to be past the end of the
>> array.
>>
> 
> Yes, this is my understanding.
> 
>> That'd mean the proposed check is generally the correct way to fix this.
>> TBH it's not clear to me why this needs to set the *next* entry at the
>> end of the loop. Well, it does that so that the loop condition can use
>> 'blk', but that seems a bit fragile / confusing, and no one noticed the
>> issue.
>>
>> Maybe this would be a better way to write the while loop?
>>
>>     while (i < apw_state->prewarm_stop_idx)
>>     {
>>         blk = block_info[i];
>>
>>         if (blk.tablespace != tablespace ||
>>             blk.filenumber != filenumber)
>>             break;
>>
>>         ...
>>     }
>>
>>
> 
> Is attached patch what are you sugesting? If yes, I agree that looks
> better, it's more safe and easier to understand.
> 

Yeah, that's roughly what I had in mind.

After looking a bit closer, I think this bug was introduced by

    commit 6acab8bdbcda735ef47b1bb0ba2284d6c465cd88
    Author: Melanie Plageman <melanieplageman@gmail.com>
    Date:   Fri Apr 4 15:25:27 2025 -0400

    Refactor autoprewarm_database_main() in preparation for read stream

which happens to advance to the next block in a couple places

    blk = block_info[++i];

before we know it the incremented "i" is a valid element. The following
commit (d9c7911e1a5f adding the read stream) ends up doing the same
thing, except the index is incremented in a callback.

Melanie, do you agree with the proposed fix?


regards

-- 
Tomas Vondra

  6. Re: Autoprewarm workers terminated due to a segmentation fault

    Melanie Plageman <melanieplageman@gmail.com> — 2026-06-15T14:35:43Z

    On Thu, Jun 11, 2026 at 10:40 AM Tomas Vondra <tomas@vondra.me> wrote:
>
> After looking a bit closer, I think this bug was introduced by
>
>     commit 6acab8bdbcda735ef47b1bb0ba2284d6c465cd88
>     Author: Melanie Plageman <melanieplageman@gmail.com>
>     Date:   Fri Apr 4 15:25:27 2025 -0400
>
>     Refactor autoprewarm_database_main() in preparation for read stream
>
> which happens to advance to the next block in a couple places
>
>     blk = block_info[++i];
>
> before we know it the incremented "i" is a valid element. The following
> commit (d9c7911e1a5f adding the read stream) ends up doing the same
> thing, except the index is incremented in a callback.
>
> Melanie, do you agree with the proposed fix?

Thanks all for investigating this! I agree that the proposed fix is
correct (both v1 and v2). I am a bit torn between the two versions.
v1:

i = p.pos;
if (i >= apw_state->prewarm_stop_idx)
    break;
blk = block_info[i];

preserves what I had intended in that it sets blk as close to where i
was advanced as possible. I had wanted this behavior because we have
the "fast-forwarding" that happens in a few places in that loop, and I
was afraid it would get quite confusing. The other thing I wanted was
the invariant that blk is always the i'th block when exiting the loop.
This was in case future code restructured the outer loop and added
other break conditions and then used blk.

However, I do hate duplicating the loop bounds checking in two places.
And future code that restructures the control flow of this function is
going to have to be extremely careful anyway. The combination of
multiple nested loops, "fast-forwarding" behavior, and skipping
forward X number of blocks with the read stream made the requirements
of this function's control flow quite complex.

v2:

        while (i < apw_state->prewarm_stop_idx)
        {
            ForkNumber    forknum;
            BlockNumber nblocks;
            struct AutoPrewarmReadStreamData p;
            ReadStream *stream;
            Buffer        buf;

            blk = block_info[i];

            /* Stop when we reach a different relation. */
            if (blk.tablespace != tablespace ||
                blk.filenumber != filenumber)
                break;

has the advantage that we set blk directly before using it instead of
at the end of the previous iteration, which people may find less
confusing.

So, overall, I am fine with either approach. Whatever other hackers
find more clear is the right approach. In my attempt to make it clear,
I obviously made it incorrect -- which is far worse :)

- Melanie

  7. Re: Autoprewarm workers terminated due to a segmentation fault

    Tomas Vondra <tomas@vondra.me> — 2026-06-26T17:55:51Z

    FWIW I've committed the v2 fix (I liked it more), and I've backpatched
it to 18.


regards

-- 
Tomas Vondra

  8. Re: Autoprewarm workers terminated due to a segmentation fault

    Matheus Alcantara <matheusssilv97@gmail.com> — 2026-06-26T18:07:44Z

    On 26/06/26 14:55, Tomas Vondra wrote:
> FWIW I've committed the v2 fix (I liked it more), and I've backpatched
> it to 18.
> 

Thank you!

--
Matheus Alcantara
EDB: https://www.enterprisedb.com

  9. Re: Autoprewarm workers terminated due to a segmentation fault

    Melanie Plageman <melanieplageman@gmail.com> — 2026-06-29T15:12:50Z

    On Fri, Jun 26, 2026 at 1:55 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> FWIW I've committed the v2 fix (I liked it more), and I've backpatched
> it to 18.

Thank you so much for doing this. I had it on my list for next week,
but a few other things got ahead of it. I really, really appreciate
you taking on committing it!

- Melanie