Horton

Re: Proposal: Limitations of palloc inside checkpointer

Xuneng Zhou <xunengzhou@gmail.com>

View thread View on postgr.es
From: Xuneng Zhou <xunengzhou@gmail.com>
To: Alexander Korotkov <aekorotkov@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>, Maxim Orlov <orlovmg@gmail.com>, Andres Freund <andres@anarazel.de>, Ekaterina Sokolova <e.sokolova@postgrespro.ru>, pgsql-hackers@lists.postgresql.org
Date: 2025-06-03T10:16:12Z
Lists: pgsql-hackers 
Hi Alexander,

Thanks a lot for reviewing!

> I have few notes about that:
> 1) Should we make CompactCheckpointerRequestQueue() process the queue
> of checkpoint requests in smaller parts for the same reason we do this
> in AbsorbSyncRequests()? That would require significant redesign of
> the algorithm, but still.

In AbsorbSyncRequests, we process requests incrementally in batches to
avoid allocating more than 1 GB of memory, which would lead to
repeated failure. I think this is less concerning in
CompactCheckpointerRequestQueue, because if we caps num_requests at 10
million, the hash table peaks at ~500 MB and skip_slot[] at ~10
MB—both under 1 GB.


> 2) That's pretty independent to the changes by the patch, but should
> CompactCheckpointerRequestQueue() fill the gaps with entries from the
> tail instead of rewriting the whole queue?  That might be a bit
> faster.
This optimization would be quite helpful for compacting large queues.
For small ones, it may also add extra costs. Can we use a hybrid
approach? If it's independent, should we create a standalone patch for
it?

> 3) For sure, we wouldn't backpatch this.  Can we prepare some simple
> solution for back branches?  Perhaps, just introduction of
> MAX_CHECKPOINT_REQUESTS is enough to save us from allocations larger
> than 1GB.
>
I think this would work well for back branches.

Commits

  1. Fix checkpointer shared memory allocation

  2. Limit checkpointer requests queue size

  3. Process sync requests incrementally in AbsorbSyncRequests