Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Matthias van de Meent <boekewurm+postgres@gmail.com>

View thread
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-07T19:38:37Z
Lists: pgsql-hackers

Attachments

On Tue, 7 Apr 2026 at 16:47, Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Apr 7, 2026 at 3:36 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > I have kept these two patches separate from the main patch so that I
> > > can remove them if others feel they are not worth including in the
> > > feature.
> >
> > Here are patches rebased on the latest HEAD. No conflicts just rebase.
> >
> > Here are differences from the previous patchset.
> >
> > o. There are two patches in this patchset now. a. 0001 which supports
> > resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
> > from the previous patchset. b. 0002 which is 0006 from the previous
> > patchset and adds support for protecting resizable shared memory
> > structures. 0003, which added diagnostics to investigate CFBot
> > failure, from the previous patchset is not required anymore since all
> > tests pass with CFBot.
> >
> > o. I have merged 0002 into 0001 from the previous patchset since with
> > that patch all platforms are green on CFBot. The resizable shared
> > memory test now uses /proc/self/smaps instead of /proc/self/status to
> > find the amount of memory allocated in the main shared memory segment
> > of PostgreSQL.
> >
> > o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
> > would be useful to protect against accidental shrinkage of the
> > resizable structures. It will help additional support for minimum
> > sizes of GUCs like shared_buffers. It also makes it easy and intuitive
> > to distinguish between fixed-size and resizable structures, and will
> > be useful to find the minimum size of the shared memory segment.

I was thinking more along the lines of attached (incremental) patch
0003 for min/max sizing. I'd say it has a slightly more natural API,
but YMMV.

-----

I also noticed that it's probably not correct to "just" check and
complain about the size of a resizable shmem segment when you attach:
Without coordination about which startup size the shmem segment should
have, how could you get the current size state correct? And
cross-process coordination of size information before shmem is
attached is not really possible, not when you may have to deal with a
very slow to start backend.

----

Attached also 0004, which makes some small adjustments to shmem.c's
resize checks.

Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.