Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Matthias van de Meent <boekewurm+postgres@gmail.com>

View thread
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-04T12:00:11Z
Lists: pgsql-hackers 
On Sat, 4 Apr 2026 at 02:45, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 03/04/2026 16:12, Ashutosh Bapat wrote:
> > On Fri, Apr 3, 2026 at 3:40 AM Matthias van de Meent
> > <boekewurm+postgres@gmail.com> wrote:
> >> While I do think it's an improvement over the current APIs, the
> >> improvement seems to be mostly concentrated in the RequestStruct/Hash
> >> department, with only marginal improvements in RegisterShmemCallbacks.
> >> I feel like it's missing the important part: I'd like
> >> direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
> >> ShmemRequestStruct/Hash had a size callback as alternative to the size
> >> field (which would then be called after preload_libraries finishes)
> >> then that would be sufficient for most shmem allocations, and it'd
> >> simplify shmem management for most subsystems.
> >> We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
> >> allow conditionally allocated shmem areas (e.g. those used in aio),
> >> but I think that, in general, we shouldn't need a separate callback
> >> function just to get started registering shmem structures.
> >>
> >> I also noticed that ShmemCallbacks.%_arg are generally undocumented,
> >> and I couldn't find any users in core (at the end of the patchset)
> >> that actually use the argument. Could it be I missed something?
>
> None of the current code currently uses it, that's correct. I felt it
> might become very handy in the future or in extensions, if you wanted to
> reuse the same function for initializing different shmem areas, for
> example.

That's cool, but if that common initialization path is common enough
to need special coding, then how come that this patch make PG use it?
I can think of many systems that "just" initialize a hash table or
"just" allocate a shmem area.

> It's a pretty common pattern to have an opaque pointer like
> that in any callbacks.

I agree that it's a rather common pattern, but from an OOP
perspective, shouldn't the argument be the ShmemCallbacks*? Users can
embed the struct to extend the data carried if they need it to.

> >> I don't understand the use of ShmemStructDesc. They generally/always
> >> are private to request_fn(), and their fields are used exclusively
> >> inside the shmem mechanisms, with no reads of its fields that can't
> >> already be deduced from context. Why do we need that struct
> >> everywhere?
> >
> > My resizable shared memory structure patches use it as a handle to the
> > structure to be resized.
>
> Right. And hash tables and SLRUs use a desc-like object already, so for
> symmetry it feels natural to have it for plain structs too.
> I wonder if we should make it optional though, for the common case that
> you have no intention of doing anything more with the shmem region that
> you'd need a desc for. I'm thinking you could just pass NULL for the
> desc pointer:
>
>      ShmemRequestStruct(NULL,
>          .name = "pg_stat_statements",
>          .size = sizeof(pgssSharedState),
>          .ptr = (void **) &pgss,
>      };

That would help, though I'd still wonder why we'd have separate Opts
and Desc structs. IIUC, they generally carry (exactly) the same data.

Maybe moving it into a `.handle` or `.desc` field in Shmem*Opts could
make that part of the code a bit cleaner; as it'd further clarify that
it's very much an optional field.

I'll check out your latest version in a bit.


Kind regards,

Matthias van de Meent

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.