Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-03-12T18:41:03Z
Lists: pgsql-hackers 
On 12/03/2026 19:03, Robert Haas wrote:
> On Fri, Mar 6, 2026 at 9:13 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> 1. _PG_init() gets called early at postmaster startup, if the library is
>> in shared_preload_libraries. If it's not in shared_preload_libraries, it
>> gets called whenever the module is loaded.
>>
>> 2. The library can install a shmem_request_hook, which gets called early
>> at postmaster startup, but after initializing the MaxBackends GUC. It
>> only gets called when the library is loaded via shared_preload_libraries.
>>
>> 3. The library can install a shmem_startup_hook. It gets called later at
>> postmaster startup, after the shared memory segment has been allocated.
>> In EXEC_BACKEND mode it also gets called at backend startup. It does not
>> get called if the library is not listed in shared_preload_libraries.
>>
>> None of these is quite the right moment to call the new
>> ShmemRegisterStruct() function. _PG_init() is too early if the extension
>> needs MaxBackends for sizing the shared memory area. shmem_request_hook
>> is otherwise good, but in EXEC_BACKEND mode, the ShmemRegisterStruct()
>> function needs to also be called backend startup and shmem_request_hook
>> is not called at backend startup. shmem_startup_hook() is too late.
> 
> I believe that the design goal of
> 4f2400cb3f10aa79f99fba680c198237da28dd38 was to make it so that people
> who had working extensions already didn't need to change their code,
> but those for whom the restrictions of doing things in _PG_init were
> annoying would have a workable alternative. I think that's a pretty
> good goal, although I don't feel we absolutely have to stick to it. It
> could easily be worth breaking that if we get something cool out of
> it. But is there a reason we can't make it so that this new mechanism
> can be used either from _PG_init() or shmem_startup_hook()? (I assume
> there is or you likely would have done it already, but it's not clear
> to me what that reason is.)

shmem_startup_hook() is too late. The shmem structs need to be 
registered at postmaster startup before the shmem segment is allocated, 
so that we can calculate the total size needed.

I'm currently leaning towards _PG_init(), except for allocations that 
depend on MaxBackends. For those, you can install a shmem_request_hook 
that sets the size in the descriptor. In other words, you can leave the 
'size' as empty in _PG_init(), but set it later in the shmem_request_hook.

Another option is to add a new bespoken callback in the descriptor for 
such size adjustments, which would get called at the same time as 
shmem_request_hook. That might be a little more ergonomic, there would 
no longer be any need for extensions to use the old 
shmem_request/startup_hooks with the new ShmemRegisterStruct() mechanism.

Except that you'd still need them for RequestNamedLWLockTranche(). I 
wonder if we should recommend extensions to embed the LWLock struct into 
their shared memory struct and use the LWLockInitialize() and 
LWLockNewTrancheId() functions instead. That fits the new 
ShmemRegisterStruct() API a little better than RequestNamedLWLockTranche().

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.