Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-03-06T14:12:57Z
Lists: pgsql-hackers

Attachments

I spent some time cleaning up the new registration machinery. I didn't 
look at the "resizeable" part yet, but it should fit in nicely, as 
Ashutosh demonstrated.

I added a lot of comments, changed the existing docs on how to allocate 
shared memory in extensions to use the machinery, and tons of other 
cleanups. I'm getting pretty happy with this, but there are a couple of 
weak spots:

Firstly, I'm not sure what to do with ShmemRegisterHash() and the 
'HASHCTL *infoP' argument to it. I feel it'd be nicer if the HASHCTL was 
just part of the ShmemHashDesc struct, but I'm not sure if that fits all 
the callers. I'll have to try that out I guess.

Secondly, I'm not 100% happy with the facilities we provide to 
extensions. The lifecycle of _PG_init() and shmem_request/startup_hooks 
is a little messy. The status quo is that a shared library gets control 
in three different places:

1. _PG_init() gets called early at postmaster startup, if the library is 
in shared_preload_libraries. If it's not in shared_preload_libraries, it 
gets called whenever the module is loaded.

2. The library can install a shmem_request_hook, which gets called early 
at postmaster startup, but after initializing the MaxBackends GUC. It 
only gets called when the library is loaded via shared_preload_libraries.

3. The library can install a shmem_startup_hook. It gets called later at 
postmaster startup, after the shared memory segment has been allocated. 
In EXEC_BACKEND mode it also gets called at backend startup. It does not 
get called if the library is not listed in shared_preload_libraries.

None of these is quite the right moment to call the new 
ShmemRegisterStruct() function. _PG_init() is too early if the extension 
needs MaxBackends for sizing the shared memory area. shmem_request_hook 
is otherwise good, but in EXEC_BACKEND mode, the ShmemRegisterStruct() 
function needs to also be called backend startup and shmem_request_hook 
is not called at backend startup. shmem_startup_hook() is too late.

For now, I documented that an extension should call 
ShmemRegisterStruct() from _PG_init(), but may adjust the size in the 
shmem_request_hook, if needed.

Another wrinkle here is that you still need the shmem_request_hook, if 
you want to call RequestNamedLWLockTranche(). It cannot be called from 
_PG_init(). I'm not sure why that is.

So I think that requires a little more refactoring, I think an extension 
should need to use shmem_request/startup_hook with the new APIs anymore. 
It should provide more ergonomic callbacks or other mechanisms to 
accomplish the same things.

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.