Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Daniel Gustafsson <daniel@yesql.se>
Cc: Robert Haas <robertmhaas@gmail.com>, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-05T16:13:00Z
Lists: pgsql-hackers 
On 26/03/2026 20:31, Daniel Gustafsson wrote:
>> On 22 Mar 2026, at 01:14, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> * The request_fn callback is called in postmaster startup, at the same stage as the old shmem_request callback was. But in EXEC_BACKEND mode, it's *also* called in each backend.
> 
> Should the request_fn be told, via an argument, from where it is called?  It
> can be figured out but it's cleaner if all implementations will do it in the
> same way.  I don't have a direct case in mind where it would be needed, but I
> was recently digging into SSL passphrase reloading which has failure cases
> precisely becasue of this so am thinking out loud to avoid similar problems
> here.

Hmm, you mean adding an argument along the lines of:

static void
pgss_shmem_request(void *arg, bool attaching)
{
     ...
}

Perhaps. The idea is that a request callback should generally do the 
exact same thing whether it's called from postmaster or from backend 
startup, though. I worry that an argument like that makes it too 
tempting to have different logic. That said, there are a couple of 
places where I'm using IsUnderPostmaster for that purpose. For example, 
I have this in lwlock.c (in latest version I'm currently working on that 
I haven't posted yet):

/* Size of MainLWLockArray.  Only valid in postmaster. */
static int	num_main_array_locks;

/*
  * Request shmem space for user-defined tranches and the main LWLock array.
  */
static void
LWLockShmemRequest(void *arg)
{
     size_t        size;

     /* Space for user-defined tranches */
     ShmemRequestStruct(.name = "LWLock tranches",
                        .size = sizeof(LWLockTrancheShmemData),
                        .ptr = (void **) &LWLockTranches,
         );

     /* Space for the LWLock array */
     if (!IsUnderPostmaster)
     {
         num_main_array_locks = NUM_FIXED_LWLOCKS + 
NumLWLocksForNamedTranches();
         size = num_main_array_locks * sizeof(LWLockPadded);
     }
     else
         size = SHMEM_ATTACH_UNKNOWN_SIZE;

     ShmemRequestStruct(.name = "Main LWLock array",
                        .size = size,
                        .ptr = (void **) &MainLWLockArray,
         );
}

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.