Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Matthias van de Meent <boekewurm+postgres@gmail.com>

View thread
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-02T22:10:21Z
Lists: pgsql-hackers 
On Wed, 1 Apr 2026 at 20:17, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> Yet another version attached (also available at:
> https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-9). The
> main change is the shape of the ShmemRequest*() calls:

I didn't read the whole thread, as it's quite long, but did look at
the patchset for a while to figure out where it's going.

0005:
A few assorted comments:

While I do think it's an improvement over the current APIs, the
improvement seems to be mostly concentrated in the RequestStruct/Hash
department, with only marginal improvements in RegisterShmemCallbacks.
I feel like it's missing the important part: I'd like
direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
ShmemRequestStruct/Hash had a size callback as alternative to the size
field (which would then be called after preload_libraries finishes)
then that would be sufficient for most shmem allocations, and it'd
simplify shmem management for most subsystems.
We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
allow conditionally allocated shmem areas (e.g. those used in aio),
but I think that, in general, we shouldn't need a separate callback
function just to get started registering shmem structures.

I also noticed that ShmemCallbacks.%_arg are generally undocumented,
and I couldn't find any users in core (at the end of the patchset)
that actually use the argument. Could it be I missed something?

I don't understand the use of ShmemStructDesc. They generally/always
are private to request_fn(), and their fields are used exclusively
inside the shmem mechanisms, with no reads of its fields that can't
already be deduced from context. Why do we need that struct
everywhere?

> +++ b/src/backend/storage/ipc/shmem.c
[...]
> +    /* Check that it's not already registered in this process */
> +    foreach_ptr(ShmemStructDesc, existing, pending_shmem_requests)
> +    {
> +        if (strcmp(existing->name, options->name) == 0)
> +            ereport(ERROR,
> +                    (errmsg("shared memory struct \"%s\" is already registered",
> +                            options->name)));
> +    }
> +
> +    request = palloc(sizeof(ShmemRequest));
> +    request->options = options;
> +    request->desc = desc;
> +    request->kind = kind;
> +    pending_shmem_requests = lappend(pending_shmem_requests, request);

Apparently, pending_shmem_requests is a list of ShmemRequest, but the
iteration just above on the same list assumes ShmemStructDesc, which
seems wrong to me.

00017:
I like this idea, but I think it missed its chance to make good on an
opportunity to reduce waste in alignments: We know which structs we're
going to allocate at which alignments, so we could save space by
packing the structs. I don't expect it to save much, but it could be a
few 100 of kbs with a few BLCKSZ-aligned allocations.


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.