Re: Better shared data structure management and resizable shared data structures

On Fri, Mar 13, 2026 at 5:11 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 12/03/2026 22:05, Robert Haas wrote:
> > On Thu, Mar 12, 2026 at 3:21 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >>>> I'm currently leaning towards _PG_init(), except for allocations that
> >>>> depend on MaxBackends. For those, you can install a shmem_request_hook
> >>>> that sets the size in the descriptor. In other words, you can leave the
> >>>> 'size' as empty in _PG_init(), but set it later in the shmem_request_hook.
> >>>
> >>> Why can't you just do the whole thing later?
> >>
> >> shmem_request_hook won't work in EXEC_BACKEND mode, because in
> >> EXEC_BACKEND mode, ShmemRegisterStruct() also needs to be called at
> >> backend startup.
> >>
> >> One of my design goals is to avoid EXEC_BACKEND specific steps so that
> >> if you write your extension oblivious to EXEC_BACKEND mode, it will
> >> still usually work with EXEC_BACKEND. For example, if it was necessary
> >> to call a separate AttachShmem() function for every shmem struct in
> >> EXEC_BACKEND mode, but which was not needed on Unix, that would be bad.
> >
> > That's *definitely* a good goal. A less important but still valuable
> > goal is to maximize the notational simplicity of the mechanism. Your
> > callback idea is elegant in theory but in practice it seems like it
> > might make it harder for people to get started quickly on a new
> > module, and having to create the object in one place and then fill in
> > the size in another sort of has the same problem. I don't really know
> > what to do about that, but it's something to think about. The
> > complexity of getting the details right is annoyingly high in this
> > area.
>
> Yeah. IMHO the existing shmem_request/startup_hook mechanism is pretty
> awkward too, and in most cases, the new mechanism is more convenient. It
> might be slightly less convenient for some things, but mostly it's
> better. Would you agree with that, or do you actually like the old hooks
> and ShmemInitStruct() better?

FWIW, I like the new way.

If your goal is to get rid of ShmemInitStruct, ShmemInitHash,
shmem_request/startup_hook, we could replace the hooks by another hook
which gets called after MaxBackends is initialized but before
CalculateShmemSize() gets called. All StructShmemRegister() can be
called in that hook. _PG_init() is used to register the hook as it's
done today. If we are going to deprecate the old hooks in a couple of
releases, we will need to maintain three hooks but given that we don't
have to wait for several releases for the extensions to adapt to the
new hooks, it should be only a temporary measure.

>
> One such wrinkle with ShmemRegisterStruct() in the patch now is that
> it's harder to do initialization that touches multiple structs or hash
> tables. Currently the callbacks are called in the same order that the
> structs are registered, so you can do all the initialization in the last
> struct's callback. The single pair of shmem_request/startup_hooks per
> module was more clear in that aspect. Fortunately, that kind of
> cross-struct dependencies are pretty rare. So I think it's fine. (The
> order that the callbacks are called needs be documented explicitly though).
>
> If we want to improve on that, one idea would be to introduce a
> ShmemRegisterCallbacks() function to register callbacks that are not
> tied to any particular struct and are called after all the per-struct
> callbacks.
>

I think as long as we stick to calling the init/attach functions in
the order of their registration (or any well defined suitable order),
the module can use perform initization/setup of individual structures
to which the callback is attached and also modify/use previously
registered structures or do the setup in the last registered
structure's init. I think this is more flexible that required; maybe
overwhelmingly flexible.

> >>> Yeah, I think RequestNamedLWLockTranche() might be fine if you just
> >>> need LWLocks, but if you need a bunch of resources, putting them all
> >>> into the same chunk of memory seems cleaner.
> >>
> >> Agreed. Then again, how often do you need just a LWLock (or multiple
> >> LWLocks)? Surely you have a struct you want to protect with the lock. I
> >> guess having shmem hash table but no struct would be pretty common, though.
> >
> > Yeah, we've developed an annoying number of different ways to do this
> > stuff. I don't entirely know how to fix that.
>
> Here's a new version that doubles down on the
> LWLockNewTrancheId+LWLockInitialize method, by changing the example in
> the docs, and contrib/pg_stat_statements, to use that method.
> RequestNamedLWLockTranche() still works, there are no changes to it,
> it's just not as convenient to use with ShmemRegisterStruct(). This has
> the advantage that we don't introduce yet another way of allocating LWLocks.

The new hook could be used to request LWLockTranche as well.

-- 
Best Wishes,
Ashutosh Bapat