Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

View thread
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-03-30T04:50:22Z
Lists: pgsql-hackers 
On Sat, Mar 28, 2026 at 4:47 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> Thanks, will incorporate your comments in next version. Replying to just
> a few of them here:
>
> On 27/03/2026 09:01, Ashutosh Bapat wrote:
> > /* Restore basic shared memory pointers */
> > if (UsedShmemSegAddr != NULL)
> > + {
> > InitShmemAllocator(UsedShmemSegAddr);
> > + ShmemCallRequestCallbacks();
> >
> > It's not clear how we keep the list of registered callbacks across the
> > backends and also after restart in-sync. How do we make sure that the
> > callbacks registered at this time are the same callbacks registered
> > before creating the shared memory? How do we make sure that the
> > callbacks registered after the startup are also registered after
> > restart?
>
> On Unix systems, the registered callbacks are inherited by fork(), and
> also survive over crash restart. With EXEC_BACKEND, the assumption is
> that calling a library's _PG_init() function will register the same
> callbacks every time. We make the same assumption today with the
> shmem_startup hook.
>

RegisterShmemCallbacks() may be called after the startup, and it will
add new areas to the shared memory. How are those registries synced
across the backends? From your answer below, those registries are not
synced across backends. They will be wiped out by the restart and
won't be registered again. Is that right? I think we need to document
this fact and also the need to call RegisterShmemCallbacks() from all
the backends where the new areas are required after the startup.

Sorry, my question was not complete.

> > +void
> > +RegisterShmemCallbacks(const ShmemCallbacks *callbacks)
> > ... snip ...
> > + foreach(lc, requested_shmem_areas)
> >
> > Doesn't this list contain all the areas, not just registered in this
> > instance of the call. Does that mean that we need to have all the
> > attach functions idempotent? Why can't we deal with the newly
> > registered areas only?
>
> registered_shmem_areas is supposed to be empty when the function is
> entered. There's an assertion for that too before the foreach().
>
> However, it's missing this, after processing the list:
>
>          list_free_deep(requested_shmem_areas);
>          requested_shmem_areas = NIL;
>
> Because of that, this will fail if you load multiple extensions that
> call RegisterShmemCallbacks() in the same session. Will fix that.


--
Best Wishes,
Ashutosh Bapat

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.