Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

View thread
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-03-30T12:20:52Z
Lists: pgsql-hackers

Attachments

On Fri, Mar 27, 2026 at 12:31 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Wed, Mar 25, 2026 at 9:35 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Tue, Mar 24, 2026 at 9:02 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > >
> > > I will continue from 0008 tomorrow.
> > >
> >
> > I reviewed the documentation part of 0008. I have a few edits attached.
> >
> > I have just one comment that's not covered in the edits
> >
> > @@ -4254,8 +4254,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
> > <para>
> > Anonymous allocations are allocations that have been made
> > with <literal>ShmemAlloc()</literal> directly, rather than via
> > - <literal>ShmemInitStruct()</literal> or
> > - <literal>ShmemInitHash()</literal>.
> > + <literal>ShmemRequestStruct()</literal> or
> > + <literal>ShmemRequestHash()</literal>.
> > </para>
> >
> > ShmemInitStruct() and ShmemInitHash() are still the functions to
> > allocate named structures. If we are going to keep ShmemInitStruct()
> > and ShmemInitHash() around for a while, I think it is more accurate to
> > mention them in this sentence along with the new functions.
> >
> > Will continue reviewing the patch tomorrow.
> >
>
> Here's a complete review of 0008 (from version 7). I see you have
> already posted v8 which I have not looked at.

I rebased my resizable shared memory structures patch on top of your
v8 patch to check if newer APIs are still useful for resizable
structures. Attached is the resultant WIP patch. I will rebase and
finalize it once these patches are committed.

Here are some review comments coming out of that exercise.

The patch subtly changes what allocated_size means in ShmemIndexEntry.
Without this patch, the next structure started from entry->location +
allocated_bytes whereas now the given structure starts at a type
aligned address whereas the next structure may start anywhere after
size bytes from the type aligned address. I think these patches have
got it right (current head the code gets it correct since it uses the
same alignment for all the structures.). But I think the comments for
allocated_size should make it clear that it's not the size available
to the structure, as it used to be. The current comments and the code
in HEAD may make one think so.

What if the caller changes the ShmemStructDesc, which seems to be the
handle we are talking about upthread? Should we make it opaque so that
the callers can not play with it?

-- 
Best Wishes,
Ashutosh Bapat

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.