Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

View thread
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-07T14:46:59Z
Lists: pgsql-hackers

Attachments

On Tue, Apr 7, 2026 at 3:36 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > I have kept these two patches separate from the main patch so that I
> > can remove them if others feel they are not worth including in the
> > feature.
>
> Here are patches rebased on the latest HEAD. No conflicts just rebase.
>
> Here are differences from the previous patchset.
>
> o. There are two patches in this patchset now. a. 0001 which supports
> resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
> from the previous patchset. b. 0002 which is 0006 from the previous
> patchset and adds support for protecting resizable shared memory
> structures. 0003, which added diagnostics to investigate CFBot
> failure, from the previous patchset is not required anymore since all
> tests pass with CFBot.
>
> o. I have merged 0002 into 0001 from the previous patchset since with
> that patch all platforms are green on CFBot. The resizable shared
> memory test now uses /proc/self/smaps instead of /proc/self/status to
> find the amount of memory allocated in the main shared memory segment
> of PostgreSQL.
>
> o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
> would be useful to protect against accidental shrinkage of the
> resizable structures. It will help additional support for minimum
> sizes of GUCs like shared_buffers. It also makes it easy and intuitive
> to distinguish between fixed-size and resizable structures, and will
> be useful to find the minimum size of the shared memory segment.
>
> o. Merged 0005, which allows ABI compatibility between the binaries
> which support resizable shared memory and those which don't, into
> 0001. Apart from ABI compatibility, the code has lesser #ifdef blocks
> and thus easier to read and maintain.
>
> I didn't find it useful to keep 0004 and 0005 separate since they were
> interdependent and made review complicated and have higher chances of
> being acceptable.
>
> o. 0006 is still separate since I am not sure whether the
> functionality is absolutely needed at this time. In an offlist
> discussion, Andres mentioned that it is not strictly needed. The
> subsystem that uses the resizable shared memory can implement their
> own protection if required and integrate it in the subsystems specific
> synchronization. But Matthias thinks different. The API to add
> protection is platform dependent, so it's better to abstract it via
> shmem.c. If we decide to accept this patch, we should merge it into
> 0001 before committing.
>
> Also did some more cleanups and changed the name of the GUC
> have_resizable_shmem to have_resizable_shared_memory since shmem is an
> internal phrase.
>
> I am looking at merging the resizable_shmem module into test_shmem module next.

Here are patches with the test modules merged.

The merged module looks a bit rough to me and so does 0006. For
example, I am not sure whether calling ShmemStructProtect() from
init_fn is a good idea. See [1] for example. But init_fn is the last
chance for the subsystem to touch and setup the resizable structure
before it's opened to the wild. So, in the current infrastructure, I
don't see any better place to call ShmemStructProtect() either. If you
run tests after applying patch 0006, you will need to apply patch
attached to [1] as well; otherwise the test will hang.

[1] https://www.postgresql.org/message-id/CAExHW5uMQGvQH6GKaBZVtH4S9O13TwN+_0Vy1gUpAW=_T_AmRA@mail.gmail.com

-- 
Best Wishes,
Ashutosh Bapat

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.