Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, Matthias van de Meent <boekewurm+postgres@gmail.com>, Robert Haas <robertmhaas@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-07T20:09:25Z
Lists: pgsql-hackers 
Hi,

On 2026-04-07 22:48:17 +0300, Heikki Linnakangas wrote:
> > +/*
> > + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> > + *
> > + * The new size must be within [minimum_size, maximum_size].  If the structure
> > + * is being shrunk, the memory pages that are no longer needed are freed. If
> > + * the structure is being expanded, the memory pages that are needed for the
> > + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> > + * pages are allocated for a resizable structure.
> > + */
> > +void
> > +ShmemResizeStruct(const char *name, Size new_size)
> 
> This interface only allows shrinking and growing the allocated region at the
> end, but the underlying mechanism is madvise(MADV_REMOVE) and
> madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e.
> freeing memory in the middle of a region. Do we gain anything by restricting
> ourselves to changing the size at the end? It seems to me that it could be
> handy to punch holes for some use cases.

Agreed. The hard part may be the "communication" with the user about how
granular the punches can be.  Because that will depend on things like
huge_pages, huge_page_size and may depend on what alignment you happened to
get.


> What's the portability story? I understand that this is Linux-only at the
> moment, but what platforms can we support in the future, and what's the
> effort? I think BSD's have similar capabilities with plain mmap() and
> MADV_FREE if I read the man pages right.

At least linux' MADV_FREE is only for private mappings. It's not clear in at
least freebsd's man page, but the described use case makes me suspect it may
be similar there.


> What about macOS and Windows? This doesn't necessarily need to be fully
> portable, if some OS's don't have the capabilities we need, but would be
> nice to know what's possible.

Looks like windows has OfferVirtualMemory
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-offervirtualmemory
but it's not clear to me if it actually does what we need when multiple
processes are attached.

I suspect it's going to be a lot easier once we're threaded...  The reason I
am ok with doing resizing this way before threading is because it's
architecturally pretty similar to what you'd want to do once threaded, so it's
not a huge dead end.  But I'm doubtful we'll find facilities that allow this
across processes in all operating systems...

Greetings,

Andres Freund

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.