Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-07T19:48:17Z
Lists: pgsql-hackers 
On 07/04/2026 17:46, Ashutosh Bapat wrote:
> Here are patches with the test modules merged.
> 
> The merged module looks a bit rough to me and so does 0006. For
> example, I am not sure whether calling ShmemStructProtect() from
> init_fn is a good idea. See [1] for example. But init_fn is the last
> chance for the subsystem to touch and setup the resizable structure
> before it's opened to the wild. So, in the current infrastructure, I
> don't see any better place to call ShmemStructProtect() either. If you
> run tests after applying patch 0006, you will need to apply patch
> attached to [1] as well; otherwise the test will hang.

I haven't really looked at these resizeable patches before, except for 
how they would fit with the new shmem allocation API, so I have some 
very basic, high-level design questions:

> +/*
> + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> + *
> + * The new size must be within [minimum_size, maximum_size].  If the structure
> + * is being shrunk, the memory pages that are no longer needed are freed. If
> + * the structure is being expanded, the memory pages that are needed for the
> + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> + * pages are allocated for a resizable structure.
> + */
> +void
> +ShmemResizeStruct(const char *name, Size new_size)

This interface only allows shrinking and growing the allocated region at 
the end, but the underlying mechanism is madvise(MADV_REMOVE) and 
madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e. 
freeing memory in the middle of a region. Do we gain anything by 
restricting ourselves to changing the size at the end? It seems to me 
that it could be handy to punch holes for some use cases.

What's the portability story? I understand that this is Linux-only at 
the moment, but what platforms can we support in the future, and what's 
the effort? I think BSD's have similar capabilities with plain mmap() 
and MADV_FREE if I read the man pages right. What about macOS and 
Windows? This doesn't necessarily need to be fully portable, if some 
OS's don't have the capabilities we need, but would be nice to know 
what's possible.

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.