Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-04-04T00:49:25Z
Lists: pgsql-hackers

Attachments

On 02/04/2026 09:58, Ashutosh Bapat wrote:
> On Wed, Apr 1, 2026 at 11:47 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>>> + /*
>>> + * Extra space to reserve in the shared memory segment, but it's not part
>>> + * of the struct itself. This is used for shared memory hash tables that
>>> + * can grow beyond the initial size when more buckets are allocated.
>>> + */
>>> + size_t extra_size;
>>>
>>> When we introduce resizable structures (where even the hash table
>>> directly itself could be resizable), we will introduce a new field
>>> max_size which is easy to get confused with extra_size. Maybe we can
>>> rename extra_size to something like "auxilliary_size" to mean size of
>>> the auxiliary parts of the structure which are not part of the main
>>> struct itself.
>>>
>>> + /*
>>> + * max_size is the estimated maximum number of hashtable entries. This is
>>> + * not a hard limit, but the access efficiency will degrade if it is
>>> + * exceeded substantially (since it's used to compute directory size and
>>> + * the hash table buckets will get overfull).
>>> + */
>>> + size_t max_size;
>>> +
>>> + /*
>>> + * init_size is the number of hashtable entries to preallocate. For a
>>> + * table whose maximum size is certain, this should be equal to max_size;
>>> + * that ensures that no run-time out-of-shared-memory failures can occur.
>>> + */
>>> + size_t init_size;
>>>
>>> Everytime I look at these two fields, I question whether those are the
>>> number of entries (i.e. size of the hash table) or number of bytes
>>> (size of the memory). I know it's the former, but it indicates that
>>> something needs to be changed here, like changing the names to have
>>> _entries instead of _size, or changing the type to int64 or some such.
>>> Renaming to _entries would conflict with dynahash APIs since they use
>>> _size, so maybe the latter?
>>
>> I hear you, but I didn't change these yet. If we go with the patches
>> from the "Shared hash table allocations" thread, max_size and init_size
>> will be merged into one. I'll try to settle that thread before making
>> changes here.
> 
> Will review those patches next.

Those are now committed, and here's a new version rebased over those 
changes. The hash options is now called 'nelems', and the 'extra_size' 
in ShmemStructOpts is gone.

Plus a bunch of other fixes and cleanups. I also reordered and 
re-grouped the patches a little, into more logical increments I hope.

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.