Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

View thread
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Date: 2026-03-24T15:32:00Z
Lists: pgsql-hackers

Attachments

On Sun, Mar 22, 2026 at 5:44 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
>
> I split this into more incremental patches. The first few are just
> refactorings that are probably useful on their own.

Here are some comments on the patches

0001
+
+# Enable pg_stat_statements to test restart of shared_preload_libraries.
+$node->append_conf(
+ 'postgresql.conf',
+ qq{shared_preload_libraries = 'pg_stat_statements'
+pg_stat_statements.max = 50000
+compute_query_id = 'regress'
+});
+

In order to make sure that the shared memory and LWLocks for
pg_stat_statements are initialized after crash restart, we need to at
least query pg_stat_statements after the restart or do something so
that it's shared memory is used. Also, we can check whether the shared
memory structures are created by querying pg_shmem_allocations after
the restart.

0002
void
InitShmemAllocator(PGShmemHeader *seghdr)
{

The new ShmemIndex initialization code is cleaner and more
straightforward. It avoids the recursive nature of ShmemInitHash.
However with this change it's hard to keep track of all the
initialization steps and their dependencies. Attached is a patch that
makes small adjustments to the code to make it more clear.

Use of variable hash_size is actually misleading since it's not the
size of the hash table but the expected/max number of entries in it.
Removing it makes code more readable.

0003, 0005, 0006 is straight forward, no comments. Usually these
patches make the code more readable. I will review it more when I see
the patches that use this refactoring.

0004 I traced back the placement of CreateLWLock in history. It feels
like the patch is moving it to its intended place since the function
was introduced. It needs a comment to explain why the function is not
called from CreateOrAttachShmemStructs where other in-memory
structures are allocated.

0007 without using new APIs is not necessarily a win. So I would
suggest committing it along with the refactoring patch.

If we are going to commit these patches separately, I would suggest
squashing all predicate.c patches into one commit.

I will continue from 0008 tomorrow.

-- 
Best Wishes,
Ashutosh Bapat

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.