Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.

  1. Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-02-13T11:47:11Z

    Hi Heikki,
As discussed in [1], starting a new thread to discuss $Subject.

0001 in the attached patchset is same as the patch shared in [1]. For
completeness, I am copy-pasting from Heikki's email the description of
what this patch does

** quote **
Attached is a proof-of-concept of what I have in mind. Don't look too
closely at how it's implemented, it's very hacky and EXEC_BACKEND mode
is slightly broken, for example. The point is to demonstrate what the
callers would look like. I converted only a few subsystems to use the
new API, the rest still use ShmemInitStruct() and ShmemInitHash().

With this, initialization of a subsystem that defines a shared memory
area looks like this:

--------------

/* This struct lives in shared memory */
typedef struct
{
int field;
} FoobarSharedCtlData;

static void FoobarShmemInit(void *arg);

/* Descriptor for the shared memory area */
ShmemStructDesc FoobarShmemDesc = {
.name = "Foobar subsystem",
.size = sizeof(FoobarSharedCtlData),
.init_fn = FoobarShmemInit,
};

/* Pointer to the shared memory struct */
#define FoobarCtl ((FoobarSharedCtlData *) FoobarShmemDesc.ptr)

/*
* Register the shared memory struct. This is called once at
* postmaster startup, before the shared memory segment is allocated,
* and in EXEC_BACKEND mode also early at backend startup.
*
* For core subsystems, there's a list of all these functions in core
* in ipci.c, similar to all the *ShmemSize() and *ShmemInit() functions
* today. In an extension, this would be done in _PG_init() or in
* the shmem_request_hook, replacing the RequestAddinShmemSpace calls
* we have today.
*/
void
FoobarShmemRegister(void)
{
ShmemRegisterStruct(&FoobarShmemDesc);
}

/*
* This callback is called once at postmaster startup, to initialize
* the shared memory struct. FoobarShmemDesc.ptr has already been
* set when this is called.
*/
static void
FoobarShmemInit(void *arg)
{
memset(FoobarCtl, 0, sizeof(FoobarSharedCtlData));
FoobarCtl->field = 123;
}

--------------

The ShmemStructDesc provides room for extending the facility in the
future. For example, you could specify alignment there, or an additional
"attach" callback when you need to do more per-backend initialization in
EXEC_BACKEND mode. And with the resizeable shared memory, a max size.

** unquote **

0002 allows pointers of the global variables pointing to the shared
memory structure to be specified in ShmemStructDesc for easier use.
This should be merged into 0001.

0003 allows resizable shared memory structures to be specified via
ShmemRegisterStruct() and then implements allocating shared memory
segments for them and allocating the structures themselves. It also
implements the ShmemResizeRegistered() API to resize registered
resizable structures. The resizable shared memory structures are
placed in their own shared memory segments which are implemented using
the same method as 0002 patch in [2]. It is also PoC, "Do not do not
look too closely". The pieces dealing with huge pages need some
rework. Portability is another issue. Most important is what method
should be used to implement resizable shared memory itself. More on
that later.

0003 adds APIs to register, allocate and resize shared memory
structures in shmem.c extending the infrastructure added by 0001. The
patch also has a test which demonstrates how to use those APIs. If we
think those APIs look good, we can work on finishing 0001 and then I
can work on completing 0003.

Thoughts?

I am copying the discussion about supporting resizable shared memory
from shared buffers resizing thread here, since those apply to 0003.
Andres is suggesting an alternate approach [3] to support resizable
shared memory. I am continuing that conversation here.

> I think the multiple memory mappings approach is just too restrictive. If we
> e.g. eventually want to make some of the other major allocations that depend
> on NBuffers react to resizing shared buffers, it's very easy to do if all it
> requires is calling
>    madvise(TYPEALIGN(start, page_size), MADV_REMOVE, TYPEALIGN_DOWN(end, page_size));

You mean madvise(TYPEALIGN(start, page_size), TYPEALIGN_DOWN(end,
page_size) - TYPEALIGN(start, page_size), MADV_REMOVE)? Right?

`man madvise` has this
       MADV_REMOVE (since Linux 2.6.16)
              Free  up a given range of pages and its associated
backing store.  This is equivalent to punching a
              hole in the corresponding byte range of the backing
store (see fallocate(2)).  Subsequent  accesses
              in the specified address range will see bytes containing zero.

              The  specified  address  range  must be mapped shared
and writable.  This flag cannot be applied to
              locked pages, Huge TLB pages, or VM_PFNMAP pages.

              In the initial implementation, only tmpfs(5) was
supported MADV_REMOVE; but since  Linux  3.5,  any
              filesystem  which  supports  the  fallocate(2)
FALLOC_FL_PUNCH_HOLE mode also supports MADV_REMOVE.
              Hugetlbfs fails with the error EINVAL and other
filesystems fail with the error EOPNOTSUPP.

It says the flag can not be applied to Huge TLB pages. We won't be
able to make resizable shared memory structures allocated with huge
pages. That seems like a serious restriction.

I may be misunderstanding something, but it seems like this is useful
to free already allocated memory, not necessarily allocate more
memory. I don't understand how a user would start with a larger
reserved address space with only small portions of that space being
backed by memory.

>
> There are several cases that are pretty easy to handle that way:
> - Buffer Blocks
> - Buffer Descriptors
> - Sync request queue (part of the "Checkpointer Data" allocation)
> - Checkpoint BufferIds (for sorting the to-be-checkpointed data)
> - Buffer IO Condition Variables
>
> But if you want to support making these resizable with the separate mappings
> approach, it gets considerably more complicated and the number of mappings
> increases more substantially.
>
> We also don't need a lot less infrastructure in shmem.c that way. We could
> e.g. make ShmemInitStruct() reservere the entire requested size (to avoid OOM
> killer issues) and have a ShmemInitStructExt() that allows the caller choose
> whether to reserve. No different segment IDs etc are needed.

I agree that if we can devise a mechanism to allocate a single mapping
with holes placed around resizable structure, we could use it for
shared memory structures other than buffer pool as well. However, as
far as I can understand we will still need the concept of segments
inside shmem.c (not necessarily in pg_shmem.h) to track the
allocations for each of the individual structures OR may be we could
use the resizable shmem structure itself to track it.

[1] https://www.postgresql.org/message-id/91265854-b3ba-45c6-aa44-7e8dcdd51470%40iki.fi
[2] https://www.postgresql.org/message-id/CAExHW5tSw8r06RLAArvf923cO4NGetitPhQ7AO0o7hsKx8jsNw%40mail.gmail.com
[3] https://www.postgresql.org/message-id/aY4v1oSmokXNpQMX%40alap3.anarazel.de

-- 
Best Wishes,
Ashutosh Bapat

  2. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-02-13T12:03:12Z

    On 13/02/2026 13:47, Ashutosh Bapat wrote:
> `man madvise` has this
>         MADV_REMOVE (since Linux 2.6.16)
>                Free  up a given range of pages and its associated
> backing store.  This is equivalent to punching a
>                hole in the corresponding byte range of the backing
> store (see fallocate(2)).  Subsequent  accesses
>                in the specified address range will see bytes containing zero.
> 
>                The  specified  address  range  must be mapped shared
> and writable.  This flag cannot be applied to
>                locked pages, Huge TLB pages, or VM_PFNMAP pages.
> 
>                In the initial implementation, only tmpfs(5) was
> supported MADV_REMOVE; but since  Linux  3.5,  any
>                filesystem  which  supports  the  fallocate(2)
> FALLOC_FL_PUNCH_HOLE mode also supports MADV_REMOVE.
>                Hugetlbfs fails with the error EINVAL and other
> filesystems fail with the error EOPNOTSUPP.
> 
> It says the flag can not be applied to Huge TLB pages. We won't be
> able to make resizable shared memory structures allocated with huge
> pages. That seems like a serious restriction.

Per https://man7.org/linux/man-pages/man2/madvise.2.html:

MADV_REMOVE (since Linux 2.6.16)
               ...

               Support for the Huge TLB filesystem was added in Linux
               v4.3.

> I may be misunderstanding something, but it seems like this is useful
> to free already allocated memory, not necessarily allocate more
> memory. I don't understand how a user would start with a larger
> reserved address space with only small portions of that space being
> backed by memory.

Hmm, I guess you'll need to use MAP_NORESERVE in the first mmap() call. 
to reserve address space for the maximum size, and then 
madvise(MADV_POPULATE_WRITE) using the initial size. Later, 
madvise(MADV_REMOVE) to shrink, and madvise(MADV_POPULATE_WRITE) to grow 
again.

- Heikki

  3. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-02-16T14:52:51Z

    On Fri, Feb 13, 2026 at 5:33 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 13/02/2026 13:47, Ashutosh Bapat wrote:
> > `man madvise` has this
> >         MADV_REMOVE (since Linux 2.6.16)
> >                Free  up a given range of pages and its associated
> > backing store.  This is equivalent to punching a
> >                hole in the corresponding byte range of the backing
> > store (see fallocate(2)).  Subsequent  accesses
> >                in the specified address range will see bytes containing zero.
> >
> >                The  specified  address  range  must be mapped shared
> > and writable.  This flag cannot be applied to
> >                locked pages, Huge TLB pages, or VM_PFNMAP pages.
> >
> >                In the initial implementation, only tmpfs(5) was
> > supported MADV_REMOVE; but since  Linux  3.5,  any
> >                filesystem  which  supports  the  fallocate(2)
> > FALLOC_FL_PUNCH_HOLE mode also supports MADV_REMOVE.
> >                Hugetlbfs fails with the error EINVAL and other
> > filesystems fail with the error EOPNOTSUPP.
> >
> > It says the flag can not be applied to Huge TLB pages. We won't be
> > able to make resizable shared memory structures allocated with huge
> > pages. That seems like a serious restriction.
>
> Per https://man7.org/linux/man-pages/man2/madvise.2.html:
>
> MADV_REMOVE (since Linux 2.6.16)
>                ...
>
>                Support for the Huge TLB filesystem was added in Linux
>                v4.3.
>
> > I may be misunderstanding something, but it seems like this is useful
> > to free already allocated memory, not necessarily allocate more
> > memory. I don't understand how a user would start with a larger
> > reserved address space with only small portions of that space being
> > backed by memory.
>
> Hmm, I guess you'll need to use MAP_NORESERVE in the first mmap() call.
> to reserve address space for the maximum size, and then
> madvise(MADV_POPULATE_WRITE) using the initial size. Later,
> madvise(MADV_REMOVE) to shrink, and madvise(MADV_POPULATE_WRITE) to grow
> again.

Thank you for the hint. Also thanks to Andres's idea, the resizable
structure patch is quite small now. Actually, after experimenting with
madvise, memfd_create and ftruncate(), I see that MADV_POPULATE_WRITE
is not required at all. We don't have to do anything to expand a
structure. Memory will be allocated as and when the program writes to
it. I also discovered things that I didn't know about.
1. ftruncate() sets the size of the file but it doesn't allocate the
memory pages.
2. to use madvise() the address needs to be backed by a file, so
memfd_create is a must.
3. We can't write to a file backed memory at a location beyond the
size of the file. Hence we have to set the size of the file to the
maximum size at the beginning.
4. the address and length passed to madvise needs to be page aligned,
but that passed to fallocate() needn't be. `man fallocate` says
"Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux
2.6.38) in mode deallocates space (i.e., creates a hole) in the byte
range starting at offset and continuing for len bytes. Within the
specified range, partial filesystem blocks are zeroed, and whole
filesystem blocks are removed from the file.". It seems to be
automatically taking care of the page size. So using fallocate()
simplifies logic. Further `man madvise` says "but since Linux 3.5, any
filesystem which supports the fallocate(2) FALLOC_FL_PUNCH_HOLE mode
also supports MADV_REMOVE." fallocate with FALLOC_FL_PUNCH_HOLE is
guaranteed to be available on a system which supports MADV_REMOVE.

Using fallocate() (or madvise()) to free memory, we don't need
multiple segments. So much less code churn compared to the multiple
mappings approach. However, there is one drawback. In the multiple
mapping approach access beyond the current size of the structure would
result in segfault or bus error. But in the fallocate/madvise approach
such an access does not cause a crash. A write beyond the pages that
fit the current size of the structure causes more memory to be
allocated silently. A read returns 0s. So, there's a possibility that
bugs in size calculations might go unnoticed. I think that's how it
works even today, access in the yet un-allocated part of the shared
memory will simply go unnoticed.

PFA the patches with 0003 implementing resizable structures using
fallocate(). There are TODOs, and also I need to make sure that
resizable structures are disabled where memfd_create(), fallocate()
and anonymous memory mappings are not available. Also the test is
unstable since it prints the memory consumption numbers obtained from
/proc/self/status. But it demonstrates that allocation and freeing of
shared memory as the shared structures undergo resizing. I don't think
there is a stable way to use the numbers though; so we might have to
remove those ultimately.


-- 
Best Wishes,
Ashutosh Bapat

  4. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-02-16T17:32:18Z

    On 16/02/2026 16:52, Ashutosh Bapat wrote:
> 2. to use madvise() the address needs to be backed by a file, so
> memfd_create is a must.

It seems to work fine for anonymous mmapped memory here. See attached 
test program.

- Heikki

  5. Re: Better shared data structure management and resizable shared data structures

    Andres Freund <andres@anarazel.de> — 2026-02-16T17:56:03Z

    Hi,

On 2026-02-16 20:22:51 +0530, Ashutosh Bapat wrote:
> On Fri, Feb 13, 2026 at 5:33 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >
> > On 13/02/2026 13:47, Ashutosh Bapat wrote:
> > > `man madvise` has this
> > >         MADV_REMOVE (since Linux 2.6.16)
> > >                Free  up a given range of pages and its associated
> > > backing store.  This is equivalent to punching a
> > >                hole in the corresponding byte range of the backing
> > > store (see fallocate(2)).  Subsequent  accesses
> > >                in the specified address range will see bytes containing zero.
> > >
> > >                The  specified  address  range  must be mapped shared
> > > and writable.  This flag cannot be applied to
> > >                locked pages, Huge TLB pages, or VM_PFNMAP pages.
> > >
> > >                In the initial implementation, only tmpfs(5) was
> > > supported MADV_REMOVE; but since  Linux  3.5,  any
> > >                filesystem  which  supports  the  fallocate(2)
> > > FALLOC_FL_PUNCH_HOLE mode also supports MADV_REMOVE.
> > >                Hugetlbfs fails with the error EINVAL and other
> > > filesystems fail with the error EOPNOTSUPP.
> > >
> > > It says the flag can not be applied to Huge TLB pages. We won't be
> > > able to make resizable shared memory structures allocated with huge
> > > pages. That seems like a serious restriction.
> >
> > Per https://man7.org/linux/man-pages/man2/madvise.2.html:
> >
> > MADV_REMOVE (since Linux 2.6.16)
> >                ...
> >
> >                Support for the Huge TLB filesystem was added in Linux
> >                v4.3.
> >
> > > I may be misunderstanding something, but it seems like this is useful
> > > to free already allocated memory, not necessarily allocate more
> > > memory. I don't understand how a user would start with a larger
> > > reserved address space with only small portions of that space being
> > > backed by memory.
> >
> > Hmm, I guess you'll need to use MAP_NORESERVE in the first mmap() call.
> > to reserve address space for the maximum size, and then
> > madvise(MADV_POPULATE_WRITE) using the initial size. Later,
> > madvise(MADV_REMOVE) to shrink, and madvise(MADV_POPULATE_WRITE) to grow
> > again.
> 
> Thank you for the hint. Also thanks to Andres's idea, the resizable
> structure patch is quite small now. Actually, after experimenting with
> madvise, memfd_create and ftruncate(), I see that MADV_POPULATE_WRITE
> is not required at all. We don't have to do anything to expand a
> structure. Memory will be allocated as and when the program writes to
> it.

I think we *do* want the MADV_POPULATE_WRITE, at least when using huge pages,
because otherwise you'll get a SIGBUS when accessing the memory if there is no
huge page available anymore.


> I also discovered things that I didn't know about.
> 1. ftruncate() sets the size of the file but it doesn't allocate the
> memory pages.

Right.


> 2. to use madvise() the address needs to be backed by a file, so
> memfd_create is a must.

I am quite sure that that is not true.  I hacked this up with today's
postgres, and the madvise works with the mmap() backed allocation from
sysv_shmem.c, which is anonymous.

What made you conclude that that is the case?


> 4. the address and length passed to madvise needs to be page aligned,
> but that passed to fallocate() needn't be. `man fallocate` says
> "Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux
> 2.6.38) in mode deallocates space (i.e., creates a hole) in the byte
> range starting at offset and continuing for len bytes. Within the
> specified range, partial filesystem blocks are zeroed, and whole
> filesystem blocks are removed from the file.". It seems to be
> automatically taking care of the page size. So using fallocate()
> simplifies logic. Further `man madvise` says "but since Linux 3.5, any
> filesystem which supports the fallocate(2) FALLOC_FL_PUNCH_HOLE mode
> also supports MADV_REMOVE." fallocate with FALLOC_FL_PUNCH_HOLE is
> guaranteed to be available on a system which supports MADV_REMOVE.

I think it makes no sense to support resizing below page size
granularity. What's the point of doing that?


> Using fallocate() (or madvise()) to free memory, we don't need
> multiple segments. So much less code churn compared to the multiple
> mappings approach. However, there is one drawback. In the multiple
> mapping approach access beyond the current size of the structure would
> result in segfault or bus error. But in the fallocate/madvise approach
> such an access does not cause a crash. A write beyond the pages that
> fit the current size of the structure causes more memory to be
> allocated silently. A read returns 0s. So, there's a possibility that
> bugs in size calculations might go unnoticed. I think that's how it
> works even today, access in the yet un-allocated part of the shared
> memory will simply go unnoticed.

If that's something you care about, you can mprotect(PROT_NONE) the relevant
regions.

Greetings,

Andres Freund

  6. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-02-17T11:36:24Z

    On Mon, Feb 16, 2026 at 11:26 PM Andres Freund <andres@anarazel.de> wrote:
>
> I think we *do* want the MADV_POPULATE_WRITE, at least when using huge pages,
> because otherwise you'll get a SIGBUS when accessing the memory if there is no
> huge page available anymore.
>

Ok.

Jakub's experiments [1] showed that fallocate()ing shared memory would
slow down postmaster start on a slow machine. I suppose the same thing
applies to MADV_POPULATE_WRITE. And we don't do that today even in the
case of huge pages; so we already have that problem.

If we perform MADV_POPULATE_WRITE, do we want it only for resizable
shared memory structures or all the structures in the shared memory?

On Mon, Feb 16, 2026 at 11:02 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 16/02/2026 16:52, Ashutosh Bapat wrote:
> > 2. to use madvise() the address needs to be backed by a file, so
> > memfd_create is a must.
>
> It seems to work fine for anonymous mmapped memory here. See attached
> test program.
On Mon, Feb 16, 2026 at 11:26 PM Andres Freund <andres@anarazel.de> wrote:
> > 2. to use madvise() the address needs to be backed by a file, so
> > memfd_create is a must.
>
> I am quite sure that that is not true.  I hacked this up with today's
> postgres, and the madvise works with the mmap() backed allocation from
> sysv_shmem.c, which is anonymous.
>
> What made you conclude that that is the case?
>

You are right. I was misled by the following sentence in the `man
madvise`: "but since Linux 3.5, any filesystem which supports the
fallocate(2) FALLOC_FL_PUNCH_HOLE mode also supports MADV_REMOVE.
Filesystems which do not support MADV_REMOVE fail with the error
EOPNOTSUPP." And in a subsequent experiment I dropped MAP_ANONYMOUS
from mmap() and used madvise() which didn't work obviously. My bad.

In the attached patches, I have got rid of memfd_create. That simplifies code.


>
> > 4. the address and length passed to madvise needs to be page aligned,
> > but that passed to fallocate() needn't be. `man fallocate` says
> > "Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux
> > 2.6.38) in mode deallocates space (i.e., creates a hole) in the byte
> > range starting at offset and continuing for len bytes. Within the
> > specified range, partial filesystem blocks are zeroed, and whole
> > filesystem blocks are removed from the file.". It seems to be
> > automatically taking care of the page size. So using fallocate()
> > simplifies logic. Further `man madvise` says "but since Linux 3.5, any
> > filesystem which supports the fallocate(2) FALLOC_FL_PUNCH_HOLE mode
> > also supports MADV_REMOVE." fallocate with FALLOC_FL_PUNCH_HOLE is
> > guaranteed to be available on a system which supports MADV_REMOVE.
>
> I think it makes no sense to support resizing below page size
> granularity. What's the point of doing that?
>

No point really. But we can not control the extensions which want to
specify a maximum size smaller than a page size. They wouldn't know
what page size the underlying machine will have, especially with huge
pages which have a wide range of sizes. Even in the case of shared
buffers, a value of max_shared_buffers may cause buffer blocks to span
pages but other structures may fit a page.

In the attached patches, if a resizable structure is such that its
max_size is smaller than a page size, it is treated as a fixed
structure with size = max_size. Any request to resize such structures
will simply update the metadata without actual madvise operation. Only
the structures whose max_size > page_size would be treated as truly
resizable and will use madvise. You bring another interesting point.
If a resizable structure has a maximum size higher than the page size,
but it is allocated such that the initial part of it is on a partially
allocated page and the last part of it is on another partially
allocated page, those pages are never freed because of adjoining
structures. Per the logic in the attached patches, all the fixed (or
pseudo-resizable structures) are packed together. The resizable
structures start on a page boundary and their max_sizes are adjusted
to be page aligned. That way we can release pages when the structure
shrinks more than a page.

>
> > Using fallocate() (or madvise()) to free memory, we don't need
> > multiple segments. So much less code churn compared to the multiple
> > mappings approach. However, there is one drawback. In the multiple
> > mapping approach access beyond the current size of the structure would
> > result in segfault or bus error. But in the fallocate/madvise approach
> > such an access does not cause a crash. A write beyond the pages that
> > fit the current size of the structure causes more memory to be
> > allocated silently. A read returns 0s. So, there's a possibility that
> > bugs in size calculations might go unnoticed. I think that's how it
> > works even today, access in the yet un-allocated part of the shared
> > memory will simply go unnoticed.
>
> If that's something you care about, you can mprotect(PROT_NONE) the relevant
> regions.

I am fine, if we let go of this protection while getting rid of
multiple segments, if we all agree to do so.

I could be wrong, but mprotect needs to be executed in every backend
where the memory is mapped and then a new backend needs to inherit it
from the postmaster. Makes resizing complex since it has to touch
every backend. So avoiding mprotect is better.

[1] https://www.postgresql.org/message-id/CAKZiRmwxVqEbp7JgOed%3DBCT6cq8RNuHk3N0vuwro65Tsw9E8NA%40mail.gmail.com

PFA patches.

-- 
Best Wishes,
Ashutosh Bapat

  7. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-02-18T15:47:07Z

    On Tue, Feb 17, 2026 at 5:06 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Mon, Feb 16, 2026 at 11:26 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > I think we *do* want the MADV_POPULATE_WRITE, at least when using huge pages,
> > because otherwise you'll get a SIGBUS when accessing the memory if there is no
> > huge page available anymore.
> >
>
> Ok.
>
> Jakub's experiments [1] showed that fallocate()ing shared memory would
> slow down postmaster start on a slow machine. I suppose the same thing
> applies to MADV_POPULATE_WRITE. And we don't do that today even in the
> case of huge pages; so we already have that problem.
>
> If we perform MADV_POPULATE_WRITE, do we want it only for resizable
> shared memory structures or all the structures in the shared memory?

In the attached patches, I have used MADV_POPULATE_WRITE during
resizing, which is run time operation. When the structures are
allocated when server starts, they are usually initialised, so we end
up allocating memory for the same. So we don't need
MADV_POPULATE_WRITE at that time, and thus avoid affecting startup
slowness, if any. Buffer blocks are not initialised at the time of
starting the server, so their memory is allocated as they are
accessed. But that's how it works today, so no change there.

>
>
> >
> > > 4. the address and length passed to madvise needs to be page aligned,
> > > but that passed to fallocate() needn't be. `man fallocate` says
> > > "Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux
> > > 2.6.38) in mode deallocates space (i.e., creates a hole) in the byte
> > > range starting at offset and continuing for len bytes. Within the
> > > specified range, partial filesystem blocks are zeroed, and whole
> > > filesystem blocks are removed from the file.". It seems to be
> > > automatically taking care of the page size. So using fallocate()
> > > simplifies logic. Further `man madvise` says "but since Linux 3.5, any
> > > filesystem which supports the fallocate(2) FALLOC_FL_PUNCH_HOLE mode
> > > also supports MADV_REMOVE." fallocate with FALLOC_FL_PUNCH_HOLE is
> > > guaranteed to be available on a system which supports MADV_REMOVE.
> >
> > I think it makes no sense to support resizing below page size
> > granularity. What's the point of doing that?
> >
>
> No point really. But we can not control the extensions which want to
> specify a maximum size smaller than a page size. They wouldn't know
> what page size the underlying machine will have, especially with huge
> pages which have a wide range of sizes. Even in the case of shared
> buffers, a value of max_shared_buffers may cause buffer blocks to span
> pages but other structures may fit a page.
>
> In the attached patches, if a resizable structure is such that its
> max_size is smaller than a page size, it is treated as a fixed
> structure with size = max_size. Any request to resize such structures
> will simply update the metadata without actual madvise operation. Only
> the structures whose max_size > page_size would be treated as truly
> resizable and will use madvise. You bring another interesting point.
> If a resizable structure has a maximum size higher than the page size,
> but it is allocated such that the initial part of it is on a partially
> allocated page and the last part of it is on another partially
> allocated page, those pages are never freed because of adjoining
> structures. Per the logic in the attached patches, all the fixed (or
> pseudo-resizable structures) are packed together. The resizable
> structures start on a page boundary and their max_sizes are adjusted
> to be page aligned. That way we can release pages when the structure
> shrinks more than a page.
>

> >
> > > Using fallocate() (or madvise()) to free memory, we don't need
> > > multiple segments. So much less code churn compared to the multiple
> > > mappings approach. However, there is one drawback. In the multiple
> > > mapping approach access beyond the current size of the structure would
> > > result in segfault or bus error. But in the fallocate/madvise approach
> > > such an access does not cause a crash. A write beyond the pages that
> > > fit the current size of the structure causes more memory to be
> > > allocated silently. A read returns 0s. So, there's a possibility that
> > > bugs in size calculations might go unnoticed. I think that's how it
> > > works even today, access in the yet un-allocated part of the shared
> > > memory will simply go unnoticed.
> >
> > If that's something you care about, you can mprotect(PROT_NONE) the relevant
> > regions.
>
> I am fine, if we let go of this protection while getting rid of
> multiple segments, if we all agree to do so.
>
> I could be wrong, but mprotect needs to be executed in every backend
> where the memory is mapped and then a new backend needs to inherit it
> from the postmaster. Makes resizing complex since it has to touch
> every backend. So avoiding mprotect is better.
>

If the general approach in the attached patches looks good, we can
work on improving the 0001 + 0002 to be committable and then work on
0003.

-- 
Best Wishes,
Ashutosh Bapat

  8. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-02-18T15:50:59Z

    On Wed, Feb 18, 2026 at 9:17 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Feb 17, 2026 at 5:06 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Mon, Feb 16, 2026 at 11:26 PM Andres Freund <andres@anarazel.de> wrote:
> > >
> > > I think we *do* want the MADV_POPULATE_WRITE, at least when using huge pages,
> > > because otherwise you'll get a SIGBUS when accessing the memory if there is no
> > > huge page available anymore.
> > >
> >
> > Ok.
> >
> > Jakub's experiments [1] showed that fallocate()ing shared memory would
> > slow down postmaster start on a slow machine. I suppose the same thing
> > applies to MADV_POPULATE_WRITE. And we don't do that today even in the
> > case of huge pages; so we already have that problem.
> >
> > If we perform MADV_POPULATE_WRITE, do we want it only for resizable
> > shared memory structures or all the structures in the shared memory?
>
> In the attached patches, I have used MADV_POPULATE_WRITE during
> resizing, which is run time operation. When the structures are
> allocated when server starts, they are usually initialised, so we end
> up allocating memory for the same. So we don't need
> MADV_POPULATE_WRITE at that time, and thus avoid affecting startup
> slowness, if any. Buffer blocks are not initialised at the time of
> starting the server, so their memory is allocated as they are
> accessed. But that's how it works today, so no change there.
>
> >
> >
> > >
> > > > 4. the address and length passed to madvise needs to be page aligned,
> > > > but that passed to fallocate() needn't be. `man fallocate` says
> > > > "Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux
> > > > 2.6.38) in mode deallocates space (i.e., creates a hole) in the byte
> > > > range starting at offset and continuing for len bytes. Within the
> > > > specified range, partial filesystem blocks are zeroed, and whole
> > > > filesystem blocks are removed from the file.". It seems to be
> > > > automatically taking care of the page size. So using fallocate()
> > > > simplifies logic. Further `man madvise` says "but since Linux 3.5, any
> > > > filesystem which supports the fallocate(2) FALLOC_FL_PUNCH_HOLE mode
> > > > also supports MADV_REMOVE." fallocate with FALLOC_FL_PUNCH_HOLE is
> > > > guaranteed to be available on a system which supports MADV_REMOVE.
> > >
> > > I think it makes no sense to support resizing below page size
> > > granularity. What's the point of doing that?
> > >
> >
> > No point really. But we can not control the extensions which want to
> > specify a maximum size smaller than a page size. They wouldn't know
> > what page size the underlying machine will have, especially with huge
> > pages which have a wide range of sizes. Even in the case of shared
> > buffers, a value of max_shared_buffers may cause buffer blocks to span
> > pages but other structures may fit a page.
> >
> > In the attached patches, if a resizable structure is such that its
> > max_size is smaller than a page size, it is treated as a fixed
> > structure with size = max_size. Any request to resize such structures
> > will simply update the metadata without actual madvise operation. Only
> > the structures whose max_size > page_size would be treated as truly
> > resizable and will use madvise. You bring another interesting point.
> > If a resizable structure has a maximum size higher than the page size,
> > but it is allocated such that the initial part of it is on a partially
> > allocated page and the last part of it is on another partially
> > allocated page, those pages are never freed because of adjoining
> > structures. Per the logic in the attached patches, all the fixed (or
> > pseudo-resizable structures) are packed together. The resizable
> > structures start on a page boundary and their max_sizes are adjusted
> > to be page aligned. That way we can release pages when the structure
> > shrinks more than a page.
> >
>
> > >
> > > > Using fallocate() (or madvise()) to free memory, we don't need
> > > > multiple segments. So much less code churn compared to the multiple
> > > > mappings approach. However, there is one drawback. In the multiple
> > > > mapping approach access beyond the current size of the structure would
> > > > result in segfault or bus error. But in the fallocate/madvise approach
> > > > such an access does not cause a crash. A write beyond the pages that
> > > > fit the current size of the structure causes more memory to be
> > > > allocated silently. A read returns 0s. So, there's a possibility that
> > > > bugs in size calculations might go unnoticed. I think that's how it
> > > > works even today, access in the yet un-allocated part of the shared
> > > > memory will simply go unnoticed.
> > >
> > > If that's something you care about, you can mprotect(PROT_NONE) the relevant
> > > regions.
> >
> > I am fine, if we let go of this protection while getting rid of
> > multiple segments, if we all agree to do so.
> >
> > I could be wrong, but mprotect needs to be executed in every backend
> > where the memory is mapped and then a new backend needs to inherit it
> > from the postmaster. Makes resizing complex since it has to touch
> > every backend. So avoiding mprotect is better.
> >
>

Sent too soon.

I have also reworked the test into a TAP test which looks stable than
the earlier version. Haven't had any failures on my laptop.

> If the general approach in the attached patches looks good, we can
> work on improving the 0001 + 0002 to be committable and then work on
> 0003.

The resizable memory patch works only in linux where
MADV_POPULATE_WRITE and MADV_REMOVE are supported on anonymous shared
memory. On other platforms and where that support doesn't exist, we
will need to disable the feature for now. That work remains. Also the
TODOs need to be addressed.

-- 
Best Wishes,
Ashutosh Bapat

  9. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-02-23T14:14:23Z

    On Wed, Feb 18, 2026 at 9:17 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
> > > 4. the address and length passed to madvise needs to be page aligned,
> > > but that passed to fallocate() needn't be. `man fallocate` says
> > > "Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux
> > > 2.6.38) in mode deallocates space (i.e., creates a hole) in the byte
> > > range starting at offset and continuing for len bytes. Within the
> > > specified range, partial filesystem blocks are zeroed, and whole
> > > filesystem blocks are removed from the file.". It seems to be
> > > automatically taking care of the page size. So using fallocate()
> > > simplifies logic. Further `man madvise` says "but since Linux 3.5, any
> > > filesystem which supports the fallocate(2) FALLOC_FL_PUNCH_HOLE mode
> > > also supports MADV_REMOVE." fallocate with FALLOC_FL_PUNCH_HOLE is
> > > guaranteed to be available on a system which supports MADV_REMOVE.
> >
> > I think it makes no sense to support resizing below page size
> > granularity. What's the point of doing that?
> >
>
> No point really. But we can not control the extensions which want to
> specify a maximum size smaller than a page size. They wouldn't know
> what page size the underlying machine will have, especially with huge
> pages which have a wide range of sizes. Even in the case of shared
> buffers, a value of max_shared_buffers may cause buffer blocks to span
> pages but other structures may fit a page.
>
> In the attached patches, if a resizable structure is such that its
> max_size is smaller than a page size, it is treated as a fixed
> structure with size = max_size. Any request to resize such structures
> will simply update the metadata without actual madvise operation. Only
> the structures whose max_size > page_size would be treated as truly
> resizable and will use madvise. You bring another interesting point.
> If a resizable structure has a maximum size higher than the page size,
> but it is allocated such that the initial part of it is on a partially
> allocated page and the last part of it is on another partially
> allocated page, those pages are never freed because of adjoining
> structures. Per the logic in the attached patches, all the fixed (or
> pseudo-resizable structures) are packed together. The resizable
> structures start on a page boundary and their max_sizes are adjusted
> to be page aligned. That way we can release pages when the structure
> shrinks more than a page.

It was a mistake on my part to assume that more memory will be freed
if we page align the start and end of a resizable structure. I didn't
account for the memory wasted in alignment itself. That amount comes
out to be same as the amount of memory wasted if we don't page align
the structure. But the code is simpler if we don't page align the
structure as seen in the attached patches.

> > >
> > > > Using fallocate() (or madvise()) to free memory, we don't need
> > > > multiple segments. So much less code churn compared to the multiple
> > > > mappings approach. However, there is one drawback. In the multiple
> > > > mapping approach access beyond the current size of the structure would
> > > > result in segfault or bus error. But in the fallocate/madvise approach
> > > > such an access does not cause a crash. A write beyond the pages that
> > > > fit the current size of the structure causes more memory to be
> > > > allocated silently. A read returns 0s. So, there's a possibility that
> > > > bugs in size calculations might go unnoticed. I think that's how it
> > > > works even today, access in the yet un-allocated part of the shared
> > > > memory will simply go unnoticed.
> > >
> > > If that's something you care about, you can mprotect(PROT_NONE) the relevant
> > > regions.
> >
> > I am fine, if we let go of this protection while getting rid of
> > multiple segments, if we all agree to do so.
> >
> > I could be wrong, but mprotect needs to be executed in every backend
> > where the memory is mapped and then a new backend needs to inherit it
> > from the postmaster. Makes resizing complex since it has to touch
> > every backend. So avoiding mprotect is better.

I discussed this point with Andres offlist. Here's a summary of that
discussion. Any serious users of resizable shared memory structures
would need to send proc signal barriers to synchronize the resizing
across the backends. This barrier can be used to perform mprotect() in
the backends and a separate signal to Postmaster, if mprotect is
needed in Postmaster. But whether mprotect is needed depends upon the
usecase. It should be responsibility of the resizable structure user
and not of the ShmemResizeRegistered()

Following points need a bit of discussion.

1. calculation of allocated_size
For fixed sized shared memory structures, allocated_size is the size
of the structure after cache aligning it. Assuming that the shared
memory is allcoated in pages, this also is the actually memory
allocated to the structure when the whole structure is written to. For
resizable structure, it's a bit more complicated. We allocate and
reserve the maximum space required by the structure. At a given point
in time, the memory page where the next structure begins and the page
which contains the end of the structure at that point in time are
allocated. The pages in-between are not allocated. Thus the
allocated_size should be the length from the start the structure to
the end of the page containing the current end of the structure + part
of the page where the next structure starts upto the start of the next
structure. That is what is implemented in the attached patches.

2. GUCs shared_memory_size, shared_memory_size_in_huge_pages
These GUCs indicate the size of the shared memory in bytes and in huge
pages. Without resizable shared memory structures calculating these is
straight forward, we sum all the sizes of all the requested
structures. With resizable shared memory structures, these GUCs do not
make much sense. Since the memory allocated to the resizable
structures can be anywhere between 0 to maximum, neither the sum of
the their initial sizes nor the sum of their maximum sizes can be
reported as shared_memory_size. Similarly for
shared_memory_size_in_huge_pages. We need two GUCs to replace each of
the existing GUCs - max_shared_memory_size, initial_shared_memory_size
and their huge page peers. max_shared_memory_size is the sum of the
maximum sizes of resizable structures + the requested sizes of the
fixed structure. initial_shared_memory_size is the sum of the initial
sizes requested for all the structures.

3. Testing the memory allocation
I couldn't find a way to reliably know the shared memory allocated at
a given address in the process. RSS Shmem given the amount of shared
memory accessed by the process which includes memory allocated to the
fixed structures accessed by the process. This value isn't stable
across runs of the test in the patch. The test adds the RSS shmem
reported against the variations in the resizable shared memory
structure which can be visually inspected to be within limits. But
those limits are hard to test in the test code. Looking for some
suggestions here.

Disabling resizable structures in the builds which do not support
resizable structures is still a TODO.

-- 
Best Wishes,
Ashutosh Bapat

  10. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-06T14:12:57Z

    I spent some time cleaning up the new registration machinery. I didn't 
look at the "resizeable" part yet, but it should fit in nicely, as 
Ashutosh demonstrated.

I added a lot of comments, changed the existing docs on how to allocate 
shared memory in extensions to use the machinery, and tons of other 
cleanups. I'm getting pretty happy with this, but there are a couple of 
weak spots:

Firstly, I'm not sure what to do with ShmemRegisterHash() and the 
'HASHCTL *infoP' argument to it. I feel it'd be nicer if the HASHCTL was 
just part of the ShmemHashDesc struct, but I'm not sure if that fits all 
the callers. I'll have to try that out I guess.

Secondly, I'm not 100% happy with the facilities we provide to 
extensions. The lifecycle of _PG_init() and shmem_request/startup_hooks 
is a little messy. The status quo is that a shared library gets control 
in three different places:

1. _PG_init() gets called early at postmaster startup, if the library is 
in shared_preload_libraries. If it's not in shared_preload_libraries, it 
gets called whenever the module is loaded.

2. The library can install a shmem_request_hook, which gets called early 
at postmaster startup, but after initializing the MaxBackends GUC. It 
only gets called when the library is loaded via shared_preload_libraries.

3. The library can install a shmem_startup_hook. It gets called later at 
postmaster startup, after the shared memory segment has been allocated. 
In EXEC_BACKEND mode it also gets called at backend startup. It does not 
get called if the library is not listed in shared_preload_libraries.

None of these is quite the right moment to call the new 
ShmemRegisterStruct() function. _PG_init() is too early if the extension 
needs MaxBackends for sizing the shared memory area. shmem_request_hook 
is otherwise good, but in EXEC_BACKEND mode, the ShmemRegisterStruct() 
function needs to also be called backend startup and shmem_request_hook 
is not called at backend startup. shmem_startup_hook() is too late.

For now, I documented that an extension should call 
ShmemRegisterStruct() from _PG_init(), but may adjust the size in the 
shmem_request_hook, if needed.

Another wrinkle here is that you still need the shmem_request_hook, if 
you want to call RequestNamedLWLockTranche(). It cannot be called from 
_PG_init(). I'm not sure why that is.

So I think that requires a little more refactoring, I think an extension 
should need to use shmem_request/startup_hook with the new APIs anymore. 
It should provide more ergonomic callbacks or other mechanisms to 
accomplish the same things.

- Heikki

  11. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-12T17:03:21Z

    On Fri, Mar 6, 2026 at 9:13 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> 1. _PG_init() gets called early at postmaster startup, if the library is
> in shared_preload_libraries. If it's not in shared_preload_libraries, it
> gets called whenever the module is loaded.
>
> 2. The library can install a shmem_request_hook, which gets called early
> at postmaster startup, but after initializing the MaxBackends GUC. It
> only gets called when the library is loaded via shared_preload_libraries.
>
> 3. The library can install a shmem_startup_hook. It gets called later at
> postmaster startup, after the shared memory segment has been allocated.
> In EXEC_BACKEND mode it also gets called at backend startup. It does not
> get called if the library is not listed in shared_preload_libraries.
>
> None of these is quite the right moment to call the new
> ShmemRegisterStruct() function. _PG_init() is too early if the extension
> needs MaxBackends for sizing the shared memory area. shmem_request_hook
> is otherwise good, but in EXEC_BACKEND mode, the ShmemRegisterStruct()
> function needs to also be called backend startup and shmem_request_hook
> is not called at backend startup. shmem_startup_hook() is too late.

I believe that the design goal of
4f2400cb3f10aa79f99fba680c198237da28dd38 was to make it so that people
who had working extensions already didn't need to change their code,
but those for whom the restrictions of doing things in _PG_init were
annoying would have a workable alternative. I think that's a pretty
good goal, although I don't feel we absolutely have to stick to it. It
could easily be worth breaking that if we get something cool out of
it. But is there a reason we can't make it so that this new mechanism
can be used either from _PG_init() or shmem_startup_hook()? (I assume
there is or you likely would have done it already, but it's not clear
to me what that reason is.)

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  12. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-12T18:41:03Z

    On 12/03/2026 19:03, Robert Haas wrote:
> On Fri, Mar 6, 2026 at 9:13 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> 1. _PG_init() gets called early at postmaster startup, if the library is
>> in shared_preload_libraries. If it's not in shared_preload_libraries, it
>> gets called whenever the module is loaded.
>>
>> 2. The library can install a shmem_request_hook, which gets called early
>> at postmaster startup, but after initializing the MaxBackends GUC. It
>> only gets called when the library is loaded via shared_preload_libraries.
>>
>> 3. The library can install a shmem_startup_hook. It gets called later at
>> postmaster startup, after the shared memory segment has been allocated.
>> In EXEC_BACKEND mode it also gets called at backend startup. It does not
>> get called if the library is not listed in shared_preload_libraries.
>>
>> None of these is quite the right moment to call the new
>> ShmemRegisterStruct() function. _PG_init() is too early if the extension
>> needs MaxBackends for sizing the shared memory area. shmem_request_hook
>> is otherwise good, but in EXEC_BACKEND mode, the ShmemRegisterStruct()
>> function needs to also be called backend startup and shmem_request_hook
>> is not called at backend startup. shmem_startup_hook() is too late.
> 
> I believe that the design goal of
> 4f2400cb3f10aa79f99fba680c198237da28dd38 was to make it so that people
> who had working extensions already didn't need to change their code,
> but those for whom the restrictions of doing things in _PG_init were
> annoying would have a workable alternative. I think that's a pretty
> good goal, although I don't feel we absolutely have to stick to it. It
> could easily be worth breaking that if we get something cool out of
> it. But is there a reason we can't make it so that this new mechanism
> can be used either from _PG_init() or shmem_startup_hook()? (I assume
> there is or you likely would have done it already, but it's not clear
> to me what that reason is.)

shmem_startup_hook() is too late. The shmem structs need to be 
registered at postmaster startup before the shmem segment is allocated, 
so that we can calculate the total size needed.

I'm currently leaning towards _PG_init(), except for allocations that 
depend on MaxBackends. For those, you can install a shmem_request_hook 
that sets the size in the descriptor. In other words, you can leave the 
'size' as empty in _PG_init(), but set it later in the shmem_request_hook.

Another option is to add a new bespoken callback in the descriptor for 
such size adjustments, which would get called at the same time as 
shmem_request_hook. That might be a little more ergonomic, there would 
no longer be any need for extensions to use the old 
shmem_request/startup_hooks with the new ShmemRegisterStruct() mechanism.

Except that you'd still need them for RequestNamedLWLockTranche(). I 
wonder if we should recommend extensions to embed the LWLock struct into 
their shared memory struct and use the LWLockInitialize() and 
LWLockNewTrancheId() functions instead. That fits the new 
ShmemRegisterStruct() API a little better than RequestNamedLWLockTranche().

- Heikki

  13. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-12T18:56:18Z

    On Thu, Mar 12, 2026 at 2:41 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> shmem_startup_hook() is too late. The shmem structs need to be
> registered at postmaster startup before the shmem segment is allocated,
> so that we can calculate the total size needed.

Sorry, I meant shmem_request_hook.

> I'm currently leaning towards _PG_init(), except for allocations that
> depend on MaxBackends. For those, you can install a shmem_request_hook
> that sets the size in the descriptor. In other words, you can leave the
> 'size' as empty in _PG_init(), but set it later in the shmem_request_hook.

Why can't you just do the whole thing later?

> Another option is to add a new bespoken callback in the descriptor for
> such size adjustments, which would get called at the same time as
> shmem_request_hook. That might be a little more ergonomic, there would
> no longer be any need for extensions to use the old
> shmem_request/startup_hooks with the new ShmemRegisterStruct() mechanism.

Yeah, worth considering.

> Except that you'd still need them for RequestNamedLWLockTranche(). I
> wonder if we should recommend extensions to embed the LWLock struct into
> their shared memory struct and use the LWLockInitialize() and
> LWLockNewTrancheId() functions instead. That fits the new
> ShmemRegisterStruct() API a little better than RequestNamedLWLockTranche().

Yeah, I think RequestNamedLWLockTranche() might be fine if you just
need LWLocks, but if you need a bunch of resources, putting them all
into the same chunk of memory seems cleaner.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  14. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-12T19:21:19Z

    On 12/03/2026 20:56, Robert Haas wrote:
> On Thu, Mar 12, 2026 at 2:41 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> shmem_startup_hook() is too late. The shmem structs need to be
>> registered at postmaster startup before the shmem segment is allocated,
>> so that we can calculate the total size needed.
> 
> Sorry, I meant shmem_request_hook.

Ah ok

>> I'm currently leaning towards _PG_init(), except for allocations that
>> depend on MaxBackends. For those, you can install a shmem_request_hook
>> that sets the size in the descriptor. In other words, you can leave the
>> 'size' as empty in _PG_init(), but set it later in the shmem_request_hook.
> 
> Why can't you just do the whole thing later?

shmem_request_hook won't work in EXEC_BACKEND mode, because in 
EXEC_BACKEND mode, ShmemRegisterStruct() also needs to be called at 
backend startup.

One of my design goals is to avoid EXEC_BACKEND specific steps so that 
if you write your extension oblivious to EXEC_BACKEND mode, it will 
still usually work with EXEC_BACKEND. For example, if it was necessary 
to call a separate AttachShmem() function for every shmem struct in 
EXEC_BACKEND mode, but which was not needed on Unix, that would be bad.

>> Except that you'd still need them for RequestNamedLWLockTranche(). I
>> wonder if we should recommend extensions to embed the LWLock struct into
>> their shared memory struct and use the LWLockInitialize() and
>> LWLockNewTrancheId() functions instead. That fits the new
>> ShmemRegisterStruct() API a little better than RequestNamedLWLockTranche().
> 
> Yeah, I think RequestNamedLWLockTranche() might be fine if you just
> need LWLocks, but if you need a bunch of resources, putting them all
> into the same chunk of memory seems cleaner.

Agreed. Then again, how often do you need just a LWLock (or multiple 
LWLocks)? Surely you have a struct you want to protect with the lock. I 
guess having shmem hash table but no struct would be pretty common, though.

- Heikki

  15. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-12T20:05:32Z

    On Thu, Mar 12, 2026 at 3:21 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >> I'm currently leaning towards _PG_init(), except for allocations that
> >> depend on MaxBackends. For those, you can install a shmem_request_hook
> >> that sets the size in the descriptor. In other words, you can leave the
> >> 'size' as empty in _PG_init(), but set it later in the shmem_request_hook.
> >
> > Why can't you just do the whole thing later?
>
> shmem_request_hook won't work in EXEC_BACKEND mode, because in
> EXEC_BACKEND mode, ShmemRegisterStruct() also needs to be called at
> backend startup.
>
> One of my design goals is to avoid EXEC_BACKEND specific steps so that
> if you write your extension oblivious to EXEC_BACKEND mode, it will
> still usually work with EXEC_BACKEND. For example, if it was necessary
> to call a separate AttachShmem() function for every shmem struct in
> EXEC_BACKEND mode, but which was not needed on Unix, that would be bad.

That's *definitely* a good goal. A less important but still valuable
goal is to maximize the notational simplicity of the mechanism. Your
callback idea is elegant in theory but in practice it seems like it
might make it harder for people to get started quickly on a new
module, and having to create the object in one place and then fill in
the size in another sort of has the same problem. I don't really know
what to do about that, but it's something to think about. The
complexity of getting the details right is annoyingly high in this
area.

> > Yeah, I think RequestNamedLWLockTranche() might be fine if you just
> > need LWLocks, but if you need a bunch of resources, putting them all
> > into the same chunk of memory seems cleaner.
>
> Agreed. Then again, how often do you need just a LWLock (or multiple
> LWLocks)? Surely you have a struct you want to protect with the lock. I
> guess having shmem hash table but no struct would be pretty common, though.

Yeah, we've developed an annoying number of different ways to do this
stuff. I don't entirely know how to fix that.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  16. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-13T11:41:33Z

    On 12/03/2026 22:05, Robert Haas wrote:
> On Thu, Mar 12, 2026 at 3:21 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>>>> I'm currently leaning towards _PG_init(), except for allocations that
>>>> depend on MaxBackends. For those, you can install a shmem_request_hook
>>>> that sets the size in the descriptor. In other words, you can leave the
>>>> 'size' as empty in _PG_init(), but set it later in the shmem_request_hook.
>>>
>>> Why can't you just do the whole thing later?
>>
>> shmem_request_hook won't work in EXEC_BACKEND mode, because in
>> EXEC_BACKEND mode, ShmemRegisterStruct() also needs to be called at
>> backend startup.
>>
>> One of my design goals is to avoid EXEC_BACKEND specific steps so that
>> if you write your extension oblivious to EXEC_BACKEND mode, it will
>> still usually work with EXEC_BACKEND. For example, if it was necessary
>> to call a separate AttachShmem() function for every shmem struct in
>> EXEC_BACKEND mode, but which was not needed on Unix, that would be bad.
> 
> That's *definitely* a good goal. A less important but still valuable
> goal is to maximize the notational simplicity of the mechanism. Your
> callback idea is elegant in theory but in practice it seems like it
> might make it harder for people to get started quickly on a new
> module, and having to create the object in one place and then fill in
> the size in another sort of has the same problem. I don't really know
> what to do about that, but it's something to think about. The
> complexity of getting the details right is annoyingly high in this
> area.

Yeah. IMHO the existing shmem_request/startup_hook mechanism is pretty 
awkward too, and in most cases, the new mechanism is more convenient. It 
might be slightly less convenient for some things, but mostly it's 
better. Would you agree with that, or do you actually like the old hooks 
and ShmemInitStruct() better?

One such wrinkle with ShmemRegisterStruct() in the patch now is that 
it's harder to do initialization that touches multiple structs or hash 
tables. Currently the callbacks are called in the same order that the 
structs are registered, so you can do all the initialization in the last 
struct's callback. The single pair of shmem_request/startup_hooks per 
module was more clear in that aspect. Fortunately, that kind of 
cross-struct dependencies are pretty rare. So I think it's fine. (The 
order that the callbacks are called needs be documented explicitly though).

If we want to improve on that, one idea would be to introduce a 
ShmemRegisterCallbacks() function to register callbacks that are not 
tied to any particular struct and are called after all the per-struct 
callbacks.

>>> Yeah, I think RequestNamedLWLockTranche() might be fine if you just
>>> need LWLocks, but if you need a bunch of resources, putting them all
>>> into the same chunk of memory seems cleaner.
>>
>> Agreed. Then again, how often do you need just a LWLock (or multiple
>> LWLocks)? Surely you have a struct you want to protect with the lock. I
>> guess having shmem hash table but no struct would be pretty common, though.
> 
> Yeah, we've developed an annoying number of different ways to do this
> stuff. I don't entirely know how to fix that.

Here's a new version that doubles down on the 
LWLockNewTrancheId+LWLockInitialize method, by changing the example in 
the docs, and contrib/pg_stat_statements, to use that method. 
RequestNamedLWLockTranche() still works, there are no changes to it, 
it's just not as convenient to use with ShmemRegisterStruct(). This has 
the advantage that we don't introduce yet another way of allocating LWLocks.

P.S. Thanks for chiming in on this. It's pretty subjective how natural a 
new API like this feels like, so opinions are very welcome.

- Heikki

  17. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-13T21:09:33Z

    On 06/03/2026 16:12, Heikki Linnakangas wrote:
> Firstly, I'm not sure what to do with ShmemRegisterHash() and the 
> 'HASHCTL *infoP' argument to it. I feel it'd be nicer if the HASHCTL was 
> just part of the ShmemHashDesc struct, but I'm not sure if that fits all 
> the callers. I'll have to try that out I guess.
I took a stab at that, and it turned out to be straightforward. I'm not 
sure why I hesitated on that earlier.

Here's a new version with that change, and a ton of little comment 
cleanups and such.

- Heikki

  18. Re: Better shared data structure management and resizable shared data structures

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-13T23:02:05Z

    Hello

 dsm_shmem_init(void)
 {
  size_t size = dsm_estimate_size();
- bool found;

  if (size == 0)
  return;

Isn't there an assignment missing from this function now? Size is
calculated but never used. With the current code if
min_dynamic_shared_memory > 0 the server can crash.

+static ShmemHashDesc WaitEventCustomHashByNameDesc =
+{
+ .name = "WaitEventCustom hash by name",
+ .ptr = &WaitEventCustomHashByName,
+
+ .init_size = WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
+ .max_size = WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ /* key is a NULL-terminated string */
+ .hash_info.keysize = sizeof(char[NAMEDATALEN]),
+ .hash_info.entrysize = sizeof(WaitEventCustomEntryByName),
+ .hash_flags = HASH_ELEM | HASH_BLOBS,
+};

This was HASH_STRINGS originally, and it is used with plain const
char* parameters. Shouldn't it use HASH_SRINGS as before?

  size = add_size(size, ShmemRegisteredSize());

  size = add_size(size, dsm_estimate_size());
- size = add_size(size, DSMRegistryShmemSize());
+
+ size = add_size(size, ShmemRegisteredSize());

ShmemRegisteredSize is now called twice.

+    /* Initialize the lock */
+    tranche_id = LWLockNewTrancheId("my tranche name");
+    LWLockInitialize(&amp;MyShmem->lock);

Second parameter is missing for LWLockInitialize

  19. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-16T09:37:34Z

    On Fri, Mar 13, 2026 at 5:11 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 12/03/2026 22:05, Robert Haas wrote:
> > On Thu, Mar 12, 2026 at 3:21 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >>>> I'm currently leaning towards _PG_init(), except for allocations that
> >>>> depend on MaxBackends. For those, you can install a shmem_request_hook
> >>>> that sets the size in the descriptor. In other words, you can leave the
> >>>> 'size' as empty in _PG_init(), but set it later in the shmem_request_hook.
> >>>
> >>> Why can't you just do the whole thing later?
> >>
> >> shmem_request_hook won't work in EXEC_BACKEND mode, because in
> >> EXEC_BACKEND mode, ShmemRegisterStruct() also needs to be called at
> >> backend startup.
> >>
> >> One of my design goals is to avoid EXEC_BACKEND specific steps so that
> >> if you write your extension oblivious to EXEC_BACKEND mode, it will
> >> still usually work with EXEC_BACKEND. For example, if it was necessary
> >> to call a separate AttachShmem() function for every shmem struct in
> >> EXEC_BACKEND mode, but which was not needed on Unix, that would be bad.
> >
> > That's *definitely* a good goal. A less important but still valuable
> > goal is to maximize the notational simplicity of the mechanism. Your
> > callback idea is elegant in theory but in practice it seems like it
> > might make it harder for people to get started quickly on a new
> > module, and having to create the object in one place and then fill in
> > the size in another sort of has the same problem. I don't really know
> > what to do about that, but it's something to think about. The
> > complexity of getting the details right is annoyingly high in this
> > area.
>
> Yeah. IMHO the existing shmem_request/startup_hook mechanism is pretty
> awkward too, and in most cases, the new mechanism is more convenient. It
> might be slightly less convenient for some things, but mostly it's
> better. Would you agree with that, or do you actually like the old hooks
> and ShmemInitStruct() better?

FWIW, I like the new way.

If your goal is to get rid of ShmemInitStruct, ShmemInitHash,
shmem_request/startup_hook, we could replace the hooks by another hook
which gets called after MaxBackends is initialized but before
CalculateShmemSize() gets called. All StructShmemRegister() can be
called in that hook. _PG_init() is used to register the hook as it's
done today. If we are going to deprecate the old hooks in a couple of
releases, we will need to maintain three hooks but given that we don't
have to wait for several releases for the extensions to adapt to the
new hooks, it should be only a temporary measure.

>
> One such wrinkle with ShmemRegisterStruct() in the patch now is that
> it's harder to do initialization that touches multiple structs or hash
> tables. Currently the callbacks are called in the same order that the
> structs are registered, so you can do all the initialization in the last
> struct's callback. The single pair of shmem_request/startup_hooks per
> module was more clear in that aspect. Fortunately, that kind of
> cross-struct dependencies are pretty rare. So I think it's fine. (The
> order that the callbacks are called needs be documented explicitly though).
>
> If we want to improve on that, one idea would be to introduce a
> ShmemRegisterCallbacks() function to register callbacks that are not
> tied to any particular struct and are called after all the per-struct
> callbacks.
>

I think as long as we stick to calling the init/attach functions in
the order of their registration (or any well defined suitable order),
the module can use perform initization/setup of individual structures
to which the callback is attached and also modify/use previously
registered structures or do the setup in the last registered
structure's init. I think this is more flexible that required; maybe
overwhelmingly flexible.

> >>> Yeah, I think RequestNamedLWLockTranche() might be fine if you just
> >>> need LWLocks, but if you need a bunch of resources, putting them all
> >>> into the same chunk of memory seems cleaner.
> >>
> >> Agreed. Then again, how often do you need just a LWLock (or multiple
> >> LWLocks)? Surely you have a struct you want to protect with the lock. I
> >> guess having shmem hash table but no struct would be pretty common, though.
> >
> > Yeah, we've developed an annoying number of different ways to do this
> > stuff. I don't entirely know how to fix that.
>
> Here's a new version that doubles down on the
> LWLockNewTrancheId+LWLockInitialize method, by changing the example in
> the docs, and contrib/pg_stat_statements, to use that method.
> RequestNamedLWLockTranche() still works, there are no changes to it,
> it's just not as convenient to use with ShmemRegisterStruct(). This has
> the advantage that we don't introduce yet another way of allocating LWLocks.

The new hook could be used to request LWLockTranche as well.

-- 
Best Wishes,
Ashutosh Bapat

  20. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-16T10:28:34Z

    On Sat, Mar 14, 2026 at 2:39 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 06/03/2026 16:12, Heikki Linnakangas wrote:
> > Firstly, I'm not sure what to do with ShmemRegisterHash() and the
> > 'HASHCTL *infoP' argument to it. I feel it'd be nicer if the HASHCTL was
> > just part of the ShmemHashDesc struct, but I'm not sure if that fits all
> > the callers. I'll have to try that out I guess.
> I took a stab at that, and it turned out to be straightforward. I'm not
> sure why I hesitated on that earlier.
>

Yeah. I wondered about that too.

> Here's a new version with that change, and a ton of little comment
> cleanups and such.

Here are initial comments on these patches.

0001

@@ -3236,6 +3239,8 @@ PostmasterStateMachine(void)
LocalProcessControlFile(true);
/* re-create shared memory and semaphores */
+ ResetShmemAllocator();

This name is misleading. The function does not touch ShmemAllocator at
all. Instead it resets the ShmemStruct registry. I suggest
ResetShememRegistry() instead.

+ *
+ * There are two kinds of shared memory data structures: fixed-size structures
+ * and hash tables.

In future we will have resizable "fixed" structures and we may also
have resizable hash tables i.e. hash tables whose directory would be
resizable. The later would be help support resizable shared buffers
lookup table. It will be good to write the above sentence so that we
can just add more types of data structures without needing to rewrite
everything. If we could find a good term for "fixed-size structures"
which are really "structures that require contiguous memory", we will
be able to write the above sentence as "There are two kinds of shared
memory structures: contiguous structures and hash tables.". When we
add resizable structures, we can just add a sentence "A contiguous
structure may be fixed size or resizable". When we add resizable hash
tables, we can just replace that with "Both of these kinds can be
fixed-size or resizable". I am not sure whether "contiguous
structures" is a good term though (for one, word contiguous can be
confused with continuous). Whatever term we use should be something
that we can carry further in the remaining paragraphs.

+ Fixed-size structures contain things like global
+ * variables for a module and should never be allocated after the shared
+ * memory initialization phase.

I think the existing comment is not accurate. The term "global
variables" in the sentence can be confused with process global
variables. We should be using the term "shared variables" or better
"shared state". If we adopt "contiguous structures" as the term for
the first kind of data structure, we can write "Contiguous structures
contain shared state, maintained in a contiguous chunk of memory, for
a module. It should never be allocated after the shared memory
initialization phase.".

+ * postmaster calls ShmemInitRegistered(), which calls the init_fn callbacks
+ * of each registered area, in the order that they were registered.

... calls the init_fn, if any, of each registered area ....

- infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size);
- infoP->alloc = ShmemAllocNoError;
- hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+ desc->hash_info.dsize = desc->hash_info.max_dsize =
hash_select_dirsize(desc->max_size);
+ desc->hash_info.alloc = ShmemAllocNoError;
+ desc->hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
/* look it up in the shmem index */

The next several lines of code look up shmem index. Should we remove
this comments or modify it to say "Register and initialize the hash
table".

+HTAB *
+ShmemInitHash(const char *name, /* table string name for shmem index */
+ int64 init_size, /* initial table size */
+ int64 max_size, /* max size of the table */
+ HASHCTL *infoP, /* info about key and bucket size */
+ int hash_flags) /* info about infoP */
+{
+ ShmemHashDesc *desc;
+

... snip ...

+
+ ShmemRegisterHash(desc);
+ return *desc->ptr;
+}

I like the way these functions are written using the new API. I think
we should keep these legacy interface at the end of section of shmem
APIs, rather than keeping those at the end of the file where we have
monitoring and arithmetic functions. If you want to get rid of the
legacy APIs in this release itself, I think it's ok to keep them at
the end of the file.

ShmemInitStruct() now calls ShmemRegisterStruct(). Earlier it could be
called from any backend, in any state to fetch a pointer to a shared
memory structure. It didn't add a new structure. Now it can add a new
structure. I am wondering whether that can cause registry in different
backends to get out of sync. Should we limit the window when it can be
called just like how shmem_request_hook call is limited. In that sense
ShmemRegisterStruct() looks something to be called from a
shmem_register_hook which is also called from EXEC_BACKEND. Sorry to
expand it here rather in my previous reply. In case we replace all the
current calls to ShmemInitStruct() with ShmemRegisterStruct(), we may
be able to get rid of the Shmem Index altogether; after all it's used
only for fetching the pointers to the shared memory areas in
EXEC_BACKEND mode. I thought that we could save the registry in the
shared memory. In EXEC_BACKEND mode, we go over the registry calling
attach_fn for each entry. But since the binary is overwritten in
EXEC_BACKEND case, attach/init fns are not guaranteed to have the same
address in all the backends. Maybe we have to resort to
launch_backend() to transfer the registry to the backend through the
file (to keep it in sycn in all the backends): a solution you may not
like.

+ void **ptr;
+} ShmemStructDesc;


I think the comments for each member should highlight which of these
fields are required (non-zero) and which can be optional (zero'ed
out).

+ */
+ ShmemStructDesc base_desc;

Once we have calculated the base_desc in ShmemRegisterHash() and
called ShmemRegisterStruct(), we don't need base_desc anymore. Even
the pointer to the allocated hash table memory is available through
*ptr. Probably we could just remove this member from here.
ShmemRegisterHash() can declare a variable of type ShmemStructDesc,
populate it based on the members in this structure and pass it to
ShmemRegisterStruct(). I am not comfortable with specification
structure being modified by the registration function.

0003
- pages = (size / FPM_PAGE_SIZE) - first_page;
- FreePageManagerPut(fpm, first_page, pages);
- }
+ ShmemRegisterStruct(&dsm_main_space_shmem_desc);

Shouldn't we be setting dsm_main_space_shmem_desc.size here to size
before calling ShmemRegisterStruct()?

@@ -102,15 +102,14 @@ CalculateShmemSize(void)
size = add_size(size, ShmemRegisteredSize());
size = add_size(size, dsm_estimate_size());

We have defined dsm_main_space_shmem_desc, but we still use
dsm_estimate_size() here and initialize the memory in
dsm_shmem_init(), which is explicitily called from
CreateOrAttachShmemStructs(). Why is that? Shouldn't we be registering
the structure in RegisterShmemStructs(), and let ShmemInitRegistered()
initialize it? Am I missing something here?

I will continue to review the patches further.

-- 
Best Wishes,
Ashutosh Bapat

  21. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-16T21:56:23Z

    On 16/03/2026 12:28, Ashutosh Bapat wrote:
> 0001
> 
> @@ -3236,6 +3239,8 @@ PostmasterStateMachine(void)
> LocalProcessControlFile(true);
> /* re-create shared memory and semaphores */
> + ResetShmemAllocator();
> 
> This name is misleading. The function does not touch ShmemAllocator at
> all. Instead it resets the ShmemStruct registry. I suggest
> ResetShememRegistry() instead.

Hmm. Come to think of it, this isn't quite right for 
shared_preload_libraries. On crash restart, we don't call _PG_init() for 
shared_preload_libraries. So if ShmemRegisterStruct() is called from 
_PG_init(), it won't get called again after restart, and hence we should 
retain the registration. This is different from ShmemInitStruct(), which 
does get called again after crash restart.

I fixed that by retaining the structs registered with 
ShmemRegisterStruct(), and only clearing out entries made with 
ShmemInitStruct(). I think that's the right thing to do, but I feel a 
little uneasy about keeping stuff across restarts. This isn't a 
completely new issue; even without this patch I feel uneasy about 
keeping the shared memory segment and not calling _PG_init() for 
shared_preload_libraries again. I wish crash restart could reset things 
even more thoroughly.

If we introduced a shmem_register_hook like you suggested, we could call 
that again after restart. But it feels silly; _PG_init() would register 
a shmem_register_hook, which would register the shared memory desc. Why 
not just register the shared memory desc directly? I don't think we 
really gain anything by the extra step. (It might be useful to allow the 
size to be calculated later, taking MaxBackends into account. What I 
mean here is that it would be a silly dance just to handle crash restarts)

> + *
> + * There are two kinds of shared memory data structures: fixed-size structures
> + * and hash tables.
> 
> In future we will have resizable "fixed" structures and we may also
> have resizable hash tables i.e. hash tables whose directory would be
> resizable. The later would be help support resizable shared buffers
> lookup table. It will be good to write the above sentence so that we
> can just add more types of data structures without needing to rewrite
> everything. If we could find a good term for "fixed-size structures"
> which are really "structures that require contiguous memory", we will
> be able to write the above sentence as "There are two kinds of shared
> memory structures: contiguous structures and hash tables.". When we
> add resizable structures, we can just add a sentence "A contiguous
> structure may be fixed size or resizable". When we add resizable hash
> tables, we can just replace that with "Both of these kinds can be
> fixed-size or resizable". I am not sure whether "contiguous
> structures" is a good term though (for one, word contiguous can be
> confused with continuous). Whatever term we use should be something
> that we can carry further in the remaining paragraphs.
> + Fixed-size structures contain things like global
> + * variables for a module and should never be allocated after the shared
> + * memory initialization phase.
> 
> I think the existing comment is not accurate. The term "global
> variables" in the sentence can be confused with process global
> variables. We should be using the term "shared variables" or better
> "shared state". If we adopt "contiguous structures" as the term for
> the first kind of data structure, we can write "Contiguous structures
> contain shared state, maintained in a contiguous chunk of memory, for
> a module. It should never be allocated after the shared memory
> initialization phase.".

I went with "variables shared between all backend processes", and did 
another pass of other copy-editing too. No doubt this needs to be 
updated again when the resizing patch lands.

> - infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size);
> - infoP->alloc = ShmemAllocNoError;
> - hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
> + desc->hash_info.dsize = desc->hash_info.max_dsize =
> hash_select_dirsize(desc->max_size);
> + desc->hash_info.alloc = ShmemAllocNoError;
> + desc->hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
> /* look it up in the shmem index */
> 
> The next several lines of code look up shmem index. Should we remove
> this comments or modify it to say "Register and initialize the hash
> table".

Replaced it with "/* Set up the base struct descriptor */"

> +HTAB *
> +ShmemInitHash(const char *name, /* table string name for shmem index */
> + int64 init_size, /* initial table size */
> + int64 max_size, /* max size of the table */
> + HASHCTL *infoP, /* info about key and bucket size */
> + int hash_flags) /* info about infoP */
> +{
> + ShmemHashDesc *desc;
> +
> 
> ... snip ...
> 
> +
> + ShmemRegisterHash(desc);
> + return *desc->ptr;
> +}
> 
> I like the way these functions are written using the new API. I think
> we should keep these legacy interface at the end of section of shmem
> APIs, rather than keeping those at the end of the file where we have
> monitoring and arithmetic functions. If you want to get rid of the
> legacy APIs in this release itself, I think it's ok to keep them at
> the end of the file.

I don't plan to get rid of the legacy API any time soon, I expect 
existing extensions to continue using it for years to come. So I moved 
them per your suggestion.

> ShmemInitStruct() now calls ShmemRegisterStruct(). Earlier it could be
> called from any backend, in any state to fetch a pointer to a shared
> memory structure. It didn't add a new structure. Now it can add a new
> structure. I am wondering whether that can cause registry in different
> backends to get out of sync. Should we limit the window when it can be
> called just like how shmem_request_hook call is limited. In that sense
> ShmemRegisterStruct() looks something to be called from a
> shmem_register_hook which is also called from EXEC_BACKEND. Sorry to
> expand it here rather in my previous reply. In case we replace all the
> current calls to ShmemInitStruct() with ShmemRegisterStruct(), we may
> be able to get rid of the Shmem Index altogether; after all it's used
> only for fetching the pointers to the shared memory areas in
> EXEC_BACKEND mode.

I think it's still useful to allow ShmemRegisterStruct() after 
postmaster startup, so that you can use it with extensions that are not 
listed in shared_preload_libraries, but need a little bit of shared 
memory. Hmm, perhaps we should add an explicit flag for that case, 
though. So that by default ShmemRegisterStruct() fails if you call it 
after postmaster startup, but you could allow it by setting a flag in 
the descriptor.

> I thought that we could save the registry in the
> shared memory. In EXEC_BACKEND mode, we go over the registry calling
> attach_fn for each entry. But since the binary is overwritten in
> EXEC_BACKEND case, attach/init fns are not guaranteed to have the same
> address in all the backends. Maybe we have to resort to
> launch_backend() to transfer the registry to the backend through the
> file (to keep it in sycn in all the backends): a solution you may not
> like.
Yep, can't have function pointers in shared memory unfortunately.

> + void **ptr;
> +} ShmemStructDesc;
> 
> 
> I think the comments for each member should highlight which of these
> fields are required (non-zero) and which can be optional (zero'ed
> out).

Added some comments on that.

> + */
> + ShmemStructDesc base_desc;
> 
> Once we have calculated the base_desc in ShmemRegisterHash() and
> called ShmemRegisterStruct(), we don't need base_desc anymore. Even
> the pointer to the allocated hash table memory is available through
> *ptr. Probably we could just remove this member from here.
> ShmemRegisterHash() can declare a variable of type ShmemStructDesc,
> populate it based on the members in this structure and pass it to
> ShmemRegisterStruct(). I am not comfortable with specification
> structure being modified by the registration function.

Hmm, the ShmemStructDesc is accessed in ShmemInitRegistered() and in the 
shmem_hash_init() callback, so 'base_desc' cannot be a local variable in 
ShmemRegisterHash(). It could be allocated in TopMemoryContext though. 
That would hide it from ShmemHashDesc.

I'm a little ambivalent about that. It would be nice to hide it if it's 
not intended to be accessed outside shmem.c. Then again, it might be 
useful to peek into it in the future, depending on what fields we add. 
and it might be handy to look at in a debugger.

> @@ -102,15 +102,14 @@ CalculateShmemSize(void)
> size = add_size(size, ShmemRegisteredSize());
> size = add_size(size, dsm_estimate_size());
> 
> We have defined dsm_main_space_shmem_desc, but we still use
> dsm_estimate_size() here and initialize the memory in
> dsm_shmem_init(), which is explicitily called from
> CreateOrAttachShmemStructs(). Why is that? Shouldn't we be registering
> the structure in RegisterShmemStructs(), and let ShmemInitRegistered()
> initialize it? Am I missing something here?

Yes you're right. Fixed that and all the the bugs that Zsolt pointed 
out. Thanks!

- Heikki

  22. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-17T11:58:05Z

    On Tue, Mar 17, 2026 at 3:26 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
>
> I don't plan to get rid of the legacy API any time soon, I expect
> existing extensions to continue using it for years to come. So I moved
> them per your suggestion.

Do you plan to get rid of the shmem_request_hook and
shmem_startup_hook? What's the plan there? Wouldn't the old APIs and
new APIs overwhelm extension writers - having two different ways and
three different hooks to allocate a structure in the shared memory.

>
> > ShmemInitStruct() now calls ShmemRegisterStruct(). Earlier it could be
> > called from any backend, in any state to fetch a pointer to a shared
> > memory structure. It didn't add a new structure. Now it can add a new
> > structure. I am wondering whether that can cause registry in different
> > backends to get out of sync. Should we limit the window when it can be
> > called just like how shmem_request_hook call is limited. In that sense
> > ShmemRegisterStruct() looks something to be called from a
> > shmem_register_hook which is also called from EXEC_BACKEND. Sorry to
> > expand it here rather in my previous reply. In case we replace all the
> > current calls to ShmemInitStruct() with ShmemRegisterStruct(), we may
> > be able to get rid of the Shmem Index altogether; after all it's used
> > only for fetching the pointers to the shared memory areas in
> > EXEC_BACKEND mode.
>
> I think it's still useful to allow ShmemRegisterStruct() after
> postmaster startup, so that you can use it with extensions that are not
> listed in shared_preload_libraries, but need a little bit of shared
> memory. Hmm, perhaps we should add an explicit flag for that case,
> though. So that by default ShmemRegisterStruct() fails if you call it
> after postmaster startup, but you could allow it by setting a flag in
> the descriptor.
>

The hash tables do not allocate all their entries upfront but they
request shared memory for their maximum size. Before they could grow
to the maximum of their size, if somebody calls ShmemInitStruct with a
huge memory size request that takes away all the reserved address
space/memory, the hash table won't get its fair share of shared
memory. I agree that it could still happen if a hash table grows
beyond the contracted request ... but it's atleast registered. I think
we should prevent such cases. We could keep track of the extra shared
memory that we have allocated and refuse an unregistered request if
that memory is exhausted. But I think we already have some unaccounted
for structures e.g. ShmemAllocator and ShmemHeader already. So it
might turn out to be more complex that required. I have feeling that
we are providing flexibility that the current infrastructure can not
support. I am not opposed to supporting ShmemRegisterStruct; I like
the idea, but it seems premature.

--
Best Wishes,
Ashutosh Bapat

  23. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-18T19:30:10Z

    On Fri, Mar 13, 2026 at 7:41 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> Yeah. IMHO the existing shmem_request/startup_hook mechanism is pretty
> awkward too, and in most cases, the new mechanism is more convenient. It
> might be slightly less convenient for some things, but mostly it's
> better. Would you agree with that, or do you actually like the old hooks
> and ShmemInitStruct() better?

I'd say it's not massively different one way or the other. Looking at
the pg_stat_statements changes in particular, I feel like in v2, it
was actually worse, because you didn't get manage to get rid of
pgss_shmem_request(), but it was no longer the thing requesting shmem.
v3 is better in that regard: you get rid of some complexity in
exchange for what you add. It's not amazing, though:
pgss_shmem_startup() now has nothing to do with the name of the hook,
which is not this patch's fault but also isn't solved by it. I wonder
why in the world somebody decided to jam all of this non-shmem-related
logic into this function, and why they didn't add a comment explaining
why it was here and not someplace else. One of the worst things about
this area is that I often end up having to trace through a bunch of
postmaster startup logic to figure out whether any given bit of code
is actually in the right place, and that makes me feel like the hooks
are badly designed. pgss_shmem_startup() is a good example of that,
and the fact that it needs an IsUnderPostmaster check is another.

We should somehow try to make it clear what happens with this new
mechanism if a module is loaded via shared_preload_libraries vs. if
it's loaded via LOAD or session_preload_libraries or whatever. Writing
modules that don't require shared_preload_libraries is a boon to
users, because they can be added to a production system without a
server restart. I wonder whether this new facility handles that case
better, worse, or the same as existing facilities.

> One such wrinkle with ShmemRegisterStruct() in the patch now is that
> it's harder to do initialization that touches multiple structs or hash
> tables. Currently the callbacks are called in the same order that the
> structs are registered, so you can do all the initialization in the last
> struct's callback. The single pair of shmem_request/startup_hooks per
> module was more clear in that aspect. Fortunately, that kind of
> cross-struct dependencies are pretty rare. So I think it's fine. (The
> order that the callbacks are called needs be documented explicitly though).
>
> If we want to improve on that, one idea would be to introduce a
> ShmemRegisterCallbacks() function to register callbacks that are not
> tied to any particular struct and are called after all the per-struct
> callbacks.

I think the whole idea of ShmemInitHash() and ShmemInitStruct() is
relatively poorly designed in this regard. Ideally, I want to
initialize all the shared memory that belongs to me, and that might
include arbitrary data structures, but the current design kind of
thinks that I want one struct or one hash table and nothing else. If
we're redesigning this mechanism, it would sure be nice to improve on
that.

Looking just at the pg_stat_statements changes, I think my overall
view on this right now is that it's not terrible, but I'm also not
that happy about introducing yet another way to do it for this amount
of gain. To me, it doesn't yet rise to the level of a clear win.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  24. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-19T10:31:10Z

    On 18/03/2026 21:30, Robert Haas wrote:
> On Fri, Mar 13, 2026 at 7:41 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> Yeah. IMHO the existing shmem_request/startup_hook mechanism is pretty
>> awkward too, and in most cases, the new mechanism is more convenient. It
>> might be slightly less convenient for some things, but mostly it's
>> better. Would you agree with that, or do you actually like the old hooks
>> and ShmemInitStruct() better?
> 
> I'd say it's not massively different one way or the other. Looking at
> the pg_stat_statements changes in particular, I feel like in v2, it
> was actually worse, because you didn't get manage to get rid of
> pgss_shmem_request(), but it was no longer the thing requesting shmem.
> v3 is better in that regard: you get rid of some complexity in
> exchange for what you add. It's not amazing, though:
> pgss_shmem_startup() now has nothing to do with the name of the hook,
> which is not this patch's fault but also isn't solved by it. I wonder
> why in the world somebody decided to jam all of this non-shmem-related
> logic into this function, and why they didn't add a comment explaining
> why it was here and not someplace else. One of the worst things about
> this area is that I often end up having to trace through a bunch of
> postmaster startup logic to figure out whether any given bit of code
> is actually in the right place, and that makes me feel like the hooks
> are badly designed. pgss_shmem_startup() is a good example of that,
> and the fact that it needs an IsUnderPostmaster check is another.

Hmm, I assumed it was important that the pg_stat_statements file is 
loaded later in the startup sequence, and that's why it was in 
pgss_shmem_startup(). But now that I look at it, I don't think there was 
any grand plan, shmem_startup_hook was just the only easy way to get 
control during postmaster startup, after shmem initialization.

So I think we can move that code to the init_fn callback with the new 
API, and that gets rid of shmem_startup_hook in pg_stat_statements. See 
attached (v6-0005-move-pgss-shmem_startup-hook-code-into-the-new-in.patch).

> We should somehow try to make it clear what happens with this new
> mechanism if a module is loaded via shared_preload_libraries vs. if
> it's loaded via LOAD or session_preload_libraries or whatever. Writing
> modules that don't require shared_preload_libraries is a boon to
> users, because they can be added to a production system without a
> server restart. I wonder whether this new facility handles that case
> better, worse, or the same as existing facilities.

Pretty much the same I think. Point taken that it could be documented 
better.

The old documentation for ShmemInitStruct() assumed that it would be 
used from shared_preload_libraries, it didn't. With the new API, I tried 
to document how it behaves when used outside shared_preload_libraries, 
but still focused on how using it from shared_preload_libraries. I'm not 
sure it helped. Perhaps the after-startup behavior should be put in a 
separate section.

>> One such wrinkle with ShmemRegisterStruct() in the patch now is that
>> it's harder to do initialization that touches multiple structs or hash
>> tables. Currently the callbacks are called in the same order that the
>> structs are registered, so you can do all the initialization in the last
>> struct's callback. The single pair of shmem_request/startup_hooks per
>> module was more clear in that aspect. Fortunately, that kind of
>> cross-struct dependencies are pretty rare. So I think it's fine. (The
>> order that the callbacks are called needs be documented explicitly though).
>>
>> If we want to improve on that, one idea would be to introduce a
>> ShmemRegisterCallbacks() function to register callbacks that are not
>> tied to any particular struct and are called after all the per-struct
>> callbacks.
> 
> I think the whole idea of ShmemInitHash() and ShmemInitStruct() is
> relatively poorly designed in this regard. Ideally, I want to
> initialize all the shared memory that belongs to me, and that might
> include arbitrary data structures, but the current design kind of
> thinks that I want one struct or one hash table and nothing else. If
> we're redesigning this mechanism, it would sure be nice to improve on
> that.
> 
> Looking just at the pg_stat_statements changes, I think my overall
> view on this right now is that it's not terrible, but I'm also not
> that happy about introducing yet another way to do it for this amount
> of gain. To me, it doesn't yet rise to the level of a clear win.

So here's another idea (not yet implemented in the attached patch): 
instead of thinking in terms of individual shmem structs and hashes, 
let's introduce a concept of a "subsystem" that ties them together:

static pgss_subsystem_desc = {
     .name = "pg_stat_statements",
     .shmem_structs = {
         {
             .ptr = (void **) &pgss,
             .size = sizeof(pgssSharedState),
         },
     },
     .shmem_hashes = {
         {
             .ptr = &pgss_hash,

             .init_size = 0,          /* set from 'pgss_max' */
             .max_size = 0,           /* set from 'pgss_max' */
             .hash_info.keysize = sizeof(pgssHashKey),
             .hash_info.entrysize = sizeof(pgssEntry),
             .hash_flags = HASH_ELEM | HASH_BLOBS,
         },
     },

     /* called after the shmem structs and hashes have been allocated */
     .init_fn = pgss_shmem_init,
}

void
_PG_init(void)
{
     ...
     RegisterSubsystem(&pgss_subsystem_desc);
}

We could add more callbacks that get called at different times. For 
example the callback that would get called before shared memory is 
allocated, which could adjust the size according to MaxBackends. That 
would fully replace shmem_request_hook. Or a callback that would get 
called later in the startup sequence, if we wanted to e.g. load the 
pg_stat_statements file later in startup.

This would be a natural place for other resources in future too. We 
could support declaring "named lwlock tranches" here to replace 
RequestNamedLWLockTranche() for example, although I think it's still 
better to encourage embedding the LWLock in the struct instead.

_PG_init in pg_stat_statements still does a lot more than register that 
struct. It declares the GUCs and installs other hooks for example. We 
could perhaps move those to the subsystem descriptor too, although I'm 
not sure if that's worth the code churn.

- Heikki

  25. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T13:36:36Z

    On Thu, Mar 19, 2026 at 6:31 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> We could add more callbacks that get called at different times. For
> example the callback that would get called before shared memory is
> allocated, which could adjust the size according to MaxBackends. That
> would fully replace shmem_request_hook. Or a callback that would get
> called later in the startup sequence, if we wanted to e.g. load the
> pg_stat_statements file later in startup.
>
> This would be a natural place for other resources in future too. We
> could support declaring "named lwlock tranches" here to replace
> RequestNamedLWLockTranche() for example, although I think it's still
> better to encourage embedding the LWLock in the struct instead.
>
> _PG_init in pg_stat_statements still does a lot more than register that
> struct. It declares the GUCs and installs other hooks for example. We
> could perhaps move those to the subsystem descriptor too, although I'm
> not sure if that's worth the code churn.

Without taking a strong position on this particular design idea, I
kind of wonder if we should be going the opposite direction: instead
of bundling more and more things into descriptors, try to let people
write declarative code that does whatever it is that they want to do.
I think descriptors are pretty limiting as a concept, because they can
only do the exact things that they're designed to do. For instance, I
find the change to use LWLockPadded rather than LWLock * in
pgssSharedState to be a clear improvement, because I'd rather have
fewer objects and less pointer-chasing. Now, that LWLockPadded is
going to need to be initialized. I would rather do that by writing
LWLockInitialize(&pgss->lock.lock, tranche_id) as you did than by
adding something to a descriptor, since doing the latter is almost
certainly going to be a less intuitive syntax for the same thing and
I'm going to have to spend time verifying that whatever locution I'm
compelled to write actually does what I want. And if somebody adds
"really light weight locks" to the system then we'll have to add
RLWLockInitialize() to the things that the descriptor system knows how
to do, and if for some reason I want to do
LWLockInitialize(&mythingy->lock.lock, some_random_condition ?
this_tranche_id : that_tranche_id), the descriptor system will
probably become an annoying straightjacket. Or if that example isn't
compelling enough, imagine that I have an array of structs each of
which contains an LWLockPadded and I need to go loop over the array
and do all the initializations. Or maybe space is at a premium and I
want to use LWLock rather than LWLockPadded. Or maybe something else.
Code is just more flexible than having to go through descriptors,
which is why a lot of modern languages go to a great deal of trouble
to make closures a first-class concept.

Let me take a step back and say what I think the problems in this area
are, to see whether we agree on the basics. I suspect the reason
you've undertaken this project is the fact that, currently, requesting
shared memory and allocating it are totally decoupled. The number of
bytes you request and the number of bytes you actually allocated could
be totally different, and then some completely unrelated subsystem can
break because it's allocating last, and even though it requested the
bytes it wanted to allocate, some other subsystem under-requested and
now the bytes this subsystem wants are not actually available.
Tracking this kind of thing down can be a giant pain in the rear end,
bordering on impossible IME. Also, if we want to be able to resize
stuff in shared memory in some happy future, the need for precise
tracking of these sorts of things presumably goes way up, although the
exact details of that are not altogether clear to me. Furthermore, as
you point out, even if everyone behaves themselves and requests and
allocates the same number of bytes, that's still annoying if it means
redoing some computation.

I think the answer to this problem is to make requests into named
objects. You're not allowed to request a number of BYTES of shared
memory any more; you have to request "the shared memory bytes for the
object named XYZ". So instead of
RequestAddinShmemSpace(pgss_memsize()), you would say something like
RequestAddinShmemSpace("pg_stat_staements", pgss_memsize()) and then
later instead of saying pgss = ShmemInitStruct("pg_stat_statements",
sizeof(pgssSharedState), &found), you say pgss =
ShmemInitStruct("pg_stat_statements", &size, &found).

The other big problem that I think we have in this area is that it's
unclear what you're allowed to do in _PG_init() vs. some other
callback, and sometimes you need IsUnderPostmaster checks or
IsPostmasterEnvironment checks or
process_shared_preload_libraries_in_progress checks. From my point of
view, good goals would include (1) moving as much logic as possible
into _PG_init() vs. having to put it elsewhere and (2) removing as
many conditional checks as possible from it and aiming for _PG_init()
functions that just run from start to finish in all cases.

What _PG_init() already does pretty well is allow you to do
per-backend setup. For instance, pg_plan_advice needs no shared
resources, so _PG_init() was easy to write and, IMHO, easy to read.
It's requesting diverse types of resources -- GUCs, an EXPLAIN
extension ID, an EXPLAIN option, and hooks, but it can just do all of
those things one after another with no conditional logic and, IMHO,
life is great. We fall down a little bit because of the fact that
PGC_POSTMASTER GUCs can't be added after startup -- see autoprewarm.c,
for instance, which calls out that problem implicitly; and I suspect
that issue is also why pg_stat_statements has the
process_shared_preload_libraries_in_progress check at the top, because
it looks to me like everything else that the function does would be
completely fine to do later. So maybe we could adjust
DefineCustomBLAHVariable to do nothing if there's a PGC_POSTMASTER
variable requested and it's too late to create one, instead of blowing
up. Or create the variable but attach some property to it that causes
it to generate an error when set, e.g. ERROR: pg_stat_statements.max
cannot be changed now because the library that created it was not
included in shared_preload_libraries at startup time (wordsmithing
likely needed).

Shared resources do require some split-up of initialization: as you
point out, if _PG_init() is called before we know MaxBackends, then we
can't even size data structures who size depends on that quantity yet,
and we certainly can't initialize anything, because shared memory
might not have been created yet. I don't think we can completely avoid
the need for callbacks here, but... just spitballing, how about
something like this:

extern void DefineShmemRegion(char *name, size_t size, void
**localptr, void (*init_callback)(void *), int flags);
extern void DefineShmemRegionDynamic(char *name, size_t
(*sizing_callback)(void *), void **localptr, void
(*init_callback)(void *), int flags);
extern void *GetShmemRegion(char *name);

#define DSR_FLAGS_NO_SLOP  0x01
#define DSR_FLAGS_DSA_OK 0x02

If DefineShmemRegion() or DefineShmemRegionDynamic() is called at
shared_preload_libraries time, it arranges to increase the size of the
main shared memory segment by the given amount, or the computed amount
(for things that depend on MaxBackends). Then, once the main shared
memory segment is created, it invokes the init_callback and sets
*localptr. If either of these functions are called after the main
shared memory segment has been created, they check for an existing
allocation, and if one is found, they just set *localptr. (They can
actually probably exit quickly if *localptr is already set.)
Otherwise, they try to allocate from DSA if DSR_FLAGS_DSA_OK is given,
or else from the slop space unless DSR_FLAGS_NO_SLOP is given. If that
works, they then call the init_callback under a suitable lock and set
*localptr. If not, they fail silently. Functions that need to use the
local pointers do something like this:

if (unlikely(pgss == NULL))
    pgss = GetShmemRegion("pg_stat_statements");

...which throws a suitable error -- not just a generic one that the
region doesn't exist, but something that's sensitive to different
failure conditions: the region was never registered, the region was
registered after shared_preload_libraries time and not enough slop
space remains, or whatever. GetShmemRegion() could even retry the
initialization in certain cases, e.g. if DSA is OK and we previously
were called too early in startup to try DSA, we can try now, or if DSA
allocation failed due to OOM, we can try again.

I *think* this design gets rid of all the IsUnderPostmaster and
shared_preload_libraries_in_progress checks in individual subsystems,
and all the use of shmem startup hooks. You just ask for what you want
and if there's a way to get it, the system gives it to you, and if
there's not, it generates an error at the latest possible time, and
also tries to self-heal if that's reasonable. If you do load your
module in shared_preload_libraries, then by the time main shared
memory initialization completes in the postmaster, everyone's localptr
values (like pgss) will be initialized, but if it happens to be an
EXEC_BACKEND build, those same calls will also happen in every
postmaster child, and will automatically re-find the shared memory
areas and reinitialize all the pointers. If you load your module
later, your localptr values will hopefully be initialized by the end
of _PG_init(), but if that doesn't work out, then the
unlikely-protected calls to GetShmemRegion will produce suitable
errors at a suitable time. And I think it all works out nicely in a
standalone backend, too.

This is all kind of a brain dump and is not fully thought-through and
might be riddled with cognitive errors, but what I'm sort of trying to
convey is where I think the complexity in the current system comes
from (which is that we require every subsystem/extension author to
know how the sausage is made, and we don't enforce consistency between
requests and allocations) and what I don't really like about
descriptors as a solution (which is that they are harder to read than
imperative code and can interfere with cases where somebody wants to
do something slightly different than what the descriptor-designer had
in mind). I hope that some of it is helpful to you...

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  26. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-19T14:34:11Z

    On 19/03/2026 15:36, Robert Haas wrote:
> Without taking a strong position on this particular design idea, I
> kind of wonder if we should be going the opposite direction: instead
> of bundling more and more things into descriptors, try to let people
> write declarative code that does whatever it is that they want to do.
> I think descriptors are pretty limiting as a concept, because they can
> only do the exact things that they're designed to do. For instance, I
> find the change to use LWLockPadded rather than LWLock * in
> pgssSharedState to be a clear improvement, because I'd rather have
> fewer objects and less pointer-chasing. Now, that LWLockPadded is
> going to need to be initialized. I would rather do that by writing
> LWLockInitialize(&pgss->lock.lock, tranche_id) as you did than by
> adding something to a descriptor, since doing the latter is almost
> certainly going to be a less intuitive syntax for the same thing and
> I'm going to have to spend time verifying that whatever locution I'm
> compelled to write actually does what I want. And if somebody adds
> "really light weight locks" to the system then we'll have to add
> RLWLockInitialize() to the things that the descriptor system knows how
> to do, and if for some reason I want to do
> LWLockInitialize(&mythingy->lock.lock, some_random_condition ?
> this_tranche_id : that_tranche_id), the descriptor system will
> probably become an annoying straightjacket. Or if that example isn't
> compelling enough, imagine that I have an array of structs each of
> which contains an LWLockPadded and I need to go loop over the array
> and do all the initializations. Or maybe space is at a premium and I
> want to use LWLock rather than LWLockPadded. Or maybe something else.
> Code is just more flexible than having to go through descriptors,
> which is why a lot of modern languages go to a great deal of trouble
> to make closures a first-class concept.

Sure, descriptors are restricting. I'm not proposing to move everything 
into descriptors.

> Let me take a step back and say what I think the problems in this area
> are, to see whether we agree on the basics. I suspect the reason
> you've undertaken this project is the fact that, currently, requesting
> shared memory and allocating it are totally decoupled. The number of
> bytes you request and the number of bytes you actually allocated could
> be totally different, and then some completely unrelated subsystem can
> break because it's allocating last, and even though it requested the
> bytes it wanted to allocate, some other subsystem under-requested and
> now the bytes this subsystem wants are not actually available.
> Tracking this kind of thing down can be a giant pain in the rear end,
> bordering on impossible IME. Also, if we want to be able to resize
> stuff in shared memory in some happy future, the need for precise
> tracking of these sorts of things presumably goes way up, although the
> exact details of that are not altogether clear to me. Furthermore, as
> you point out, even if everyone behaves themselves and requests and
> allocates the same number of bytes, that's still annoying if it means
> redoing some computation.

Agreed, yes, that's what I'm trying to address.

> I think the answer to this problem is to make requests into named
> objects. You're not allowed to request a number of BYTES of shared
> memory any more; you have to request "the shared memory bytes for the
> object named XYZ". So instead of
> RequestAddinShmemSpace(pgss_memsize()), you would say something like
> RequestAddinShmemSpace("pg_stat_staements", pgss_memsize()) and then
> later instead of saying pgss = ShmemInitStruct("pg_stat_statements",
> sizeof(pgssSharedState), &found), you say pgss =
> ShmemInitStruct("pg_stat_statements", &size, &found).

Yeah, that's a little better than what we have today.

> The other big problem that I think we have in this area is that it's
> unclear what you're allowed to do in _PG_init() vs. some other
> callback, and sometimes you need IsUnderPostmaster checks or
> IsPostmasterEnvironment checks or
> process_shared_preload_libraries_in_progress checks. From my point of
> view, good goals would include (1) moving as much logic as possible
> into _PG_init() vs. having to put it elsewhere and (2) removing as
> many conditional checks as possible from it and aiming for _PG_init()
> functions that just run from start to finish in all cases.

Agreed.

> What _PG_init() already does pretty well is allow you to do
> per-backend setup. For instance, pg_plan_advice needs no shared
> resources, so _PG_init() was easy to write and, IMHO, easy to read.
> It's requesting diverse types of resources -- GUCs, an EXPLAIN
> extension ID, an EXPLAIN option, and hooks, but it can just do all of
> those things one after another with no conditional logic and, IMHO,
> life is great. We fall down a little bit because of the fact that
> PGC_POSTMASTER GUCs can't be added after startup -- see autoprewarm.c,
> for instance, which calls out that problem implicitly; and ...

Agreed.

> I suspect
> that issue is also why pg_stat_statements has the
> process_shared_preload_libraries_in_progress check at the top, because
> it looks to me like everything else that the function does would be
> completely fine to do later.

I think a bigger problem is loading and saving the statistics file. The 
file needs to be saved on postmaster exit, where do you do that if the 
library was not in shared_preload_libraries?

> Shared resources do require some split-up of initialization: as you
> point out, if _PG_init() is called before we know MaxBackends, then we
> can't even size data structures who size depends on that quantity yet,
> and we certainly can't initialize anything, because shared memory
> might not have been created yet. I don't think we can completely avoid
> the need for callbacks here, but... just spitballing, how about
> something like this:
> 
> extern void DefineShmemRegion(char *name, size_t size, void
> **localptr, void (*init_callback)(void *), int flags);
> extern void DefineShmemRegionDynamic(char *name, size_t
> (*sizing_callback)(void *), void **localptr, void
> (*init_callback)(void *), int flags);
> extern void *GetShmemRegion(char *name);
> 
> #define DSR_FLAGS_NO_SLOP  0x01
> #define DSR_FLAGS_DSA_OK 0x02
> 
> If DefineShmemRegion() or DefineShmemRegionDynamic() is called at
> shared_preload_libraries time, it arranges to increase the size of the
> main shared memory segment by the given amount, or the computed amount
> (for things that depend on MaxBackends). Then, once the main shared
> memory segment is created, it invokes the init_callback and sets
> *localptr. If either of these functions are called after the main
> shared memory segment has been created, they check for an existing
> allocation, and if one is found, they just set *localptr. (They can
> actually probably exit quickly if *localptr is already set.)

If you squint a little, this is pretty much the same as my descriptor 
design. Let's start from your DefineShmemRegion function, but in order 
to have some flexibility to add optional optional in the future, without 
having to create DefineShmemRegionNew(), DefineShmemRegionExt() or 
similar functions, let's pass the arguments in a struct. So instead of:

DefineShmemRegion("pg_stat_statements", sizeof(pgssSharedState), &pgss, 
&pgss,pgss_shmem_init, 0);

you would call it like this:

DefineShmemRegion(&(ShmemStructDesc) {
     .name = "pg_stat_statements",
     .size = sizeof(pgssSharedState),
     .ptr = (void **) &pgss,
     .init_fn = pgss_shmem_init,
     .flags = 0,
});

This flexibility will come handy as soon as we add the ability to resize.

Now if you rename DefineShmemRegion() to ShmemRegisterStruct(), this is 
equivalent to my descriptor design. (One detail is whether you require 
the descriptor struct to stay around after the call, or if 
DefineShmemRegion/ShmemRegisterStruct will copy all the information)

> Otherwise, they try to allocate from DSA if DSR_FLAGS_DSA_OK is given,
> or else from the slop space unless DSR_FLAGS_NO_SLOP is given.

Ok, falling back to DSA is a new idea. You could implement that in 
either design.

> If that
> works, they then call the init_callback under a suitable lock and set
> *localptr. If not, they fail silently. Functions that need to use the
> local pointers do something like this:
> 
> if (unlikely(pgss == NULL))
>      pgss = GetShmemRegion("pg_stat_statements");
> 
> ...which throws a suitable error -- not just a generic one that the
> region doesn't exist, but something that's sensitive to different
> failure conditions: the region was never registered, the region was
> registered after shared_preload_libraries time and not enough slop
> space remains, or whatever. GetShmemRegion() could even retry the
> initialization in certain cases, e.g. if DSA is OK and we previously
> were called too early in startup to try DSA, we can try now, or if DSA
> allocation failed due to OOM, we can try again.

Ok, we could add GetShmemRegion() in either design. Do we have any place 
where we'd use that though, instead the backend-private pointer global 
variable? I can't think of any examples where we currently call 
ShmemInitStruct() to get a pointer "on demand" like that.

In pg_stat_statements, this would replace these tests:

     if (!pgss || !pgss_hash)
         ereport(ERROR,
                 (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
                  errmsg("pg_stat_statements must be loaded via 
\"shared_preload_libraries\"")));

But I don't think pg_stat_statements could still allocate the region 
after postmaster startup. GetShmemRegion() would just be a different way 
of throwing that error.

> I *think* this design gets rid of all the IsUnderPostmaster and
> shared_preload_libraries_in_progress checks in individual subsystems,
> and all the use of shmem startup hooks. You just ask for what you want
> and if there's a way to get it, the system gives it to you, and if
> there's not, it generates an error at the latest possible time, and
> also tries to self-heal if that's reasonable. If you do load your
> module in shared_preload_libraries, then by the time main shared
> memory initialization completes in the postmaster, everyone's localptr
> values (like pgss) will be initialized, but if it happens to be an
> EXEC_BACKEND build, those same calls will also happen in every
> postmaster child, and will automatically re-find the shared memory
> areas and reinitialize all the pointers. If you load your module
> later, your localptr values will hopefully be initialized by the end
> of _PG_init(), but if that doesn't work out, then the
> unlikely-protected calls to GetShmemRegion will produce suitable
> errors at a suitable time. And I think it all works out nicely in a
> standalone backend, too.
> 
> This is all kind of a brain dump and is not fully thought-through and
> might be riddled with cognitive errors, but what I'm sort of trying to
> convey is where I think the complexity in the current system comes
> from (which is that we require every subsystem/extension author to
> know how the sausage is made, and we don't enforce consistency between
> requests and allocations) and what I don't really like about
> descriptors as a solution (which is that they are harder to read than
> imperative code and can interfere with cases where somebody wants to
> do something slightly different than what the descriptor-designer had
> in mind). I hope that some of it is helpful to you...

Thanks, this hasn't changed my opinions, but I really appreciate 
pressure-testing the design. I don't want to rewrite this again in a 
year, because we didn't get it quite right.

- Heikki

  27. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T15:17:45Z

    On Thu, Mar 19, 2026 at 10:34 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> > I suspect
> > that issue is also why pg_stat_statements has the
> > process_shared_preload_libraries_in_progress check at the top, because
> > it looks to me like everything else that the function does would be
> > completely fine to do later.
>
> I think a bigger problem is loading and saving the statistics file. The
> file needs to be saved on postmaster exit, where do you do that if the
> library was not in shared_preload_libraries?

Well, there's no way to install a hook in the postmaster in that case,
so you can't. But I'm not sure that justifies skipping everything
_PG_init() would have done. A problem with the status quo is that
every module author makes their own decision about how to handle the
s_p_l problem, and they don't all decide differently, even in our own
tree. For example, autoprewarm chooses to register the GUC that it can
while skipping the other one, while pg_stat_statements skips
everything, including hook installation. Maybe that's properly
considered flexibility that should be left to each individual author,
but to me it seems more like happenstance than anything else. I'd
favor a design that emphasizes severability - i.e. you always try to
do as much as possible, and defer errors until later. So you always
create the GUCs but then restrict setting them to values that you
can't support without a restart, instead of not creating them. You
install the hooks and then maybe they will have to no-op out. It's
just weird if you load a library and the GUCs aren't defined and
there's not even a diagnostic telling you why.

> If you squint a little, this is pretty much the same as my descriptor
> design. Let's start from your DefineShmemRegion function, but in order
> to have some flexibility to add optional optional in the future, without
> having to create DefineShmemRegionNew(), DefineShmemRegionExt() or
> similar functions, let's pass the arguments in a struct. So instead of:
>
> DefineShmemRegion("pg_stat_statements", sizeof(pgssSharedState), &pgss,
> &pgss,pgss_shmem_init, 0);
>
> you would call it like this:
>
> DefineShmemRegion(&(ShmemStructDesc) {
>      .name = "pg_stat_statements",
>      .size = sizeof(pgssSharedState),
>      .ptr = (void **) &pgss,
>      .init_fn = pgss_shmem_init,
>      .flags = 0,
> });
>
> This flexibility will come handy as soon as we add the ability to resize.

I see your point. I'm not really convinced, though. In practice,
what's now going to happen is that you're probably going to move that
struct out of _PG_init() and define it elsewhere, so the logic is
getting split between multiple places, which does not improve
readability, and it also becomes much more worrying to what degree the
struct needs to be const, whereas if you just pass a bunch of
parameters by value you kind of understand what has to be happening.
Also, what flexibility have you really purchased? Sure, now you can
add arguments to the function call without breaking existing call
sites, but (1) there are other ways to do that, like by creating an
object first and then using methods to assign properties to it
afterward (i.e. RegisterCallbackOnShmemRegion) and (2) adding
arguments without breaking existing callers is not an unmixed blessing
in the first place. I know the world won't end if you go with this
style, but I guess I'm not much of a fan. I find this sort of thing
hard to read.

> Ok, we could add GetShmemRegion() in either design. Do we have any place
> where we'd use that though, instead the backend-private pointer global
> variable? I can't think of any examples where we currently call
> ShmemInitStruct() to get a pointer "on demand" like that.
>
> In pg_stat_statements, this would replace these tests:
>
>      if (!pgss || !pgss_hash)
>          ereport(ERROR,
>                  (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
>                   errmsg("pg_stat_statements must be loaded via
> \"shared_preload_libraries\"")));
>
> But I don't think pg_stat_statements could still allocate the region
> after postmaster startup. GetShmemRegion() would just be a different way
> of throwing that error.

In that case, yes. But something like autoprewarm only needs a small
amount of shared memory, and can potentially initialize itself on the
fly. The not-yet-committed pg_collect_advice module has similar needs,
which it currently satisfies using GetNamedDSMSegment(); see in
particular pg_collect_advice_attach() if you feel like wandering over
to the pg_plan_advice thread.

> Thanks, this hasn't changed my opinions, but I really appreciate
> pressure-testing the design. I don't want to rewrite this again in a
> year, because we didn't get it quite right.

Yeah, I'm somewhat concerned about an ever-proliferating number of
ways to do things that all sort of suck in different ways.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  28. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-22T00:14:11Z

    On 19/03/2026 17:17, Robert Haas wrote:
> On Thu, Mar 19, 2026 at 10:34 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>>> I suspect
>>> that issue is also why pg_stat_statements has the
>>> process_shared_preload_libraries_in_progress check at the top, because
>>> it looks to me like everything else that the function does would be
>>> completely fine to do later.
>>
>> I think a bigger problem is loading and saving the statistics file. The
>> file needs to be saved on postmaster exit, where do you do that if the
>> library was not in shared_preload_libraries?
> 
> Well, there's no way to install a hook in the postmaster in that case,
> so you can't. But I'm not sure that justifies skipping everything
> _PG_init() would have done. A problem with the status quo is that
> every module author makes their own decision about how to handle the
> s_p_l problem, and they don't all decide differently, even in our own
> tree. For example, autoprewarm chooses to register the GUC that it can
> while skipping the other one, while pg_stat_statements skips
> everything, including hook installation. Maybe that's properly
> considered flexibility that should be left to each individual author,
> but to me it seems more like happenstance than anything else. I'd
> favor a design that emphasizes severability - i.e. you always try to
> do as much as possible, and defer errors until later. So you always
> create the GUCs but then restrict setting them to values that you
> can't support without a restart, instead of not creating them. You
> install the hooks and then maybe they will have to no-op out. It's
> just weird if you load a library and the GUCs aren't defined and
> there's not even a diagnostic telling you why.
> 
>> If you squint a little, this is pretty much the same as my descriptor
>> design. Let's start from your DefineShmemRegion function, but in order
>> to have some flexibility to add optional optional in the future, without
>> having to create DefineShmemRegionNew(), DefineShmemRegionExt() or
>> similar functions, let's pass the arguments in a struct. So instead of:
>>
>> DefineShmemRegion("pg_stat_statements", sizeof(pgssSharedState), &pgss,
>> &pgss,pgss_shmem_init, 0);
>>
>> you would call it like this:
>>
>> DefineShmemRegion(&(ShmemStructDesc) {
>>       .name = "pg_stat_statements",
>>       .size = sizeof(pgssSharedState),
>>       .ptr = (void **) &pgss,
>>       .init_fn = pgss_shmem_init,
>>       .flags = 0,
>> });
>>
>> This flexibility will come handy as soon as we add the ability to resize.
> 
> I see your point. I'm not really convinced, though. In practice,
> what's now going to happen is that you're probably going to move that
> struct out of _PG_init() and define it elsewhere, so the logic is
> getting split between multiple places, which does not improve
> readability, and it also becomes much more worrying to what degree the
> struct needs to be const, whereas if you just pass a bunch of
> parameters by value you kind of understand what has to be happening.
> Also, what flexibility have you really purchased? Sure, now you can
> add arguments to the function call without breaking existing call
> sites, but (1) there are other ways to do that, like by creating an
> object first and then using methods to assign properties to it
> afterward (i.e. RegisterCallbackOnShmemRegion) and (2) adding
> arguments without breaking existing callers is not an unmixed blessing
> in the first place. I know the world won't end if you go with this
> style, but I guess I'm not much of a fan. I find this sort of thing
> hard to read.

I see your point too, the options will indeed easily start to split 
between constant parts and parts that are set later.

To avoid that, perhaps it's best to adopt the same style we use for 
HASHCTL structs:

static ShmemHashDesc pgssSharedHashDesc = { };

pgssSharedHashDesc.name = "pg_stat_statements hash";
pgssSharedHashDesc.ptr = &pgss_hash;
pgssSharedHashDesc.hash_info.keysize = sizeof(pgssHashKey);
pgssSharedHashDesc.hash_info.entrysize = sizeof(pgssEntry);
pgssSharedHashDesc.hash_flags = HASH_ELEM | HASH_BLOBS;
pgssSharedHashDesc.init_size = pgss_max;
pgssSharedHashDesc.max_size = pgss_max;
ShmemRequestHash(&pgssSharedHashDesc);

I don't like the idea of just using the string name as the handle. I 
think it will come handy to have a backend-private descriptor or handle 
to the shared memory region. But perhaps the "name" and "ptr" should be 
regular arguments, for example. If some parameters are passed as 
arguments and some in the struct, that also splits things in an awkward 
way, though.

>> Ok, we could add GetShmemRegion() in either design. Do we have any place
>> where we'd use that though, instead the backend-private pointer global
>> variable? I can't think of any examples where we currently call
>> ShmemInitStruct() to get a pointer "on demand" like that.
>>
>> In pg_stat_statements, this would replace these tests:
>>
>>       if (!pgss || !pgss_hash)
>>           ereport(ERROR,
>>                   (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
>>                    errmsg("pg_stat_statements must be loaded via
>> \"shared_preload_libraries\"")));
>>
>> But I don't think pg_stat_statements could still allocate the region
>> after postmaster startup. GetShmemRegion() would just be a different way
>> of throwing that error.
> 
> In that case, yes. But something like autoprewarm only needs a small
> amount of shared memory, and can potentially initialize itself on the
> fly. The not-yet-committed pg_collect_advice module has similar needs,
> which it currently satisfies using GetNamedDSMSegment(); see in
> particular pg_collect_advice_attach() if you feel like wandering over
> to the pg_plan_advice thread.

I wonder if we should set aside a small amount of memory, like 10-20 kB, 
in the fixed shmem segment for small structs like that. Currently, such 
allocations will usually succeed because we leave some wiggle room, but 
the  can also be consumed by other things like locks. But we could 
reserve a small amount of memory specifically for small allocations in 
extensions like this.

I didn't implement that, but I improved the documentation on how 
after-startup allocations work with the new facility.

Or if we get the feature to resize allocations, we could set aside a few 
pages of *address space* for them without allocating the actual memory 
until it's needed.



Attachd is new version with lots of changes again. I've experimented 
with different ways that the interface could look like, like with the 
"adjust" callback we discussed earlier. This is the version I like best 
so far:

Shmem callbacks
---------------

I separated the request/init/fn callbacks from the structs. There's now 
a concept of "shmem callbacks", which you register in _PG_init(). For 
example:

static void pgss_shmem_request(void *arg);
static void pgss_shmem_init(void *arg);

static const ShmemCallbacks pgss_shmem_callbacks = {
     .request_fn = pgss_shmem_request,
     .init_fn = pgss_shmem_init,
     .attach_fn = NULL, /* no special attach actions needed */
};

void
_PG_init(void)
{
     ...

     RegisterShmemCallbacks(&pgss_shmem_callbacks);
}

This is a similar to the old shmem_request and shmem_startup hooks. It's 
not a one-to-one replacement though, the callbacks are called at 
different stages than the old hooks (to make things convenient with the 
new facility):

* The request_fn callback is called in postmaster startup, at the same 
stage as the old shmem_request callback was. But in EXEC_BACKEND mode, 
it's *also* called in each backend.

* The init_fn callback is only called in postmaster startup, when it's 
time to initialize the area. (Ignoring the special "after startup" 
allocations for now).

Shmem requests
--------------

To register a shmem area, you call ShmemRequestStruct() or 
ShmmeRequestHash() from the request callback function. For example:

static void
pgss_shmem_request(void *arg)
{
     static ShmemHashDesc pgssSharedHashDesc = {
        .name = "pg_stat_statements hash",
         .ptr = &pgss_hash,
         .hash_info.keysize = sizeof(pgssHashKey),
         .hash_info.entrysize = sizeof(pgssEntry),
         .hash_flags = HASH_ELEM | HASH_BLOBS,
     };
     static ShmemStructDesc pgssSharedStateDesc = {
         .name = "pg_stat_statements",
         .size = sizeof(pgssSharedState),
         .ptr = (void **) &pgss,
     };

     pgssSharedHashDesc.init_size = pgss_max;
     pgssSharedHashDesc.max_size = pgss_max;
     ShmemRequestHash(&pgssSharedHashDesc);
     ShmemRequestStruct(&pgssSharedStateDesc);
}

Initialization happens in the init callback, which is called after all 
the pointers (pgss, pgss_hash) have been set.


Built-in subsystems
-------------------

I refactored all the core subsystems to also use these callbacks. The 
built-in callbacks are listed in one big array, see the new 
"subsystemlist.h" header file.

SLRUs
-----

I also refactored the SLRU initialization to use the same principle: You 
call SimpleLruRequest() in the shmem request callback, and it calculates 
and requests the required shmem size for you.


I split this into more incremental patches. The first few are just 
refactorings that are probably useful on their own.

- Heikki

  29. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-24T15:32:00Z

    On Sun, Mar 22, 2026 at 5:44 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
>
> I split this into more incremental patches. The first few are just
> refactorings that are probably useful on their own.

Here are some comments on the patches

0001
+
+# Enable pg_stat_statements to test restart of shared_preload_libraries.
+$node->append_conf(
+ 'postgresql.conf',
+ qq{shared_preload_libraries = 'pg_stat_statements'
+pg_stat_statements.max = 50000
+compute_query_id = 'regress'
+});
+

In order to make sure that the shared memory and LWLocks for
pg_stat_statements are initialized after crash restart, we need to at
least query pg_stat_statements after the restart or do something so
that it's shared memory is used. Also, we can check whether the shared
memory structures are created by querying pg_shmem_allocations after
the restart.

0002
void
InitShmemAllocator(PGShmemHeader *seghdr)
{

The new ShmemIndex initialization code is cleaner and more
straightforward. It avoids the recursive nature of ShmemInitHash.
However with this change it's hard to keep track of all the
initialization steps and their dependencies. Attached is a patch that
makes small adjustments to the code to make it more clear.

Use of variable hash_size is actually misleading since it's not the
size of the hash table but the expected/max number of entries in it.
Removing it makes code more readable.

0003, 0005, 0006 is straight forward, no comments. Usually these
patches make the code more readable. I will review it more when I see
the patches that use this refactoring.

0004 I traced back the placement of CreateLWLock in history. It feels
like the patch is moving it to its intended place since the function
was introduced. It needs a comment to explain why the function is not
called from CreateOrAttachShmemStructs where other in-memory
structures are allocated.

0007 without using new APIs is not necessarily a win. So I would
suggest committing it along with the refactoring patch.

If we are going to commit these patches separately, I would suggest
squashing all predicate.c patches into one commit.

I will continue from 0008 tomorrow.

-- 
Best Wishes,
Ashutosh Bapat

  30. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-25T16:05:03Z

    On Tue, Mar 24, 2026 at 9:02 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
>
> I will continue from 0008 tomorrow.
>

I reviewed the documentation part of 0008. I have a few edits attached.

I have just one comment that's not covered in the edits

@@ -4254,8 +4254,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
Anonymous allocations are allocations that have been made
with <literal>ShmemAlloc()</literal> directly, rather than via
- <literal>ShmemInitStruct()</literal> or
- <literal>ShmemInitHash()</literal>.
+ <literal>ShmemRequestStruct()</literal> or
+ <literal>ShmemRequestHash()</literal>.
</para>

ShmemInitStruct() and ShmemInitHash() are still the functions to
allocate named structures. If we are going to keep ShmemInitStruct()
and ShmemInitHash() around for a while, I think it is more accurate to
mention them in this sentence along with the new functions.

Will continue reviewing the patch tomorrow.

-- 
Best Wishes,
Ashutosh Bapat

  31. Re: Better shared data structure management and resizable shared data structures

    Robert Haas <robertmhaas@gmail.com> — 2026-03-25T18:37:18Z

    On Sat, Mar 21, 2026 at 8:14 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> I see your point too, the options will indeed easily start to split
> between constant parts and parts that are set later.

Yeah. That definitely seems worth trying to fix.

> I don't like the idea of just using the string name as the handle. I
> think it will come handy to have a backend-private descriptor or handle
> to the shared memory region. But perhaps the "name" and "ptr" should be
> regular arguments, for example. If some parameters are passed as
> arguments and some in the struct, that also splits things in an awkward
> way, though.

I agree that having a handle object seems potentially useful. Putting
arguments that are required to be constant in the struct and passing
other things as arguments or in some other way seems like a good idea.

> I wonder if we should set aside a small amount of memory, like 10-20 kB,
> in the fixed shmem segment for small structs like that. Currently, such
> allocations will usually succeed because we leave some wiggle room, but
> the  can also be consumed by other things like locks. But we could
> reserve a small amount of memory specifically for small allocations in
> extensions like this.

Yeah, I don't really understand why we let the lock table use up that
space. I mean, I think it would be useful to have a way to let the
lock table expand without a server restart, and I also suspect that we
could come up with a less-silly data structure than the PROCLOCK hash,
but also if the only thing keeping you from running out of lock space
is the wiggle room, maybe you just need to bump up
max_locks_per_transaction. Like, you could easily burn through the
wiggle room, get an error anyway, and then later find that you also
now can't load certain extensions without a server restart.

> Shmem callbacks
> ---------------
>
> I separated the request/init/fn callbacks from the structs. There's now
> a concept of "shmem callbacks", which you register in _PG_init(). For
> example:
>
> static void pgss_shmem_request(void *arg);
> static void pgss_shmem_init(void *arg);
>
> static const ShmemCallbacks pgss_shmem_callbacks = {
>      .request_fn = pgss_shmem_request,
>      .init_fn = pgss_shmem_init,
>      .attach_fn = NULL, /* no special attach actions needed */
> };

What's the advantage of coupling the functions together this way, vs.
just registering each callback individually?

Also, I don't understand what "arg" is supposed to be doing. It
doesn't seem to be getting used for anything.

I wonder if we should think about just adjusting the timing of the
existing callbacks instead of adding new things. I mean, maybe that's
too likely to cause silent breakage, in which case we could also
replace them with new callbacks with slightly different names. But
there's something to be said for a hard cutover -- or at least not
letting both ways coexist for more than a few releases.

> This is a similar to the old shmem_request and shmem_startup hooks. It's
> not a one-to-one replacement though, the callbacks are called at
> different stages than the old hooks (to make things convenient with the
> new facility):
>
> * The request_fn callback is called in postmaster startup, at the same
> stage as the old shmem_request callback was. But in EXEC_BACKEND mode,
> it's *also* called in each backend.
>
> * The init_fn callback is only called in postmaster startup, when it's
> time to initialize the area. (Ignoring the special "after startup"
> allocations for now).

Trying to improve the timing of the callbacks makes sense to me as a
concept, without taking a position on these particular choices (and
definitely hoping for some after-startup improvements).

Looking at v7-0018 (any chance you have a public branch where I can
look at this without having to download and apply all the patches one
by one?) it seems like this gets rid of the need for a bunch of
IsUnderPostmaster checks in individual subsystems, and some if
(found)/if (!found) checks, which I definitely like.

> Shmem requests
> --------------
>
> To register a shmem area, you call ShmemRequestStruct() or
> ShmmeRequestHash() from the request callback function. For example:
>
> static void
> pgss_shmem_request(void *arg)
> {
>      static ShmemHashDesc pgssSharedHashDesc = {
>         .name = "pg_stat_statements hash",
>          .ptr = &pgss_hash,
>          .hash_info.keysize = sizeof(pgssHashKey),
>          .hash_info.entrysize = sizeof(pgssEntry),
>          .hash_flags = HASH_ELEM | HASH_BLOBS,
>      };
>      static ShmemStructDesc pgssSharedStateDesc = {
>          .name = "pg_stat_statements",
>          .size = sizeof(pgssSharedState),
>          .ptr = (void **) &pgss,
>      };
>
>      pgssSharedHashDesc.init_size = pgss_max;
>      pgssSharedHashDesc.max_size = pgss_max;
>      ShmemRequestHash(&pgssSharedHashDesc);
>      ShmemRequestStruct(&pgssSharedStateDesc);
> }
>
> Initialization happens in the init callback, which is called after all
> the pointers (pgss, pgss_hash) have been set.

I think this is not bad. I suppose it lets you get a complete list of
all of the descriptors someplace. It seems to avoid double-computing
the request size, or any possibility of drift between the requested
size and the allocated size. Having to create the struct and then set
the size afterward in some cases is a tiny bit awkward-looking, but
it's not awful. I do wonder if some coding pattern might creep in
where people create the struct and register it and then try to change
the size, or other fields, afterward. I'm tempted to propose making
the struct non-static and having the registration functions copy it,
so that there can be absolutely no question of getting away with such
antisocial behavior.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  32. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-26T10:10:41Z

    On 24/03/2026 17:32, Ashutosh Bapat wrote:
> 0002
> void
> InitShmemAllocator(PGShmemHeader *seghdr)
> {
> 
> The new ShmemIndex initialization code is cleaner and more
> straightforward. It avoids the recursive nature of ShmemInitHash.
> However with this change it's hard to keep track of all the
> initialization steps and their dependencies. Attached is a patch that
> makes small adjustments to the code to make it more clear.
> 
> Use of variable hash_size is actually misleading since it's not the
> size of the hash table but the expected/max number of entries in it.
> Removing it makes code more readable.

Thanks, committed this 0002 patch with those changes, to get that out of 
the way.

- Heikki

  33. Re: Better shared data structure management and resizable shared data structures

    Daniel Gustafsson <daniel@yesql.se> — 2026-03-26T18:31:12Z

    > On 22 Mar 2026, at 01:14, Heikki Linnakangas <hlinnaka@iki.fi> wrote:

> Attachd is new version with lots of changes again. I've experimented with different ways that the interface could look like, like with the "adjust" callback we discussed earlier.

I had a look at this version today, mainly 0008 and onwards, and I quite like
the API.  It is a bit verbose in places, but the improved readability outweighs
that IMHO.

> * The request_fn callback is called in postmaster startup, at the same stage as the old shmem_request callback was. But in EXEC_BACKEND mode, it's *also* called in each backend.

Should the request_fn be told, via an argument, from where it is called?  It
can be figured out but it's cleaner if all implementations will do it in the
same way.  I don't have a direct case in mind where it would be needed, but I
was recently digging into SSL passphrase reloading which has failure cases
precisely becasue of this so am thinking out loud to avoid similar problems
here.

> static void
> pgss_shmem_request(void *arg)
> {
>    static ShmemHashDesc pgssSharedHashDesc = {
>       .name = "pg_stat_statements hash",
>        .ptr = &pgss_hash,
>        .hash_info.keysize = sizeof(pgssHashKey),
>        .hash_info.entrysize = sizeof(pgssEntry),
>        .hash_flags = HASH_ELEM | HASH_BLOBS,
>    };
>    static ShmemStructDesc pgssSharedStateDesc = {
>        .name = "pg_stat_statements",
>        .size = sizeof(pgssSharedState),
>        .ptr = (void **) &pgss,
>    };
> 
>    pgssSharedHashDesc.init_size = pgss_max;
>    pgssSharedHashDesc.max_size = pgss_max;
>    ShmemRequestHash(&pgssSharedHashDesc);
>    ShmemRequestStruct(&pgssSharedStateDesc);
> }

Roberts suggestion upthread to copy the structure to ensure that changing any
part of the struct after registration isn't causing subtle bugs seem like a
good improvement.

> I split this into more incremental patches. The first few are just refactorings that are probably useful on their own.

Reviewing these I wasn't able to spot any issues, but below are a few comments
on mostly 0008 but also a few others:

0008:
====

+      doesn'' immediately allocate or initialize the memory, it merely

s/doesn''/doesn't/


+      registers the space to be allocated later in the startup sequence.  When
+      the memory is allocated, it is initialized to zero.  To any more complex
+      initialization, set the <function>init_fn()</function> callback, which

A word is missing here, perhaps: s/To any more complex/To perform any more complex/ ?


+      An example of allocating shared memory can be found in
       <filename>contrib/pg_stat_statements/pg_stat_statements.c</filename> in
       the <productname>PostgreSQL</productname> source tree.

While not the fault of this patch, I wonder if directing readers to a 3000 line
C file which is growing increasingly complicated is all that helpful.  Maybe we
should (as a separate piece of work) construct more direct examples/tutorials
for this?


+      shared memory available. The system reserves a somes memory for

s/a somes/some/


+      another backend. The callbacks will be held while holding an internal
+      lock, which prevents concurrent two backends from initializating the

"will be held" reads a bit odd, perhaps "will be called" or "will be executed"?


+ * Nowadays, there is also third way to allocate shared memory called Dynamic

"a third way"


+ * ShmemInitStruct()/ShmemInitHash() is another way of registring shmem areas.

s/registring/registering/


+/*
+ * # of additional entries to reserve in the shmem index table, for allocations
+ * after postmaster startup (not a hard limit)
+ */
+#define SHMEM_INDEX_ADDITIONAL_SIZE         (64)

This comment no longer contains the word "estimated", so the "not a hard limit"
portion is harder to understand.  Does mean one can change the define freely
and recompile without crashes due to wrong number, or can the hash grow
dynamically during runtime?  Both interpretations are quite possible.


+   /*
+    * When we call the shmem_request callbacks, we enter the SB_REQUESTING
+    * phase. All ShmemRequestStruct calls happen in this state.
+    */
+   SB_REQUESTING,

Daft question, but what does the "B" stand for?


+   ShmemInitCallback init_fn;
+   void       *init_fn_arg;

init_fn_arg seems quite useful bu is under-documented, perhaps add something to
xfunc.sgml would be worthwhile? The same can be said for request_fn_arg.


+   /* Check that it's not already registered in this process */
+   foreach(lc, requested_shmem_areas)
+   {
+       ShmemStructDesc *existing = (ShmemStructDesc *) lfirst(lc);
+
+       if (strcmp(existing->name, desc->name) == 0)
+           ereport(ERROR,
+                   (errmsg("shared memory struct \"%s\" is already registered",
+                           desc->name)));
+   }
+
+   requested_shmem_areas = lappend(requested_shmem_areas, desc);

As a side-note, I wish we had a list_append_unique flavour which would invoke a
function pointer instead of just list_member() to avoid boilerplate like this.


+   if (found)
+   {
+       /* Already present, just attach to it */
+       if (!attach_allowed)
+           elog(ERROR, "shared memory struct \"%s\" is already initialized", desc->name);

I guess it depends a lot on the caller, but couldn't it be argued that this
case is a lot like !init_allowed and thus a FATAL?


+/*
+ * ShmemGetRequestedSize() --- estimate the total size of all registered shared
+ *                              memory structures.
+ *
+ * This is called once at postmaster startup, before the shared memory segment
+ * has been created.

It's actually called twice, once for ShmemGUCs as well.</nitpickery>


+           /* out of memory; remove the failed ShmemIndex entry */
+           hash_search(ShmemIndex, desc->name, HASH_REMOVE, NULL);
+           ereport(ERROR,
+                   (errcode(ERRCODE_OUT_OF_MEMORY),
+                    errmsg("not enough shared memory for data structure"
+                           " \"%s\" (%zu bytes requested)",
+                           desc->name, desc->size)));

When shmem_startup_state is set to SB_LATE_ATTACH_OR_INIT, would it be
worthwhile to add an errhint to move allocation to init phase instead?


Various
=====

In the 0014 commit message, s/ProgGlobal/ProcGlobal/


-   if (IsUnderPostmaster && !process_shared_preload_libraries_in_progress)
+   if (shmem_startup_state == SB_DONE && IsUnderPostmaster)

This hunk in 0014 makes an assertion a few lines down pointless as it checks
the same as the if conditional.


- * have been created by initdb, and CLOGShmemInit must have been
+ * have been created by initdb, and CLOGShmemInit must have been XXX

Stray XXX in 0015.


-   ControlFile = palloc_object(ControlFileData);
+   LocalControlFile = palloc_object(ControlFileData);
+   ControlFile = LocalControlFile;

I'm likely missing something obvious but is the LocalControlFile still needed?

--
Daniel Gustafsson

  34. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-27T00:51:22Z

    On 25/03/2026 20:37, Robert Haas wrote:
> On Sat, Mar 21, 2026 at 8:14 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:>> Shmem callbacks
>> ---------------
>>
>> I separated the request/init/fn callbacks from the structs. There's now
>> a concept of "shmem callbacks", which you register in _PG_init(). For
>> example:
>>
>> static void pgss_shmem_request(void *arg);
>> static void pgss_shmem_init(void *arg);
>>
>> static const ShmemCallbacks pgss_shmem_callbacks = {
>>       .request_fn = pgss_shmem_request,
>>       .init_fn = pgss_shmem_init,
>>       .attach_fn = NULL, /* no special attach actions needed */
>> };
> 
> What's the advantage of coupling the functions together this way, vs.
> just registering each callback individually?

One reason is to support allocations after postmaster startup. The 
RegisterShmemCallbacks() call ties together all the resources requested 
by the request_fn callback, with the the init_fn or attach_fn callbacks 
that will later initialize/attach them. The init_fn/attach_fn callbacks 
are called only after *all* the resources requested by the request_fn 
callback have been initialized, and it holds a lock while doing all that.

If the callbacks were registered separately, shmem.c wouldn't know when 
to call the init_fn/attach_fn. There's no problem during postmaster or 
backend startup, because we run all init_fn or attach_fn callbacks in 
the whole system, after requesting all the resources, but after startup, 
you must only call the callbacks related to the newly-requested resources.

Aside from that after-startup allocation issue, though, IMO the 
ShmemCallbacks struct makes it more clear that the callbacks are meant 
to work together on the same resources.

One way to think of this is that all the resources requested by the 
request_fn callback are implicitly part of the same "subsystem", and 
need to be initialized/attached to together. We discussed that before, 
and I still wonder if we should make that concept of a subsystem more 
explicit. If we just renamed ShmemCallbacks to ShmemSubsystem, and give 
each subsystem a name, it'd look like this:

static void pgss_shmem_request(void *arg);
static void pgss_shmem_init(void *arg);

static const ShmemSubsystem pgss_shmem_subsystem = {
     .name = "pg_stat_statements"
     .request_fn = pgss_shmem_request,
     .init_fn = pgss_shmem_init,
     .attach_fn = NULL, /* no special attach actions needed */
};

static void
pgss_shmem_request(void *arg)
{
     ShmemRequestStruct(&pgssSharedStateDesc, &(ShmemRequestStructOpts) {
          /*
           * name is optional in this design, subsystem's name is used if
           * not given
           */
         .name = "pg_stat_statements",
         .size = sizeof(pgssSharedState),
         .ptr = (void **) &pgss,
     });
}

static void
pgss_shmem_init(void *arg)
{
     /* initialize contents of pgss */
     ...
}

void
_PG_init(void)
{
     RegisterShmemSubsystem(&pgss_shmem_subsystem);
}



Thinking how this might work without such a struct, registering the 
callbacks separately, here's an alternative design:

static void pgss_shmem_request(void *arg);
static void pgss_shmem_init(void *arg);

static void
pgss_shmem_request(void *arg)
{
     ShmemRequestStruct(&pgssSharedStateDesc, &(ShmemRequestStructOpts) {
         .name = "pg_stat_statements",
         .size = sizeof(pgssSharedState),
         .ptr = (void **) &pgss,
     });

     ShmemRegisterInitCallback(&pgss_shmem_init);
     /* no attach callback needed, but for illustration: */
     ShmemRegisterInitCallback(&pgss_shmem_attach);
}

static void
pgss_shmem_init(void *arg)
{
     /* initialize contents of pgss */
     ...
}

void
_PG_init(void)
{
     ShmemRegisterRequestCallback(&pgss_shmem_request);
}

In this design, the ShmemRegisterRequestCallback() call still ties 
together all the related resources. All the resources requested in the 
request-callback are initialized together, and the fact that the 
init/attach callbacks are registered within the request callback 
associates them with the resources. This feels a little Rube 
Goldbergian, with one callback registering more callbacks, but would 
also work.

> Also, I don't understand what "arg" is supposed to be doing. It
> doesn't seem to be getting used for anything.

It's an opaque pointer that's passed through to the callbacks. Some 
callers might want to pass extra data to the callback. None of the 
current callers use it, so maybe it's not needed, but it seemed like 
good future-proofing.

>> Shmem requests
>> --------------
>>
>> To register a shmem area, you call ShmemRequestStruct() or
>> ShmmeRequestHash() from the request callback function. For example:
>>
>> static void
>> pgss_shmem_request(void *arg)
>> {
>>       static ShmemHashDesc pgssSharedHashDesc = {
>>          .name = "pg_stat_statements hash",
>>           .ptr = &pgss_hash,
>>           .hash_info.keysize = sizeof(pgssHashKey),
>>           .hash_info.entrysize = sizeof(pgssEntry),
>>           .hash_flags = HASH_ELEM | HASH_BLOBS,
>>       };
>>       static ShmemStructDesc pgssSharedStateDesc = {
>>           .name = "pg_stat_statements",
>>           .size = sizeof(pgssSharedState),
>>           .ptr = (void **) &pgss,
>>       };
>>
>>       pgssSharedHashDesc.init_size = pgss_max;
>>       pgssSharedHashDesc.max_size = pgss_max;
>>       ShmemRequestHash(&pgssSharedHashDesc);
>>       ShmemRequestStruct(&pgssSharedStateDesc);
>> }
>>
>> Initialization happens in the init callback, which is called after all
>> the pointers (pgss, pgss_hash) have been set.
> 
> I think this is not bad. I suppose it lets you get a complete list of
> all of the descriptors someplace. It seems to avoid double-computing
> the request size, or any possibility of drift between the requested
> size and the allocated size. Having to create the struct and then set
> the size afterward in some cases is a tiny bit awkward-looking, but
> it's not awful. I do wonder if some coding pattern might creep in
> where people create the struct and register it and then try to change
> the size, or other fields, afterward. I'm tempted to propose making
> the struct non-static and having the registration functions copy it,
> so that there can be absolutely no question of getting away with such
> antisocial behavior.

Here's another version, where that now looks like this:

static void
pgss_shmem_request(void *arg)
{
     static ShmemHashDesc pgssSharedHashDesc;
     static ShmemStructDesc pgssSharedStateDesc;

     ShmemRequestHash(&pgssSharedHashDesc, &(ShmemRequestHashOpts) {
         .name = "pg_stat_statements hash",
         .ptr = &pgss_hash,
         .init_size = pgss_max,
         .max_size = pgss_max,
         .hash_info.keysize = sizeof(pgssHashKey),
         .hash_info.entrysize = sizeof(pgssEntry),
         .hash_flags = HASH_ELEM | HASH_BLOBS,
     });
     ShmemRequestStruct(&pgssSharedStateDesc, &(ShmemRequestStructOpts) {
         .name = "pg_stat_statements",
         .size = sizeof(pgssSharedState),
         .ptr = (void **) &pgss,
     });
}

Notable differences to that since last version:

I separated the backend-private "handle" and the options structs. So 
ShmemRequestStruct/Hash now takes two arguments:

The first argument is a pointer to the "descriptor", which is a 
backend-private handle for the shared memory area. It's currently not 
very interesting for plain structs because you can't really do anything 
with it, but in the future the handle could be used to resize the area 
for example. Also, one of the later patches in this patch set refactors 
SLRUs to be requested in this fashion too. For SLRUs the handle is 
already useful; SLRUs have always had a backend-private "SlruCtl" handle 
like that.

The second argument is an "options" struct, which is really just 
syntactic sugar to pass multiple arguments, some of which can be left 
empty. The options are now copied, so the struct can be short-lived. 
Alternatively, the arguments could be passed as normal function 
arguments like you suggested earlier. But I quite like this style, 
especially with hash tables and SLRUs which take more options and 
already used structs to pass the options.

There's one annoying problem with that syntax though: pgindent doesn't 
like it and gives errors like this:

Failure in contrib/pg_stat_statements/pg_stat_statements.c: Error@508: 
Unbalanced parens
Warning@516: Extra )
Error@517: Unbalanced parens
Warning@521: Extra )

That would be nice to fix in pgindent in any case but I haven't looked 
into it yet.

Another idea is to use a macro to hide that from pgindent, which would 
make the calls little less verbose anyway:

#define ShmemRequestStruct(desc, ...) ShmemRequestStructWithOpts(desc, 
&(ShmemRequestStructOpts) { __VA_ARGS__ })

Then the call would be simply:

     ShmemRequestStruct(&pgssSharedStateDesc,
         .name = "pg_stat_statements",
         .size = sizeof(pgssSharedState),
         .ptr = (void **) &pgss,
     );


New version attached. I also pushed this to 
https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-8.

- Heikki

  35. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-27T07:01:54Z

    On Wed, Mar 25, 2026 at 9:35 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Mar 24, 2026 at 9:02 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> >
> > I will continue from 0008 tomorrow.
> >
>
> I reviewed the documentation part of 0008. I have a few edits attached.
>
> I have just one comment that's not covered in the edits
>
> @@ -4254,8 +4254,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
> <para>
> Anonymous allocations are allocations that have been made
> with <literal>ShmemAlloc()</literal> directly, rather than via
> - <literal>ShmemInitStruct()</literal> or
> - <literal>ShmemInitHash()</literal>.
> + <literal>ShmemRequestStruct()</literal> or
> + <literal>ShmemRequestHash()</literal>.
> </para>
>
> ShmemInitStruct() and ShmemInitHash() are still the functions to
> allocate named structures. If we are going to keep ShmemInitStruct()
> and ShmemInitHash() around for a while, I think it is more accurate to
> mention them in this sentence along with the new functions.
>
> Will continue reviewing the patch tomorrow.
>

Here's a complete review of 0008 (from version 8). I see you have
already posted v9 which I have not looked at. Please feel free to
ignore comments which aren't applicable anymore. Attached patch has
minor edits and some code arrangement to make it more readable as
mentioned in the comments below. Please incorporate those changes as
applicable if you find them useful.

+ The <function>ShmemRequestStruct()</function> can also be called after
+ system startup, which is useful to allow small allocations in add-in
+ libraries that are not specified in
+ <xref linkend="guc-shared-preload-libraries"/><indexterm><primary>shared_preload_libraries</primary></indexterm>.
+ However, after startup the allocation can fail if there is not enough
+ shared memory available. The system reserves a somes memory for
+ allocations after startup, but that reservation is small.

Do we check the request against the reserved amount or overall memory
available? If later, a large request after startup can cause memory
allocated for hash tables to be taken away. This was a problem in the
previous implementation as well, since ShmemAlloc() could be called
after startup. But giving a formal API to do it might encourage more
usage, so it would be good to have some checks in place.

/* Restore basic shared memory pointers */
if (UsedShmemSegAddr != NULL)
+ {
InitShmemAllocator(UsedShmemSegAddr);
+ ShmemCallRequestCallbacks();

It's not clear how we keep the list of registered callbacks across the
backends and also after restart in-sync. How do we make sure that the
callbacks registered at this time are the same callbacks registered
before creating the shared memory? How do we make sure that the
callbacks registered after the startup are also registered after
restart?

- /* re-create shared memory and semaphores */
+ /*
+ * Re-initialize shared memory and semaphores. Note: We don't call
+ * RegisterShmemStructs() here, we keep the old registrations. In

There is no RegisterShmemStructs(). Probably this comment is not reuquired.

+ * This module provides facilities to allocate fixed-size structures in shared
+ * memory, for things like variables shared between all backend processes.
+ * Each such structure has a string name to identify it, specified in the
+ * descriptor when it is requested. shmem_hash.c provides a shared hash table
+ * implementation on top of that.

This wording works well for resizable structures. Thanks.

+ * Shared memory managed by shmem.c can never be freed, once allocated. Each
+ * hash table has its own free list, so hash buckets can be reused when an
+ * item is deleted. However, if one hash table grows very large and then
+ * shrinks, its space cannot be redistributed to other tables. We could build
+ * a simple hash bucket garbage collector if need be. Right now, it seems
+ * unnecessary.

The second sentence onwards belong to shmem_hash.c. Don't they?

+} shmem_startup_state;

This isn't just startup state since the backend can toggle between
DONE and LATE_ATTACH_OR_INIT states after the startup. Probably
"shmem_state" would be a better name.

Also, it might be better to separate the enum and the variable
declaration. I was confused for a moment.

What does B stand for in the enum values?

+static bool AttachOrInit(ShmemStructDesc *desc, bool init_allowed,
bool attach_allowed);

Init in the name can easily lead into thinking that the function is
going to invoke the init callback. I think a better name would be
AttachOrAllocate() or something which can not be confused with Init.

+/*
+ * ShmemRequestStruct() --- request a named shared memory area
+ *
+ * Subsystems call this to register their shared memory needs. This is
+ * usually done early in postmaster startup, before the shared memory segment
+ * has been created, so that the size can be included in the estimate for
+ * total amount of shared memory needed. We set aside a small amount of
+ * memory for allocations that happen later, for the benefit of non-preloaded
+ * extensions, but that should not be relied upon.

I don't think we need to reiterate the last sentence here, since it's
already mentioned in the "Usage" section of the documentation and this
API is unrelated to that.

+ * Attach to all the requested memory areas.
+ */
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+ while (!dclist_is_empty(&requested_shmem_areas))
+ {
+ requested_shmem_area *area = dlist_container(requested_shmem_area, node,
+ dclist_pop_head_node(&requested_shmem_areas));

Isn't requested_shmem_areas a List*? Why do we need to pop nodes from it?

+ ShmemStructDesc *desc = area->desc;
+
+ AttachOrInit(desc, false, true);
+ }
+ list_free(requested_shmem_areas);
+ requested_shmem_areas = NIL;

If we pop all the nodes from the list, then the list should be NIL
right? Why do we need to free it?

+ else if (!init_allowed)
+ {

For the sake of documentation and sanity, I would add
Assert(!index_entry) here, possibly with a comment. Otherwise it feels
like we might be leaving a half-initialized entry in the hash table.

What if attach_allowed is false and the entry is not found? Should we
throw an error in that case too? It would be foolish to call
AttachOrInit with both init_allowed and attach_allowed set to false,
but the API allows it and we should check for that.

It feels like we should do something about the arguments. The function
is hard to read. init_allowed is actually the action the caller wants
to take if the entry is not found, and attach_allowed is the action
the caller wants to take if the entry is found.

Also explain in the comment what does attach mean here especially in
case of fixed sized structures.

Restructuring the code as attached reads better to me.

+/*
+ * Reset state on postmaster crash restart.
+ */
+void
+ResetShmemAllocator(void)
+{

I still think this requires a different name since it's not undoing
what InitShmemAllocator() did. Maybe ResetShmemState()?

+void
+RegisterShmemCallbacks(const ShmemCallbacks *callbacks)
... snip ...
+ foreach(lc, requested_shmem_areas)

Doesn't this list contain all the areas, not just registered in this
instance of the call. Does that mean that we need to have all the
attach functions idempotent? Why can't we deal with the newly
registered areas only?

+ * FIXME: What to do if multiple shmem areas were requested, and some
+ * of them are already initialized but not all?
*/

I doubt if we want to allow attaching to areas which are already
attached since the attach_fn may not be idempotent.
/* Initialize the hash header, plus a copy of the table name */
+ Assert(tabname != NULL);
+ Assert(CurrentDynaHashCxt != NULL);

This looks like a separate patch and separate commit.

+
+ /*
+ * Extra space to reserve in the shared memory segment, but it's not part
+ * of the struct itself. This is used for shared memory hash tables that
+ * can grow beyond the initial size when more buckets are allocated.
+ */
+ size_t extra_size;

When we introduce resizable structures (where even the hash table
directly itself could be resizable), we will introduce a new field
max_size which is easy to get confused with extra_size. Maybe we can
rename extra_size to something like "auxilliary_size" to mean size of
the auxiliary parts of the structure which are not part of the main
struct itself.

+ /*
+ * max_size is the estimated maximum number of hashtable entries. This is
+ * not a hard limit, but the access efficiency will degrade if it is
+ * exceeded substantially (since it's used to compute directory size and
+ * the hash table buckets will get overfull).
+ */
+ size_t max_size;
+
+ /*
+ * init_size is the number of hashtable entries to preallocate. For a
+ * table whose maximum size is certain, this should be equal to max_size;
+ * that ensures that no run-time out-of-shared-memory failures can occur.
+ */
+ size_t init_size;

Everytime I look at these two fields, I question whether those are the
number of entries (i.e. size of the hash table) or number of bytes
(size of the memory). I know it's the former, but it indicates that
something needs to be changed here, like changing the names to have
_entries instead of _size, or changing the type to int64 or some such.
Renaming to _entries would conflict with dynahash APIs since they use
_size, so maybe the latter?

+
+/*
+ * Shared memory is reserved and allocated in stages at postmaster startup,
+ * and in EXEC_BACKEND mode, there's some extra work done to "attach" to them

The comma after EXEC_BACKEND mode is a bit confusing. It makes me
think that the clause after the comma is detached from the
EXEC_BACKEND mode. Maybe revise as
"Shared memory is reserved and allocated to various shared memory
structures in stages at postmaster startup. In EXEC_BACKEND mode,
there's some extra work done to "attach" to them at backend startup.
ShmemCallbacks holds callback functions that are called at different
stages."

+ * at backend startup. ShmemCallbacks holds callback functions that are
+ * called at different stages.
+ */
+typedef struct ShmemCallbacks
+{
+ /* SHMEM_* flags */
+ int flags;

I think we should define the flags before this. Also SHMEM_ looks too
generic prefix, maybe SHMEM_CALLBACKS_ or SHMEM_CB_. With those
changes it will look something like the attached patch.

@@ -50,7 +50,6 @@ static InjIoErrorState *inj_io_error_state;
static shmem_request_hook_type prev_shmem_request_hook = NULL;
static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
-

It's good to get rid of an extra line, but maybe a separate commit.

-- 
Best Wishes,
Ashutosh Bapat

  36. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-27T08:58:05Z

    Hi Heikki,

A correction in my previous email.

On Fri, Mar 27, 2026 at 12:31 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
>
> Here's a complete review of 0008 (from version 8).

This should be 0008 from version 7. I haven't looked at v8 yet.

-- 
Best Wishes,
Ashutosh Bapat

  37. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-27T23:17:13Z

    Thanks, will incorporate your comments in next version. Replying to just 
a few of them here:

On 27/03/2026 09:01, Ashutosh Bapat wrote:
> /* Restore basic shared memory pointers */
> if (UsedShmemSegAddr != NULL)
> + {
> InitShmemAllocator(UsedShmemSegAddr);
> + ShmemCallRequestCallbacks();
> 
> It's not clear how we keep the list of registered callbacks across the
> backends and also after restart in-sync. How do we make sure that the
> callbacks registered at this time are the same callbacks registered
> before creating the shared memory? How do we make sure that the
> callbacks registered after the startup are also registered after
> restart?

On Unix systems, the registered callbacks are inherited by fork(), and 
also survive over crash restart. With EXEC_BACKEND, the assumption is 
that calling a library's _PG_init() function will register the same 
callbacks every time. We make the same assumption today with the 
shmem_startup hook.

> +void
> +RegisterShmemCallbacks(const ShmemCallbacks *callbacks)
> ... snip ...
> + foreach(lc, requested_shmem_areas)
> 
> Doesn't this list contain all the areas, not just registered in this
> instance of the call. Does that mean that we need to have all the
> attach functions idempotent? Why can't we deal with the newly
> registered areas only?

registered_shmem_areas is supposed to be empty when the function is 
entered. There's an assertion for that too before the foreach().

However, it's missing this, after processing the list:

         list_free_deep(requested_shmem_areas);
         requested_shmem_areas = NIL;

Because of that, this will fail if you load multiple extensions that 
call RegisterShmemCallbacks() in the same session. Will fix that.

> + /*
> + * Extra space to reserve in the shared memory segment, but it's not part
> + * of the struct itself. This is used for shared memory hash tables that
> + * can grow beyond the initial size when more buckets are allocated.
> + */
> + size_t extra_size;
> 
> When we introduce resizable structures (where even the hash table
> directly itself could be resizable), we will introduce a new field
> max_size which is easy to get confused with extra_size. Maybe we can
> rename extra_size to something like "auxilliary_size" to mean size of
> the auxiliary parts of the structure which are not part of the main
> struct itself.
> 
> + /*
> + * max_size is the estimated maximum number of hashtable entries. This is
> + * not a hard limit, but the access efficiency will degrade if it is
> + * exceeded substantially (since it's used to compute directory size and
> + * the hash table buckets will get overfull).
> + */
> + size_t max_size;
> +
> + /*
> + * init_size is the number of hashtable entries to preallocate. For a
> + * table whose maximum size is certain, this should be equal to max_size;
> + * that ensures that no run-time out-of-shared-memory failures can occur.
> + */
> + size_t init_size;
> 
> Everytime I look at these two fields, I question whether those are the
> number of entries (i.e. size of the hash table) or number of bytes
> (size of the memory). I know it's the former, but it indicates that
> something needs to be changed here, like changing the names to have
> _entries instead of _size, or changing the type to int64 or some such.
> Renaming to _entries would conflict with dynahash APIs since they use
> _size, so maybe the latter?

Agreed.

- Heikki

  38. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-30T04:50:22Z

    On Sat, Mar 28, 2026 at 4:47 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> Thanks, will incorporate your comments in next version. Replying to just
> a few of them here:
>
> On 27/03/2026 09:01, Ashutosh Bapat wrote:
> > /* Restore basic shared memory pointers */
> > if (UsedShmemSegAddr != NULL)
> > + {
> > InitShmemAllocator(UsedShmemSegAddr);
> > + ShmemCallRequestCallbacks();
> >
> > It's not clear how we keep the list of registered callbacks across the
> > backends and also after restart in-sync. How do we make sure that the
> > callbacks registered at this time are the same callbacks registered
> > before creating the shared memory? How do we make sure that the
> > callbacks registered after the startup are also registered after
> > restart?
>
> On Unix systems, the registered callbacks are inherited by fork(), and
> also survive over crash restart. With EXEC_BACKEND, the assumption is
> that calling a library's _PG_init() function will register the same
> callbacks every time. We make the same assumption today with the
> shmem_startup hook.
>

RegisterShmemCallbacks() may be called after the startup, and it will
add new areas to the shared memory. How are those registries synced
across the backends? From your answer below, those registries are not
synced across backends. They will be wiped out by the restart and
won't be registered again. Is that right? I think we need to document
this fact and also the need to call RegisterShmemCallbacks() from all
the backends where the new areas are required after the startup.

Sorry, my question was not complete.

> > +void
> > +RegisterShmemCallbacks(const ShmemCallbacks *callbacks)
> > ... snip ...
> > + foreach(lc, requested_shmem_areas)
> >
> > Doesn't this list contain all the areas, not just registered in this
> > instance of the call. Does that mean that we need to have all the
> > attach functions idempotent? Why can't we deal with the newly
> > registered areas only?
>
> registered_shmem_areas is supposed to be empty when the function is
> entered. There's an assertion for that too before the foreach().
>
> However, it's missing this, after processing the list:
>
>          list_free_deep(requested_shmem_areas);
>          requested_shmem_areas = NIL;
>
> Because of that, this will fail if you load multiple extensions that
> call RegisterShmemCallbacks() in the same session. Will fix that.


--
Best Wishes,
Ashutosh Bapat

  39. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-03-30T12:20:52Z

    On Fri, Mar 27, 2026 at 12:31 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Wed, Mar 25, 2026 at 9:35 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Tue, Mar 24, 2026 at 9:02 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > >
> > > I will continue from 0008 tomorrow.
> > >
> >
> > I reviewed the documentation part of 0008. I have a few edits attached.
> >
> > I have just one comment that's not covered in the edits
> >
> > @@ -4254,8 +4254,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
> > <para>
> > Anonymous allocations are allocations that have been made
> > with <literal>ShmemAlloc()</literal> directly, rather than via
> > - <literal>ShmemInitStruct()</literal> or
> > - <literal>ShmemInitHash()</literal>.
> > + <literal>ShmemRequestStruct()</literal> or
> > + <literal>ShmemRequestHash()</literal>.
> > </para>
> >
> > ShmemInitStruct() and ShmemInitHash() are still the functions to
> > allocate named structures. If we are going to keep ShmemInitStruct()
> > and ShmemInitHash() around for a while, I think it is more accurate to
> > mention them in this sentence along with the new functions.
> >
> > Will continue reviewing the patch tomorrow.
> >
>
> Here's a complete review of 0008 (from version 7). I see you have
> already posted v8 which I have not looked at.

I rebased my resizable shared memory structures patch on top of your
v8 patch to check if newer APIs are still useful for resizable
structures. Attached is the resultant WIP patch. I will rebase and
finalize it once these patches are committed.

Here are some review comments coming out of that exercise.

The patch subtly changes what allocated_size means in ShmemIndexEntry.
Without this patch, the next structure started from entry->location +
allocated_bytes whereas now the given structure starts at a type
aligned address whereas the next structure may start anywhere after
size bytes from the type aligned address. I think these patches have
got it right (current head the code gets it correct since it uses the
same alignment for all the structures.). But I think the comments for
allocated_size should make it clear that it's not the size available
to the structure, as it used to be. The current comments and the code
in HEAD may make one think so.

What if the caller changes the ShmemStructDesc, which seems to be the
handle we are talking about upthread? Should we make it opaque so that
the callers can not play with it?

-- 
Best Wishes,
Ashutosh Bapat

  40. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-03-30T20:15:44Z

    On 30/03/2026 07:50, Ashutosh Bapat wrote:
> On Sat, Mar 28, 2026 at 4:47 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> On 27/03/2026 09:01, Ashutosh Bapat wrote:
>>> /* Restore basic shared memory pointers */
>>> if (UsedShmemSegAddr != NULL)
>>> + {
>>> InitShmemAllocator(UsedShmemSegAddr);
>>> + ShmemCallRequestCallbacks();
>>>
>>> It's not clear how we keep the list of registered callbacks across the
>>> backends and also after restart in-sync. How do we make sure that the
>>> callbacks registered at this time are the same callbacks registered
>>> before creating the shared memory? How do we make sure that the
>>> callbacks registered after the startup are also registered after
>>> restart?
>>
>> On Unix systems, the registered callbacks are inherited by fork(), and
>> also survive over crash restart. With EXEC_BACKEND, the assumption is
>> that calling a library's _PG_init() function will register the same
>> callbacks every time. We make the same assumption today with the
>> shmem_startup hook.
> 
> RegisterShmemCallbacks() may be called after the startup, and it will
> add new areas to the shared memory. How are those registries synced
> across the backends? From your answer below, those registries are not
> synced across backends. They will be wiped out by the restart and
> won't be registered again. Is that right? I think we need to document
> this fact and also the need to call RegisterShmemCallbacks() from all
> the backends where the new areas are required after the startup.

Correct. Ok, I'll add a note to comment on RegisterShmemCallbacks() to 
call that out more explicitly, hope it helps.

- Heikki

  41. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-01T11:59:13Z

    On Tue, Mar 31, 2026 at 1:45 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 30/03/2026 07:50, Ashutosh Bapat wrote:
> > On Sat, Mar 28, 2026 at 4:47 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >> On 27/03/2026 09:01, Ashutosh Bapat wrote:
> >>> /* Restore basic shared memory pointers */
> >>> if (UsedShmemSegAddr != NULL)
> >>> + {
> >>> InitShmemAllocator(UsedShmemSegAddr);
> >>> + ShmemCallRequestCallbacks();
> >>>
> >>> It's not clear how we keep the list of registered callbacks across the
> >>> backends and also after restart in-sync. How do we make sure that the
> >>> callbacks registered at this time are the same callbacks registered
> >>> before creating the shared memory? How do we make sure that the
> >>> callbacks registered after the startup are also registered after
> >>> restart?
> >>
> >> On Unix systems, the registered callbacks are inherited by fork(), and
> >> also survive over crash restart. With EXEC_BACKEND, the assumption is
> >> that calling a library's _PG_init() function will register the same
> >> callbacks every time. We make the same assumption today with the
> >> shmem_startup hook.
> >
> > RegisterShmemCallbacks() may be called after the startup, and it will
> > add new areas to the shared memory. How are those registries synced
> > across the backends? From your answer below, those registries are not
> > synced across backends. They will be wiped out by the restart and
> > won't be registered again. Is that right? I think we need to document
> > this fact and also the need to call RegisterShmemCallbacks() from all
> > the backends where the new areas are required after the startup.
>
> Correct. Ok, I'll add a note to comment on RegisterShmemCallbacks() to
> call that out more explicitly, hope it helps.
>
> - Heikki
>

Continuing review starting
0007
-------
Subject: [PATCH v8 07/16] Add test module to test after-startup shmem
allocations

I like the idea.

+ *
+ * XXX This module provides interface functions for C functionality to SQL, to
+ * make it possible to test AIO related behavior in a targeted way from SQL.
+ * It'd not generally be safe to export these functions to SQL, but for a test
+ * that's fine.

This mentions test_aio - needs to be rewritten for test_shmem.

+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/shmem.h"

I don't think we need access/relation.h. Others seem ok, but I haven't checked.

In order to better test the difference between EXEC_BACKEND and
non-EXEC_BACKEND builds, please consider incorporating the attached
patch v8-0009-edits.diff

0008
------
- LWLockRelease(AddinShmemInitLock);
+ /* The hash table must be initialized already */
+ Assert(pgss_hash != NULL);

Does it make sense to also Assert(pgss)? A broader question is do we
want to make it a pattern that every user of ShmemRequest*() also
Assert()s that the pointer is non-NULL in the init callback? It is a
test that the ShmemRequest*(), which is far from, init_fn is working
correctly.

/*
- * If we're in the postmaster (or a standalone backend...), set up a shmem
- * exit hook to dump the statistics to disk.
+ * Set up a shmem exit hook to dump the statistics to disk on postmaster
+ * (or standalone backend) exit.
*/
- if (!IsUnderPostmaster)
- on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
-
- /*
- * Done if some other process already completed our initialization.
- */
- if (found)
- return;
+ on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
Given that the structures are registered only at the startup, this
function will be called only from Postmaster, but given that the
structures can be registered and initialized after startup in any
backend, it's better to at least Assert(!IsUnderPostmaster) at the
beginning of this function. The code below is not expected to be
called in any backend too. So Assert(IsUnderPostmaster) at the
beginning of the function would be good safety catch too.

/*
+ * Load any pre-existing statistics from file.
+ *
* Note: we don't bother with locks here, because there should be no other
* processes running when this code is reached.
*/

I was a bit worried that the code next to read stat files is being
crammed in init_fn, but given that the contents of the files are used
to initialize the shared hash table, I think this is fine.

0009
-------
+void
+RegisterBuiltinShmemCallbacks(void)
+{
+ const ShmemCallbacks *builtin_subsystems[] = {
+#define PG_SHMEM_SUBSYSTEM(subsystem_callbacks) &subsystem_callbacks,
+#include "storage/subsystemlist.h"
+#undef PG_SHMEM_SUBSYSTEM
+ };
+
+ for (int i = 0; i < lengthof(builtin_subsystems); i++)
+ RegisterShmemCallbacks(builtin_subsystems[i]);
+}
+

I don't think we need to use a separate array here, we can just call
RegisterShmemCallbacks() directly in the macro as attached.

0011
------
+ InjectionPointAttach("aio-process-completion-before-shared",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("aio-process-completion-before-shared");
+
+ InjectionPointAttach("aio-worker-after-reopen",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("aio-worker-after-reopen");

Attaching and loading an injection point shouldn't be part of the
shared memory initialization. It doens't feel like it should be part
of shmem_startup_hook as well. So not a fault of this patch. I am
wondering why can't it be done in the tests themselves?

0012
------
@@ -663,6 +663,8 @@ SubPostmasterMain(int argc, char *argv[])
*/
LocalProcessControlFile(false);
+ RegisterBuiltinShmemCallbacks();
+

Shouldn't this be part of the previous patch?

-void
-InitProcGlobal(void)
+static void
+ProcGlobalShmemInit(void *arg)
{

I have reviewed most of this patch in earlier versions of this
patchset except this part, which is better than its last version.

Will continue to review the rest of the patches tomorrow.

-- 
Best Wishes,
Ashutosh Bapat

  42. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-01T18:17:12Z

    Yet another version attached (also available at: 
https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-9). The 
main change is the shape of the ShmemRequest*() calls:

On 27/03/2026 02:51, Heikki Linnakangas wrote:
> Another idea is to use a macro to hide that from pgindent, which would 
> make the calls little less verbose anyway:
> 
> #define ShmemRequestStruct(desc, ...) ShmemRequestStructWithOpts(desc, 
> &(ShmemRequestStructOpts) { __VA_ARGS__ })
> 
> Then the call would be simply:
> 
>      ShmemRequestStruct(&pgssSharedStateDesc,
>          .name = "pg_stat_statements",
>          .size = sizeof(pgssSharedState),
>          .ptr = (void **) &pgss,
>      );

I went with that approach. We're already doing something similar with 
XL_ROUTINE in xlogreader.h:

#define XL_ROUTINE(...) &(XLogReaderRoutine){__VA_ARGS__}

The calls look like this:

xlogreader =
     XLogReaderAllocate(wal_segment_size, NULL,
                        XL_ROUTINE(.page_read = &XLogPageRead,
                                   .segment_open = NULL,
                                   .segment_close = wal_segment_close),
                        private);

If we followed that example, ShmemRequestStruct() calls would look like 
this:

ShmemRequestStruct(&pgssSharedStateDesc,
                    SHMEM_STRUCT_OPTS(.name = "pg_stat_statements",
                                      .size = sizeof(pgssSharedState),
                                      .ptr = (void **) &pgss,
     );

However, I don't like the deep indentation, it feels like the important 
stuff is buried to the right. And pgindent insists on that. So I went 
with the proposal I quoted above, turning ShmemRequestStruct(...) itself 
into a macro. If you need more complex options setup, you can set up the 
struct without the macro and call ShmemRequestStructWithOpts() directly, 
but so far all of the callers can use the macro.


Ashutosh, I think I've addressed most of your comments so far. I'm 
replying to just a few of them here that might need more discussion:

> 
> +} shmem_startup_state;
> 
> This isn't just startup state since the backend can toggle between
> DONE and LATE_ATTACH_OR_INIT states after the startup. Probably
> "shmem_state" would be a better name.

Renamed to "shmem_request_state". And renamed "LATE_ATTACH_OR_INIT" to 
"AFTER_STARTUP_ATTACH_OR_INIT" to match the terminology I used elsewhere.

I'm still not entirely happy with this state machine. It seems useful to 
have it for sanity checking, but it still feels a little unclear what 
state you're in at different points in the code, and as an aesthetic 
thing, the whole enum feels too prominent given that it's just for 
sanity checks.

> + ShmemStructDesc *desc = area->desc;
> +
> + AttachOrInit(desc, false, true);
> + }
> + list_free(requested_shmem_areas);
> + requested_shmem_areas = NIL;
> 
> If we pop all the nodes from the list, then the list should be NIL
> right? Why do we need to free it?
> 
> + else if (!init_allowed)
> + {
> 
> For the sake of documentation and sanity, I would add
> Assert(!index_entry) here, possibly with a comment. Otherwise it feels
> like we might be leaving a half-initialized entry in the hash table.
> 
> What if attach_allowed is false and the entry is not found? Should we
> throw an error in that case too? It would be foolish to call
> AttachOrInit with both init_allowed and attach_allowed set to false,
> but the API allows it and we should check for that.
> 
> It feels like we should do something about the arguments. The function
> is hard to read. init_allowed is actually the action the caller wants
> to take if the entry is not found, and attach_allowed is the action
> the caller wants to take if the entry is found.
> 
> Also explain in the comment what does attach mean here especially in
> case of fixed sized structures.

I renamed it to AttachOrInitShmemIndexEntry, and the args to 'may_init' 
and 'may_attach'. But more importantly I added comments to explain the 
different usages. Hope that helps..

On 01/04/2026 14:59, Ashutosh Bapat wrote:
> 0008
> ------
> - LWLockRelease(AddinShmemInitLock);
> + /* The hash table must be initialized already */
> + Assert(pgss_hash != NULL);
> 
> Does it make sense to also Assert(pgss)? A broader question is do we
> want to make it a pattern that every user of ShmemRequest*() also
> Assert()s that the pointer is non-NULL in the init callback? It is a
> test that the ShmemRequest*(), which is far from, init_fn is working
> correctly.

The function does a lot of accesses of 'pgss' so if that's NULL you'll 
get a crash pretty quickly. I'm not sure if the Assert(pgss_hash != 
NULL) is really needed either, but I'm inclined to keep it, as pgss_hash 
might not otherwise be accessed in the function, and there are runtime 
checks for it in the other functions, so if it's not initialized for 
some reason, things might still appear to work to some extent. I don't 
think I want to have that as a broader pattern though.

> + /*
> + * Extra space to reserve in the shared memory segment, but it's not part
> + * of the struct itself. This is used for shared memory hash tables that
> + * can grow beyond the initial size when more buckets are allocated.
> + */
> + size_t extra_size;
> 
> When we introduce resizable structures (where even the hash table
> directly itself could be resizable), we will introduce a new field
> max_size which is easy to get confused with extra_size. Maybe we can
> rename extra_size to something like "auxilliary_size" to mean size of
> the auxiliary parts of the structure which are not part of the main
> struct itself.
> 
> + /*
> + * max_size is the estimated maximum number of hashtable entries. This is
> + * not a hard limit, but the access efficiency will degrade if it is
> + * exceeded substantially (since it's used to compute directory size and
> + * the hash table buckets will get overfull).
> + */
> + size_t max_size;
> +
> + /*
> + * init_size is the number of hashtable entries to preallocate. For a
> + * table whose maximum size is certain, this should be equal to max_size;
> + * that ensures that no run-time out-of-shared-memory failures can occur.
> + */
> + size_t init_size;
> 
> Everytime I look at these two fields, I question whether those are the
> number of entries (i.e. size of the hash table) or number of bytes
> (size of the memory). I know it's the former, but it indicates that
> something needs to be changed here, like changing the names to have
> _entries instead of _size, or changing the type to int64 or some such.
> Renaming to _entries would conflict with dynahash APIs since they use
> _size, so maybe the latter?

I hear you, but I didn't change these yet. If we go with the patches 
from the "Shared hash table allocations" thread, max_size and init_size 
will be merged into one. I'll try to settle that thread before making 
changes here.

> /*
> - * If we're in the postmaster (or a standalone backend...), set up a shmem
> - * exit hook to dump the statistics to disk.
> + * Set up a shmem exit hook to dump the statistics to disk on postmaster
> + * (or standalone backend) exit.
> */
> - if (!IsUnderPostmaster)
> - on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
> -
> - /*
> - * Done if some other process already completed our initialization.
> - */
> - if (found)
> - return;
> + on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
> Given that the structures are registered only at the startup, this
> function will be called only from Postmaster, but given that the
> structures can be registered and initialized after startup in any
> backend, it's better to at least Assert(!IsUnderPostmaster) at the
> beginning of this function. The code below is not expected to be
> called in any backend too. So Assert(IsUnderPostmaster) at the
> beginning of the function would be good safety catch too.

Ok, added an assertion.

> /*
> + * Load any pre-existing statistics from file.
> + *
> * Note: we don't bother with locks here, because there should be no other
> * processes running when this code is reached.
> */
> 
> I was a bit worried that the code next to read stat files is being
> crammed in init_fn, but given that the contents of the files are used
> to initialize the shared hash table, I think this is fine.

Yeah, I went through that train of thought too. Loading the file into 
the hash table is a kind of initialization.

> 0009
> -------
> +void
> +RegisterBuiltinShmemCallbacks(void)
> +{
> + const ShmemCallbacks *builtin_subsystems[] = {
> +#define PG_SHMEM_SUBSYSTEM(subsystem_callbacks) &subsystem_callbacks,
> +#include "storage/subsystemlist.h"
> +#undef PG_SHMEM_SUBSYSTEM
> + };
> +
> + for (int i = 0; i < lengthof(builtin_subsystems); i++)
> + RegisterShmemCallbacks(builtin_subsystems[i]);
> +}
> +
> 
> I don't think we need to use a separate array here, we can just call
> RegisterShmemCallbacks() directly in the macro as attached.

Ah, clever.

> 0011
> ------
> + InjectionPointAttach("aio-process-completion-before-shared",
> + "test_aio",
> + "inj_io_short_read",
> + NULL,
> + 0);
> + InjectionPointLoad("aio-process-completion-before-shared");
> +
> + InjectionPointAttach("aio-worker-after-reopen",
> + "test_aio",
> + "inj_io_reopen",
> + NULL,
> + 0);
> + InjectionPointLoad("aio-worker-after-reopen");
> 
> Attaching and loading an injection point shouldn't be part of the
> shared memory initialization. It doens't feel like it should be part
> of shmem_startup_hook as well. So not a fault of this patch. I am
> wondering why can't it be done in the tests themselves?

I think it's the same reason that's explained in the comment in 
test_aio_shmem_attach():

> 	/*
> 	 * Pre-load the injection points now, so we can call them in a critical
> 	 * section.
> 	 */
> #ifdef USE_INJECTION_POINTS
> 	InjectionPointLoad("aio-process-completion-before-shared");
> 	InjectionPointLoad("aio-worker-after-reopen");
> 	elog(LOG, "injection point loaded");
> #endif

> -void
> -InitProcGlobal(void)
> +static void
> +ProcGlobalShmemInit(void *arg)
> {

I'm not sure what you meant to say here, but I did notice that there 
were a bunch of references to InitProcGlobal() left over in comments. 
Fixed those.

- Heikki

  43. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-02T06:58:12Z

    On Wed, Apr 1, 2026 at 11:47 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> Yet another version attached (also available at:
> https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-9). The
> main change is the shape of the ShmemRequest*() calls:
>
> On 27/03/2026 02:51, Heikki Linnakangas wrote:
> > Another idea is to use a macro to hide that from pgindent, which would
> > make the calls little less verbose anyway:
> >
> > #define ShmemRequestStruct(desc, ...) ShmemRequestStructWithOpts(desc,
> > &(ShmemRequestStructOpts) { __VA_ARGS__ })
> >
> > Then the call would be simply:
> >
> >      ShmemRequestStruct(&pgssSharedStateDesc,
> >          .name = "pg_stat_statements",
> >          .size = sizeof(pgssSharedState),
> >          .ptr = (void **) &pgss,
> >      );
>
> I went with that approach. We're already doing something similar with
> XL_ROUTINE in xlogreader.h:
>
> #define XL_ROUTINE(...) &(XLogReaderRoutine){__VA_ARGS__}
>
> The calls look like this:
>
> xlogreader =
>      XLogReaderAllocate(wal_segment_size, NULL,
>                         XL_ROUTINE(.page_read = &XLogPageRead,
>                                    .segment_open = NULL,
>                                    .segment_close = wal_segment_close),
>                         private);
>
> If we followed that example, ShmemRequestStruct() calls would look like
> this:
>
> ShmemRequestStruct(&pgssSharedStateDesc,
>                     SHMEM_STRUCT_OPTS(.name = "pg_stat_statements",
>                                       .size = sizeof(pgssSharedState),
>                                       .ptr = (void **) &pgss,
>      );
>
> However, I don't like the deep indentation, it feels like the important
> stuff is buried to the right. And pgindent insists on that. So I went
> with the proposal I quoted above, turning ShmemRequestStruct(...) itself
> into a macro. If you need more complex options setup, you can set up the
> struct without the macro and call ShmemRequestStructWithOpts() directly,
> but so far all of the callers can use the macro.
>

I like this. I have tried it only for the resizable_shmem structure
which is not complex.

>
> Ashutosh, I think I've addressed most of your comments so far. I'm
> replying to just a few of them here that might need more discussion:
>

Thanks.

> >
> > +} shmem_startup_state;
> >
> > This isn't just startup state since the backend can toggle between
> > DONE and LATE_ATTACH_OR_INIT states after the startup. Probably
> > "shmem_state" would be a better name.
>
> Renamed to "shmem_request_state". And renamed "LATE_ATTACH_OR_INIT" to
> "AFTER_STARTUP_ATTACH_OR_INIT" to match the terminology I used elsewhere.
>
> I'm still not entirely happy with this state machine. It seems useful to
> have it for sanity checking, but it still feels a little unclear what
> state you're in at different points in the code, and as an aesthetic
> thing, the whole enum feels too prominent given that it's just for
> sanity checks.

I am ok even if it is used just for sanity checks - but with the
shared structure requests coming at any time during the life of a
server, it would be easy to get lost without those sanity checks. I
also see it being used in RegisterShmemCallbacks(), so it's not for
just sanity checks, right?

>
> > + ShmemStructDesc *desc = area->desc;
> > +
> > + AttachOrInit(desc, false, true);
> > + }
> > + list_free(requested_shmem_areas);
> > + requested_shmem_areas = NIL;
> >
> > If we pop all the nodes from the list, then the list should be NIL
> > right? Why do we need to free it?
> >
> > + else if (!init_allowed)
> > + {
> >
> > For the sake of documentation and sanity, I would add
> > Assert(!index_entry) here, possibly with a comment. Otherwise it feels
> > like we might be leaving a half-initialized entry in the hash table.
> >
> > What if attach_allowed is false and the entry is not found? Should we
> > throw an error in that case too? It would be foolish to call
> > AttachOrInit with both init_allowed and attach_allowed set to false,
> > but the API allows it and we should check for that.
> >
> > It feels like we should do something about the arguments. The function
> > is hard to read. init_allowed is actually the action the caller wants
> > to take if the entry is not found, and attach_allowed is the action
> > the caller wants to take if the entry is found.
> >
> > Also explain in the comment what does attach mean here especially in
> > case of fixed sized structures.
>
> I renamed it to AttachOrInitShmemIndexEntry, and the args to 'may_init'
> and 'may_attach'. But more importantly I added comments to explain the
> different usages. Hope that helps..

The explanation in the prologue looks good. But the function is still
confusing. Instead of if ... else fi ... chain, I feel organizing this
as below would make it more readable. (this was part of one of my
earlier edit patches).
if (found)
...
else
{
 if (!may_init)
 error
if (!index_entry)
 error

... rest of the code to initialize and attach
}

But other than that I don't have any other brilliant ideas.

>
> On 01/04/2026 14:59, Ashutosh Bapat wrote:
> > 0008
> > ------
> > - LWLockRelease(AddinShmemInitLock);
> > + /* The hash table must be initialized already */
> > + Assert(pgss_hash != NULL);
> >
> > Does it make sense to also Assert(pgss)? A broader question is do we
> > want to make it a pattern that every user of ShmemRequest*() also
> > Assert()s that the pointer is non-NULL in the init callback? It is a
> > test that the ShmemRequest*(), which is far from, init_fn is working
> > correctly.
>
> The function does a lot of accesses of 'pgss' so if that's NULL you'll
> get a crash pretty quickly. I'm not sure if the Assert(pgss_hash !=
> NULL) is really needed either, but I'm inclined to keep it, as pgss_hash
> might not otherwise be accessed in the function, and there are runtime
> checks for it in the other functions, so if it's not initialized for
> some reason, things might still appear to work to some extent. I don't
> think I want to have that as a broader pattern though.

In Assert build, an Assert() at least appears in the server log file,
that gives a good direction to start investigation. Without Assert, it
gives segmentation faults without any idea where it came from. That's
a mild benefit of assert.

>
> > + /*
> > + * Extra space to reserve in the shared memory segment, but it's not part
> > + * of the struct itself. This is used for shared memory hash tables that
> > + * can grow beyond the initial size when more buckets are allocated.
> > + */
> > + size_t extra_size;
> >
> > When we introduce resizable structures (where even the hash table
> > directly itself could be resizable), we will introduce a new field
> > max_size which is easy to get confused with extra_size. Maybe we can
> > rename extra_size to something like "auxilliary_size" to mean size of
> > the auxiliary parts of the structure which are not part of the main
> > struct itself.
> >
> > + /*
> > + * max_size is the estimated maximum number of hashtable entries. This is
> > + * not a hard limit, but the access efficiency will degrade if it is
> > + * exceeded substantially (since it's used to compute directory size and
> > + * the hash table buckets will get overfull).
> > + */
> > + size_t max_size;
> > +
> > + /*
> > + * init_size is the number of hashtable entries to preallocate. For a
> > + * table whose maximum size is certain, this should be equal to max_size;
> > + * that ensures that no run-time out-of-shared-memory failures can occur.
> > + */
> > + size_t init_size;
> >
> > Everytime I look at these two fields, I question whether those are the
> > number of entries (i.e. size of the hash table) or number of bytes
> > (size of the memory). I know it's the former, but it indicates that
> > something needs to be changed here, like changing the names to have
> > _entries instead of _size, or changing the type to int64 or some such.
> > Renaming to _entries would conflict with dynahash APIs since they use
> > _size, so maybe the latter?
>
> I hear you, but I didn't change these yet. If we go with the patches
> from the "Shared hash table allocations" thread, max_size and init_size
> will be merged into one. I'll try to settle that thread before making
> changes here.

Will review those patches next.

>
> > -void
> > -InitProcGlobal(void)
> > +static void
> > +ProcGlobalShmemInit(void *arg)
> > {
>
> I'm not sure what you meant to say here, but I did notice that there
> were a bunch of references to InitProcGlobal() left over in comments.
> Fixed those.

Oh, I just wanted to say that the new version reads much better than
the old version, which had ShmemStructInit() sprinkled at seemingly
random places. I missed writing that. Nothing serious there.

I also rebased my resizable shmem patch on v9. Attached here. I have
addressed the following open items from the list at [1]
1. The test is stable now. I found a way to make (roughly) sure that
we are not allocating more than required memory for a resizable
structure.
2. Disable the feature on platforms that do not have
MADV_POPULATE_WRITE and MADV_REMOVE. The feature is also disabled for
EXEC_BACKEND case. I have tested the EXEC_BACKEND case, but I have not
tested platforms which do not have those constants defined or on
Windows.

The first two items from [1] need some discussion still.

[1] https://www.postgresql.org/message-id/CAExHW5so6VSxBC-1V=35229Z1+dw5vhw8HxHg9ry7UzceKcXzA@mail.gmail.com

-- 
Best Wishes,
Ashutosh Bapat

  44. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-02T22:10:21Z

    On Wed, 1 Apr 2026 at 20:17, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> Yet another version attached (also available at:
> https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-9). The
> main change is the shape of the ShmemRequest*() calls:

I didn't read the whole thread, as it's quite long, but did look at
the patchset for a while to figure out where it's going.

0005:
A few assorted comments:

While I do think it's an improvement over the current APIs, the
improvement seems to be mostly concentrated in the RequestStruct/Hash
department, with only marginal improvements in RegisterShmemCallbacks.
I feel like it's missing the important part: I'd like
direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
ShmemRequestStruct/Hash had a size callback as alternative to the size
field (which would then be called after preload_libraries finishes)
then that would be sufficient for most shmem allocations, and it'd
simplify shmem management for most subsystems.
We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
allow conditionally allocated shmem areas (e.g. those used in aio),
but I think that, in general, we shouldn't need a separate callback
function just to get started registering shmem structures.

I also noticed that ShmemCallbacks.%_arg are generally undocumented,
and I couldn't find any users in core (at the end of the patchset)
that actually use the argument. Could it be I missed something?

I don't understand the use of ShmemStructDesc. They generally/always
are private to request_fn(), and their fields are used exclusively
inside the shmem mechanisms, with no reads of its fields that can't
already be deduced from context. Why do we need that struct
everywhere?

> +++ b/src/backend/storage/ipc/shmem.c
[...]
> +    /* Check that it's not already registered in this process */
> +    foreach_ptr(ShmemStructDesc, existing, pending_shmem_requests)
> +    {
> +        if (strcmp(existing->name, options->name) == 0)
> +            ereport(ERROR,
> +                    (errmsg("shared memory struct \"%s\" is already registered",
> +                            options->name)));
> +    }
> +
> +    request = palloc(sizeof(ShmemRequest));
> +    request->options = options;
> +    request->desc = desc;
> +    request->kind = kind;
> +    pending_shmem_requests = lappend(pending_shmem_requests, request);

Apparently, pending_shmem_requests is a list of ShmemRequest, but the
iteration just above on the same list assumes ShmemStructDesc, which
seems wrong to me.

00017:
I like this idea, but I think it missed its chance to make good on an
opportunity to reduce waste in alignments: We know which structs we're
going to allocate at which alignments, so we could save space by
packing the structs. I don't expect it to save much, but it could be a
few 100 of kbs with a few BLCKSZ-aligned allocations.


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

  45. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-03T13:12:58Z

    On Fri, Apr 3, 2026 at 3:40 AM Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
>
> On Wed, 1 Apr 2026 at 20:17, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >
>
> While I do think it's an improvement over the current APIs, the
> improvement seems to be mostly concentrated in the RequestStruct/Hash
> department, with only marginal improvements in RegisterShmemCallbacks.
> I feel like it's missing the important part: I'd like
> direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
> ShmemRequestStruct/Hash had a size callback as alternative to the size
> field (which would then be called after preload_libraries finishes)
> then that would be sufficient for most shmem allocations, and it'd
> simplify shmem management for most subsystems.
> We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
> allow conditionally allocated shmem areas (e.g. those used in aio),
> but I think that, in general, we shouldn't need a separate callback
> function just to get started registering shmem structures.
>
> I also noticed that ShmemCallbacks.%_arg are generally undocumented,
> and I couldn't find any users in core (at the end of the patchset)
> that actually use the argument. Could it be I missed something?
>
> I don't understand the use of ShmemStructDesc. They generally/always
> are private to request_fn(), and their fields are used exclusively
> inside the shmem mechanisms, with no reads of its fields that can't
> already be deduced from context. Why do we need that struct
> everywhere?

My resizable shared memory structure patches use it as a handle to the
structure to be resized.

-- 
Best Wishes,
Ashutosh Bapat

  46. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-04T00:45:18Z

    On 03/04/2026 16:12, Ashutosh Bapat wrote:
> On Fri, Apr 3, 2026 at 3:40 AM Matthias van de Meent
> <boekewurm+postgres@gmail.com> wrote:
>> While I do think it's an improvement over the current APIs, the
>> improvement seems to be mostly concentrated in the RequestStruct/Hash
>> department, with only marginal improvements in RegisterShmemCallbacks.
>> I feel like it's missing the important part: I'd like
>> direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
>> ShmemRequestStruct/Hash had a size callback as alternative to the size
>> field (which would then be called after preload_libraries finishes)
>> then that would be sufficient for most shmem allocations, and it'd
>> simplify shmem management for most subsystems.
>> We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
>> allow conditionally allocated shmem areas (e.g. those used in aio),
>> but I think that, in general, we shouldn't need a separate callback
>> function just to get started registering shmem structures.
>>
>> I also noticed that ShmemCallbacks.%_arg are generally undocumented,
>> and I couldn't find any users in core (at the end of the patchset)
>> that actually use the argument. Could it be I missed something?

None of the current code currently uses it, that's correct. I felt it 
might become very handy in the future or in extensions, if you wanted to 
reuse the same function for initializing different shmem areas, for 
example. It's a pretty common pattern to have an opaque pointer like 
that in any callbacks.

>> I don't understand the use of ShmemStructDesc. They generally/always
>> are private to request_fn(), and their fields are used exclusively
>> inside the shmem mechanisms, with no reads of its fields that can't
>> already be deduced from context. Why do we need that struct
>> everywhere?
> 
> My resizable shared memory structure patches use it as a handle to the
> structure to be resized.

Right. And hash tables and SLRUs use a desc-like object already, so for 
symmetry it feels natural to have it for plain structs too.

I wonder if we should make it optional though, for the common case that 
you have no intention of doing anything more with the shmem region that 
you'd need a desc for. I'm thinking you could just pass NULL for the 
desc pointer:

     ShmemRequestStruct(NULL,
         .name = "pg_stat_statements",
         .size = sizeof(pgssSharedState),
         .ptr = (void **) &pgss,
     };

- Heikki

  47. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-04T00:49:25Z

    On 02/04/2026 09:58, Ashutosh Bapat wrote:
> On Wed, Apr 1, 2026 at 11:47 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>>> + /*
>>> + * Extra space to reserve in the shared memory segment, but it's not part
>>> + * of the struct itself. This is used for shared memory hash tables that
>>> + * can grow beyond the initial size when more buckets are allocated.
>>> + */
>>> + size_t extra_size;
>>>
>>> When we introduce resizable structures (where even the hash table
>>> directly itself could be resizable), we will introduce a new field
>>> max_size which is easy to get confused with extra_size. Maybe we can
>>> rename extra_size to something like "auxilliary_size" to mean size of
>>> the auxiliary parts of the structure which are not part of the main
>>> struct itself.
>>>
>>> + /*
>>> + * max_size is the estimated maximum number of hashtable entries. This is
>>> + * not a hard limit, but the access efficiency will degrade if it is
>>> + * exceeded substantially (since it's used to compute directory size and
>>> + * the hash table buckets will get overfull).
>>> + */
>>> + size_t max_size;
>>> +
>>> + /*
>>> + * init_size is the number of hashtable entries to preallocate. For a
>>> + * table whose maximum size is certain, this should be equal to max_size;
>>> + * that ensures that no run-time out-of-shared-memory failures can occur.
>>> + */
>>> + size_t init_size;
>>>
>>> Everytime I look at these two fields, I question whether those are the
>>> number of entries (i.e. size of the hash table) or number of bytes
>>> (size of the memory). I know it's the former, but it indicates that
>>> something needs to be changed here, like changing the names to have
>>> _entries instead of _size, or changing the type to int64 or some such.
>>> Renaming to _entries would conflict with dynahash APIs since they use
>>> _size, so maybe the latter?
>>
>> I hear you, but I didn't change these yet. If we go with the patches
>> from the "Shared hash table allocations" thread, max_size and init_size
>> will be merged into one. I'll try to settle that thread before making
>> changes here.
> 
> Will review those patches next.

Those are now committed, and here's a new version rebased over those 
changes. The hash options is now called 'nelems', and the 'extra_size' 
in ShmemStructOpts is gone.

Plus a bunch of other fixes and cleanups. I also reordered and 
re-grouped the patches a little, into more logical increments I hope.

- Heikki

  48. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-04T12:00:11Z

    On Sat, 4 Apr 2026 at 02:45, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 03/04/2026 16:12, Ashutosh Bapat wrote:
> > On Fri, Apr 3, 2026 at 3:40 AM Matthias van de Meent
> > <boekewurm+postgres@gmail.com> wrote:
> >> While I do think it's an improvement over the current APIs, the
> >> improvement seems to be mostly concentrated in the RequestStruct/Hash
> >> department, with only marginal improvements in RegisterShmemCallbacks.
> >> I feel like it's missing the important part: I'd like
> >> direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
> >> ShmemRequestStruct/Hash had a size callback as alternative to the size
> >> field (which would then be called after preload_libraries finishes)
> >> then that would be sufficient for most shmem allocations, and it'd
> >> simplify shmem management for most subsystems.
> >> We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
> >> allow conditionally allocated shmem areas (e.g. those used in aio),
> >> but I think that, in general, we shouldn't need a separate callback
> >> function just to get started registering shmem structures.
> >>
> >> I also noticed that ShmemCallbacks.%_arg are generally undocumented,
> >> and I couldn't find any users in core (at the end of the patchset)
> >> that actually use the argument. Could it be I missed something?
>
> None of the current code currently uses it, that's correct. I felt it
> might become very handy in the future or in extensions, if you wanted to
> reuse the same function for initializing different shmem areas, for
> example.

That's cool, but if that common initialization path is common enough
to need special coding, then how come that this patch make PG use it?
I can think of many systems that "just" initialize a hash table or
"just" allocate a shmem area.

> It's a pretty common pattern to have an opaque pointer like
> that in any callbacks.

I agree that it's a rather common pattern, but from an OOP
perspective, shouldn't the argument be the ShmemCallbacks*? Users can
embed the struct to extend the data carried if they need it to.

> >> I don't understand the use of ShmemStructDesc. They generally/always
> >> are private to request_fn(), and their fields are used exclusively
> >> inside the shmem mechanisms, with no reads of its fields that can't
> >> already be deduced from context. Why do we need that struct
> >> everywhere?
> >
> > My resizable shared memory structure patches use it as a handle to the
> > structure to be resized.
>
> Right. And hash tables and SLRUs use a desc-like object already, so for
> symmetry it feels natural to have it for plain structs too.
> I wonder if we should make it optional though, for the common case that
> you have no intention of doing anything more with the shmem region that
> you'd need a desc for. I'm thinking you could just pass NULL for the
> desc pointer:
>
>      ShmemRequestStruct(NULL,
>          .name = "pg_stat_statements",
>          .size = sizeof(pgssSharedState),
>          .ptr = (void **) &pgss,
>      };

That would help, though I'd still wonder why we'd have separate Opts
and Desc structs. IIUC, they generally carry (exactly) the same data.

Maybe moving it into a `.handle` or `.desc` field in Shmem*Opts could
make that part of the code a bit cleaner; as it'd further clarify that
it's very much an optional field.

I'll check out your latest version in a bit.


Kind regards,

Matthias van de Meent

  49. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-04T13:51:09Z

    On Sat, 4 Apr 2026 at 02:49, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> Those are now committed, and here's a new version rebased over those
> changes. The hash options is now called 'nelems', and the 'extra_size'
> in ShmemStructOpts is gone.
>
> Plus a bunch of other fixes and cleanups. I also reordered and
> re-grouped the patches a little, into more logical increments I hope.

0001: LGTM

0002:

> +++ b/src/backend/storage/ipc/shmem.c

> + * Nowadays, there is also a third way to allocate shared memory called

There's no clear indicator of the second way to allocate shared
memory, nor is the first one clearly defined in the new verson of the
comment block.


> + * item is deleted.  However, if one hash table grows very large and then
> + * shrinks, its space cannot be redistributed to other tables.  We could build
> + * a simple hash bucket garbage collector if need be.  Right now, it seems
> + * unnecessary.

I think this new text is outdated, given that we don't have growing
hash tables anymore.

I also think it should've referred to elements, not buckets;
dynahash's buckets cannot readily be deallocated as they're generally
always "in use" (they might be NULL, but they're still accessed in
read operations on missing keys). Elements are put in the freelist if
not used, and those could be released into a memory pool if so desired
(and coded).

> + * In builtin PostgreSQL code, add the callbacks to the list in
> + * src/include/storage/subsystemlist.h.

This refers to an automation system that's introduced a few commits
later, in commit 0005, and therefore probably should be added only in
that commit.

> + * Legacy ShmemInitStruct()/ShmemInitHash() functions
> + * --------------------------------------------------

Should we have checks in place to avoid calls to new APIs from old
callbacks, and vice versa?

> ShmemRequestInternal(...
> +    ShmemRequest *request;
[...]
> +    foreach_ptr(ShmemStructDesc, existing, pending_shmem_requests)
[...]
> +    request = palloc(sizeof(ShmemRequest));
[...]
> +    pending_shmem_requests = lappend(pending_shmem_requests, request);

It looks like you missed my earlier comment about type confusion.
Here, pending_shmem_requests is a List of ShmemRequest pointers, while
the foreach_ptr() uses ShmemStructDesc, which is a type confusion. The
loop checks the 'char *name' field of ShmemStructDesc, which in a
ShmemRequest is the 'ShmemStructDesc *desc'. This bug would cause
issues if different ShmemStructDescs are registered by the same name,
as the ShmemStructDescs wouldn't (necessarily) be strcmp()-equal for
the same name.

> ShmemAttachRequested(void)
> +    /* Call attach callbacks */
> +    foreach(lc, registered_shmem_callbacks)
> +    {
> +        const ShmemCallbacks *callbacks = (const ShmemCallbacks *) lfirst(lc);

This would be more concise with foreach_ptr(const ShmemCallbacks,
callbacks, registered_shmem_callbacks), like in ShmemInitRequested.


> +++ b/src/include/storage/shmem.h
> +/*
> + * Shared memory is reserved and allocated in stages at postmaster startup,
> + * and in EXEC_BACKEND mode, there's some extra work done to "attach" to them
> + * at backend startup.  ShmemCallbacks holds callback functions that are
> + * called at different stages.
> + */
> +typedef struct ShmemCallbacks

Maybe this should also have the opportunity for a (before_)shmem_exit callback?

> + * on-demaind in a backend.  If a subsystem sets this flag, the callbacks are
> + * called immediately after registration, to initialize or attach to the
> + * requested shared memory areas.

Ideally we only immediately call the callbacks if we're under
postmaster, or in a standalone backend; we shouldn't allocate shmem
for some preloaded libraries that set this flag, at least not ahead of
loading all preload libraries.

0003: Maybe this could also test that the protections we're putting in
place against double-registration of shmem areas actually detect the
duplication issue?
Otherwise, LGTM

0004-0014: TBD

While it's mostly mechanical changes, it did make me notice the rather
annoying allocation patterns by XLOGShmemRequest. It allocates various
types of data in one go (which, in principle, is fine) but in doing so
it adds its own alignment tricks etc, and I'm not super stoked about
that. If time allows, could we clean that up?


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

  50. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-04T16:32:47Z

    On Sat, Apr 4, 2026 at 6:19 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 02/04/2026 09:58, Ashutosh Bapat wrote:
> > On Wed, Apr 1, 2026 at 11:47 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >>> + /*
> >>> + * Extra space to reserve in the shared memory segment, but it's not part
> >>> + * of the struct itself. This is used for shared memory hash tables that
> >>> + * can grow beyond the initial size when more buckets are allocated.
> >>> + */
> >>> + size_t extra_size;
> >>>
> >>> When we introduce resizable structures (where even the hash table
> >>> directly itself could be resizable), we will introduce a new field
> >>> max_size which is easy to get confused with extra_size. Maybe we can
> >>> rename extra_size to something like "auxilliary_size" to mean size of
> >>> the auxiliary parts of the structure which are not part of the main
> >>> struct itself.
> >>>
> >>> + /*
> >>> + * max_size is the estimated maximum number of hashtable entries. This is
> >>> + * not a hard limit, but the access efficiency will degrade if it is
> >>> + * exceeded substantially (since it's used to compute directory size and
> >>> + * the hash table buckets will get overfull).
> >>> + */
> >>> + size_t max_size;
> >>> +
> >>> + /*
> >>> + * init_size is the number of hashtable entries to preallocate. For a
> >>> + * table whose maximum size is certain, this should be equal to max_size;
> >>> + * that ensures that no run-time out-of-shared-memory failures can occur.
> >>> + */
> >>> + size_t init_size;
> >>>
> >>> Everytime I look at these two fields, I question whether those are the
> >>> number of entries (i.e. size of the hash table) or number of bytes
> >>> (size of the memory). I know it's the former, but it indicates that
> >>> something needs to be changed here, like changing the names to have
> >>> _entries instead of _size, or changing the type to int64 or some such.
> >>> Renaming to _entries would conflict with dynahash APIs since they use
> >>> _size, so maybe the latter?
> >>
> >> I hear you, but I didn't change these yet. If we go with the patches
> >> from the "Shared hash table allocations" thread, max_size and init_size
> >> will be merged into one. I'll try to settle that thread before making
> >> changes here.
> >
> > Will review those patches next.
>
> Those are now committed, and here's a new version rebased over those
> changes. The hash options is now called 'nelems', and the 'extra_size'
> in ShmemStructOpts is gone.
>

Thanks. Adjusted my resizable shared memory patch on top of this. The
result looks better.

> Plus a bunch of other fixes and cleanups. I also reordered and
> re-grouped the patches a little, into more logical increments I hope.

Some more comments

test_shmem declares MODULE_big and OBJS which seems to be old
fashioned, newer modules seem to be using MODULES. Also it should use
NO_INSTALLCHECK.

/*
* Alignment of the starting address. If not set, defaults to cacheline
* boundary. Must be a power of two.
*/
size_t alignment;

We don't seem to enforce the "must be a power of two" rule anywhere.
We should at least validate it.

I like the way buffer manager related changes untangle sub-sub-systems
of Buffer manager viz. StrategyControl and buffer look up table.
Simplifies code very much.

I also eyeballed some of the changes in 0014. If time permits, I will
review those closely soon. But the changes look ok.

Before this change, replication_states_ctl in origin.c was not
initialized explicitly when max_active_replication_origins = 0. With
this change, the structure is not registered and thus global static
pointer is not initialized. However, given that it's implicit, I
suggest adding Asserts as attached.

-- 
Best Wishes,
Ashutosh Bapat

  51. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-04T17:32:33Z

    On 04/04/2026 15:00, Matthias van de Meent wrote:
> On Sat, 4 Apr 2026 at 02:45, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>>>> I don't understand the use of ShmemStructDesc. They generally/always
>>>> are private to request_fn(), and their fields are used exclusively
>>>> inside the shmem mechanisms, with no reads of its fields that can't
>>>> already be deduced from context. Why do we need that struct
>>>> everywhere?
>>>
>>> My resizable shared memory structure patches use it as a handle to the
>>> structure to be resized.
>>
>> Right. And hash tables and SLRUs use a desc-like object already, so for
>> symmetry it feels natural to have it for plain structs too.
>> I wonder if we should make it optional though, for the common case that
>> you have no intention of doing anything more with the shmem region that
>> you'd need a desc for. I'm thinking you could just pass NULL for the
>> desc pointer:
>>
>>       ShmemRequestStruct(NULL,
>>           .name = "pg_stat_statements",
>>           .size = sizeof(pgssSharedState),
>>           .ptr = (void **) &pgss,
>>       };
> 
> That would help, though I'd still wonder why we'd have separate Opts
> and Desc structs. IIUC, they generally carry (exactly) the same data.
> 
> Maybe moving it into a `.handle` or `.desc` field in Shmem*Opts could
> make that part of the code a bit cleaner; as it'd further clarify that
> it's very much an optional field.

Yeah. OTOH, I'd like to separate the options from what's effectively a 
return value. But maybe you're right and it's nevertheless better that way.

Some options on this:

a) What's in the patch now

     static ShmemStructDesc pgssSharedStateDesc;

     ShmemRequestStruct(&pgssSharedStateDesc,
                        .name = "pg_stat_statements",
                        .size = sizeof(pgssSharedState),
                        .ptr = (void **) &pgss);

b) Allow passing NULL for the desc

     ShmemRequestStruct(NULL,
                        .name = "pg_stat_statements",
                        .size = sizeof(pgssSharedState),
                        .ptr = (void **) &pgss);

c) Return the Desc as a return value

     static ShmemStructDesc *pgssSharedStateDesc;

     pgssSharedStateDesc =
         ShmemRequestStruct(.name = "pg_stat_statements",
                            .size = sizeof(pgssSharedState),
                            .ptr = (void **) &pgss);

In option c) you can just throw away the result if you don't need it. I 
kind of like this as a notational thing. However it has some downsides:

This changes the return value to be a pointer. I'm thinking that 
ShmemRequestStruct() palloc's the descriptor struct in TopMemoryContext. 
This is a little ugly because the descriptor struct is leaked if the 
caller throws it away. It's not a lot of memory, but still.

I'm also not sure how well this fits in with the SLRU code. On 'master', 
you already have SlruCtlData which is like the "desc" struct. Would we 
turn that into a pointer too, adding one indirection to all the SLRU 
calls. It's probably fine from a performance point of view, but it feels 
like it's going in the wrong direction.

d) Make it part of Opts, as you suggested

     static ShmemStructDesc pgssSharedStateDesc;

     ShmemRequestStruct(.name = "pg_stat_statements",
                        .size = sizeof(pgssSharedState),
                        .ptr = (void **) &pgss,
                        .desc = &pgssSharedStateDesc);

In the attached new version, though, I stepped back and decided to 
remove the whole ShmemStructDesc after all. I still think having a 
handle like that is a good idea, and the follow-up patches for resizing 
need it. However, with option d) it can easily be added later. With 
option d), it seems silly to have it be part of the patch now, when the 
desc struct doesn't really do anything. SLRU's still have a similar 
SlruDesc struct, however. For SLRUs it's essentially the same as the old 
SlruCtlData struct before these patches.

The Desc structs were being used for one thing though: I used the 'size' 
from the Desc struct in ProcGlobalShmemInit() to get the allocated size 
of each shmem area. The size computation there is complicated enough 
that I'd rather not repeat it, and avoiding the repeated size 
calculation was the raison d'être for these patches. I replaced it with 
global variables to hold the sizes from the ShmemRequest() step to 
ShmemInit(). But that would be one case where having the desc would 
already be useful. Then again, I'm not sure we want to expose the 'size' 
in the descriptor like that anyway, because as soon as we make shmem 
regions resizable, we might not be able to keep the size in the 
descriptor up-to-date. The size of these structs won't change, but we 
might not want to expose the information because it would be confusing 
for other structs where it can change to show outdated information.

On a related note, when we add back the ".desc" concept later, is 
".desc" a good name, or ".handle" as you also suggested? More widely, do 
we call the concept and the struct a "handle" or "descriptor" or what? 
Or if we follow the precedence with the existing SlruCtlData struct, it 
could be ".ctl". I'm not a fan of the "Ctl" naming though, because we 
already have a lot of structs with "Ctl" in the name and it's not always 
clear whether a "Ctl" struct refers to the shared memory parts or the 
handle to it. Now that the "desc" structs are not part of these patches 
anymore, however, we can punt on that decision.

On 02/04/2026 09:58, Ashutosh Bapat wrote:
>>
>> I renamed it to AttachOrInitShmemIndexEntry, and the args to 'may_init'
>> and 'may_attach'. But more importantly I added comments to explain the
>> different usages. Hope that helps..
> 
> The explanation in the prologue looks good. But the function is still
> confusing. Instead of if ... else fi ... chain, I feel organizing this
> as below would make it more readable. (this was part of one of my
> earlier edit patches).
> if (found)
> ...
> else
> {
>  if (!may_init)
>  error
> if (!index_entry)
>  error
> 
> ... rest of the code to initialize and attach
> }
> 
> But other than that I don't have any other brilliant ideas.

I did another refactoring in this area: I split 
AttachOrInitShmemIndexEntry() into separate AttachShmemIndexEntry() and 
InitShmemIndexEntry functions again. There's a little bit of repetition 
that way, but IMO it makes it much clearer overall.

Other changes in this patch version:

- I moved some of the stuff from shmem.h to a new shmem_internal.h 
header. The idea is that what remains in shmem.h provides the public API 
for allocating shared memory.

- I refactored the "after-startup request" code. It now detects the case 
that some of the shmem areas, but not all, have already been initialized 
and throws an error.

Still processing the rest of the feedback from the past days. This patch 
version is also available at 
https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-11.

- Heikki

  52. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-04T23:17:29Z

    On Sat, 4 Apr 2026 at 19:32, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 04/04/2026 15:00, Matthias van de Meent wrote:
> > On Sat, 4 Apr 2026 at 02:45, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >>>> I don't understand the use of ShmemStructDesc. They generally/always
> >>>> are private to request_fn(), and their fields are used exclusively
> >>>> inside the shmem mechanisms, with no reads of its fields that can't
> >>>> already be deduced from context. Why do we need that struct
> >>>> everywhere?
> >>>
> >>> My resizable shared memory structure patches use it as a handle to the
> >>> structure to be resized.
> >>
> >> Right. And hash tables and SLRUs use a desc-like object already, so for
> >> symmetry it feels natural to have it for plain structs too.
> >> I wonder if we should make it optional though, for the common case that
> >> you have no intention of doing anything more with the shmem region that
> >> you'd need a desc for. I'm thinking you could just pass NULL for the
> >> desc pointer:
> >>
> >>       ShmemRequestStruct(NULL,
> >>           .name = "pg_stat_statements",
> >>           .size = sizeof(pgssSharedState),
> >>           .ptr = (void **) &pgss,
> >>       };
> >
> > That would help, though I'd still wonder why we'd have separate Opts
> > and Desc structs. IIUC, they generally carry (exactly) the same data.
> >
> > Maybe moving it into a `.handle` or `.desc` field in Shmem*Opts could
> > make that part of the code a bit cleaner; as it'd further clarify that
> > it's very much an optional field.
>
> Yeah. OTOH, I'd like to separate the options from what's effectively a
> return value. But maybe you're right and it's nevertheless better that way.
>
> Some options on this:
>
> a) What's in the patch now
[...]
> b) Allow passing NULL for the desc
[...]
> c) Return the Desc as a return value
[...]
> In option c) you can just throw away the result if you don't need it. I
> kind of like this as a notational thing. However it has some downsides:
>
> This changes the return value to be a pointer. I'm thinking that
> ShmemRequestStruct() palloc's the descriptor struct in TopMemoryContext.
> This is a little ugly because the descriptor struct is leaked if the
> caller throws it away. It's not a lot of memory, but still.

Yeah, it'd be bad if we'd leak it, as it could cause some
semipermanent memory leaks when the server keeps restarting after
crash without resetting TopMemoryContext.

> d) Make it part of Opts, as you suggested
[...]
> In the attached new version, though, I stepped back and decided to
> remove the whole ShmemStructDesc after all. I still think having a
> handle like that is a good idea, and the follow-up patches for resizing
> need it. However, with option d) it can easily be added later. With
> option d), it seems silly to have it be part of the patch now, when the
> desc struct doesn't really do anything.

Thanks!

> Other changes in this patch version:
>
> - I moved some of the stuff from shmem.h to a new shmem_internal.h
> header. The idea is that what remains in shmem.h provides the public API
> for allocating shared memory.
>
> - I refactored the "after-startup request" code. It now detects the case
> that some of the shmem areas, but not all, have already been initialized
> and throws an error.
>
> Still processing the rest of the feedback from the past days. This patch
> version is also available at
> https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-11.

Thanks!

> On Sat, 4 Apr 2026 at 02:49, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> > Those are now committed, and here's a new version rebased over those
> > changes. The hash options is now called 'nelems', and the 'extra_size'
> > in ShmemStructOpts is gone.
> >
> > Plus a bunch of other fixes and cleanups. I also reordered and
> > re-grouped the patches a little, into more logical increments I hope.
>
> 0004-0014: TBD

Review continued, based on v11.

0004: LGTM, with some nits:

> + * This is called at postmaster startup.  Note that the shared memory isn't
> + * allocated here yet, this merely register our needs.

Typo: register -> registers

Formatting:
> +    ShmemRequestHash(.name = "pg_stat_statements hash",
> +                     .nelems = pgss_max,
> +                     .hash_info.keysize = sizeof(pgssHashKey),
> +                     .hash_info.entrysize = sizeof(pgssEntry),
> +                     .hash_flags = HASH_ELEM | HASH_BLOBS,
> +                     .ptr = &pgss_hash,
> +        );
(note that additional unit of indentation for the closing bracket)

Is this malformatting caused by pgindent? If so, could you see if
there's a better way of defining ShmemRequestHash/Struct that doesn't
have this indent as output?

> +    pgss->extent = 0;
> +    pgss->n_writers = 0;
> +    pgss->gc_count = 0;
> +    pgss->stats.dealloc = 0;

Shmem is said to be zero-initialized, should we remove the manual
zero-initialization?

> +    on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
See my upthread comment about adding optional on_shmem_exit callbacks
to ShmemCallbacks.

0005: LGTM

0006: I don't think it is a great idea to make the LwLock machinery
the first to get allocation requests:
It has the RequestNamedLWLockTranche infrastructure, which can only
register new requests while process_shmem_requests_in_progress, and
making it request its memory ahead of everything else is likely to
cause an undersized tranche to be allocated. You could make sure that
this isn't an issue by maintaining a flag in lwlock.c that's set when
the shmem request is made (and reset on shmem exit), which must be
false when RequestNamedLWLockTranche() is called, and if not then it
should throw an error.

0007: LGTM, Nits:

Patch description: ProgGlobal -> ProcGlobal

> %_sema.c
Not my favourite pieces of code, but that's not your patch's fault. To
me, it seems this code area has too much duplication, but that's not
something you have to fix.

0008: LGTM

> +#ifdef USE_ASSERT_CHECKING
> +    SerialPagePrecedesLogicallyUnitTests();
> +#endif

Huh, interesting. I hadn't seen such inline unit testing before.
Mailing list history seems to agree with this, so, TIL.

0009: LGTM

0010: Not looked at everything yet, but a few comment:

> +++ b/src/include/access/slru.h

With the changes in the signatures for most/all SLRU functions from a
hidden-by-typedef pointer to a visible pointer type, maybe this could
be an opportunity to swap them to `const SlruDesc *ctl` wherever
possible? I don't think there are many backend-local changes that
happen to SlruDescs once we've properly started the backend. I'm happy
to provide an incremental patch if you'd like me to spend cycles on it
if you're busy.

> +++ b/src/backend/access/transam/clog.c

> +    SimpleLruRequest(.desc = &XactSlruDesc,
> +                     .name = "transaction",
> +                     .Dir = "pg_xact",
> +                     .long_segment_names = false,
> +
> +                     .nslots = CLOGShmemBuffers(),
> +                     .nlsns = CLOG_LSNS_PER_PAGE,
> +
> +                     .sync_handler = SYNC_HANDLER_CLOG,
> +                     .PagePrecedes = CLOGPagePrecedes,
> +                     .errdetail_for_io_error = clog_errdetail_for_io_error,

That awfully inconsistent field name styling is ... awful, but not
this patch's fault. If something can be done about it in a cheap
fashion in this patch, that'd be great, but I won't hold it against
you if that's skipped.

> +++ b/src/backend/access/transam/multixact.c

> static void
> MultiXactShmemRequest(void *arg)
> [...]
> +    /*
> +     * members SLRU doesn't call SimpleLruTruncate() or meet criteria for unit
> +     * tests
> +     */
I think this comment is misplaced, it should probably be put in
MultiXactShmemInit(), below MultiXactOffset's UnitTests (which is just
a few lines below its current location).

The rest of 0010; all of 0011-0014: TBD


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

  53. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-05T05:48:10Z

    On Sat, Apr 4, 2026 at 11:02 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 04/04/2026 15:00, Matthias van de Meent wrote:
> > On Sat, 4 Apr 2026 at 02:45, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >>>> I don't understand the use of ShmemStructDesc. They generally/always
> >>>> are private to request_fn(), and their fields are used exclusively
> >>>> inside the shmem mechanisms, with no reads of its fields that can't
> >>>> already be deduced from context. Why do we need that struct
> >>>> everywhere?
> >>>
> >>> My resizable shared memory structure patches use it as a handle to the
> >>> structure to be resized.
> >>
> >> Right. And hash tables and SLRUs use a desc-like object already, so for
> >> symmetry it feels natural to have it for plain structs too.
> >> I wonder if we should make it optional though, for the common case that
> >> you have no intention of doing anything more with the shmem region that
> >> you'd need a desc for. I'm thinking you could just pass NULL for the
> >> desc pointer:
> >>
> >>       ShmemRequestStruct(NULL,
> >>           .name = "pg_stat_statements",
> >>           .size = sizeof(pgssSharedState),
> >>           .ptr = (void **) &pgss,
> >>       };
> >
> > That would help, though I'd still wonder why we'd have separate Opts
> > and Desc structs. IIUC, they generally carry (exactly) the same data.
> >
> > Maybe moving it into a `.handle` or `.desc` field in Shmem*Opts could
> > make that part of the code a bit cleaner; as it'd further clarify that
> > it's very much an optional field.
>
> Yeah. OTOH, I'd like to separate the options from what's effectively a
> return value. But maybe you're right and it's nevertheless better that way.
>
> Some options on this:
>
> a) What's in the patch now
>
>      static ShmemStructDesc pgssSharedStateDesc;
>
>      ShmemRequestStruct(&pgssSharedStateDesc,
>                         .name = "pg_stat_statements",
>                         .size = sizeof(pgssSharedState),
>                         .ptr = (void **) &pgss);
>
> b) Allow passing NULL for the desc
>
>      ShmemRequestStruct(NULL,
>                         .name = "pg_stat_statements",
>                         .size = sizeof(pgssSharedState),
>                         .ptr = (void **) &pgss);
>
> c) Return the Desc as a return value
>
>      static ShmemStructDesc *pgssSharedStateDesc;
>
>      pgssSharedStateDesc =
>          ShmemRequestStruct(.name = "pg_stat_statements",
>                             .size = sizeof(pgssSharedState),
>                             .ptr = (void **) &pgss);
>
> In option c) you can just throw away the result if you don't need it. I
> kind of like this as a notational thing. However it has some downsides:
>
> This changes the return value to be a pointer. I'm thinking that
> ShmemRequestStruct() palloc's the descriptor struct in TopMemoryContext.
> This is a little ugly because the descriptor struct is leaked if the
> caller throws it away. It's not a lot of memory, but still.
>
> I'm also not sure how well this fits in with the SLRU code. On 'master',
> you already have SlruCtlData which is like the "desc" struct. Would we
> turn that into a pointer too, adding one indirection to all the SLRU
> calls. It's probably fine from a performance point of view, but it feels
> like it's going in the wrong direction.
>
> d) Make it part of Opts, as you suggested
>
>      static ShmemStructDesc pgssSharedStateDesc;
>
>      ShmemRequestStruct(.name = "pg_stat_statements",
>                         .size = sizeof(pgssSharedState),
>                         .ptr = (void **) &pgss,
>                         .desc = &pgssSharedStateDesc);
>
> In the attached new version, though, I stepped back and decided to
> remove the whole ShmemStructDesc after all. I still think having a
> handle like that is a good idea, and the follow-up patches for resizing
> need it. However, with option d) it can easily be added later. With
> option d), it seems silly to have it be part of the patch now, when the
> desc struct doesn't really do anything. SLRU's still have a similar
> SlruDesc struct, however. For SLRUs it's essentially the same as the old
> SlruCtlData struct before these patches.
>
> The Desc structs were being used for one thing though: I used the 'size'
> from the Desc struct in ProcGlobalShmemInit() to get the allocated size
> of each shmem area. The size computation there is complicated enough
> that I'd rather not repeat it, and avoiding the repeated size
> calculation was the raison d'être for these patches. I replaced it with
> global variables to hold the sizes from the ShmemRequest() step to
> ShmemInit(). But that would be one case where having the desc would
> already be useful. Then again, I'm not sure we want to expose the 'size'
> in the descriptor like that anyway, because as soon as we make shmem
> regions resizable, we might not be able to keep the size in the
> descriptor up-to-date. The size of these structs won't change, but we
> might not want to expose the information because it would be confusing
> for other structs where it can change to show outdated information.
>
> On a related note, when we add back the ".desc" concept later, is
> ".desc" a good name, or ".handle" as you also suggested? More widely, do
> we call the concept and the struct a "handle" or "descriptor" or what?
> Or if we follow the precedence with the existing SlruCtlData struct, it
> could be ".ctl". I'm not a fan of the "Ctl" naming though, because we
> already have a lot of structs with "Ctl" in the name and it's not always
> clear whether a "Ctl" struct refers to the shared memory parts or the
> handle to it. Now that the "desc" structs are not part of these patches
> anymore, however, we can punt on that decision.

Resizing patches can do without Desc, they use name has the handle
instead. I was not comfortable with current state of Desc either
because they are not opaque as I had pointed out earlier. A caller can
scribble on them. There is not need to decide on the handle decision
right now, even for resizing patches. If we decide to add a handle, I
would like it to be opaque. I thought about using ShmemIndexEnt *
itself as the opaque pointer; we shouldn't expose it to the users of
shmem.c that it's ShmemIndexEnt * though. There is downside that we
are giving a much riskier handle in the shmem.c users' hands - they
can now corrupt shared memory itself. We could encapsulate the
ShmemIndexEntry * like how HTAB encapsulates HASHHDR if needed.
Advantage of this approach is that ShmemResizeStruct() or any shmem.c
API accepting the handle doesn't need to perform a ShmemIndex lookup.
Just ideas, nothing required right now.

>
> On 02/04/2026 09:58, Ashutosh Bapat wrote:
> >>
> >> I renamed it to AttachOrInitShmemIndexEntry, and the args to 'may_init'
> >> and 'may_attach'. But more importantly I added comments to explain the
> >> different usages. Hope that helps..
> >
> > The explanation in the prologue looks good. But the function is still
> > confusing. Instead of if ... else fi ... chain, I feel organizing this
> > as below would make it more readable. (this was part of one of my
> > earlier edit patches).
> > if (found)
> > ...
> > else
> > {
> >  if (!may_init)
> >  error
> > if (!index_entry)
> >  error
> >
> > ... rest of the code to initialize and attach
> > }
> >
> > But other than that I don't have any other brilliant ideas.
>
> I did another refactoring in this area: I split
> AttachOrInitShmemIndexEntry() into separate AttachShmemIndexEntry() and
> InitShmemIndexEntry functions again. There's a little bit of repetition
> that way, but IMO it makes it much clearer overall.
>

Yes.

I will post my resizable shmem structures patch in a separate email in
this thread but continue to review your patches.

-- 
Best Wishes,
Ashutosh Bapat

  54. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-05T05:58:51Z

    On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> I will post my resizable shmem structures patch in a separate email in
> this thread but continue to review your patches.
>

Attached is your patchset (0001 - 0014) + resizable shared memory
structures patchset 0015.

Resizable shared memory structures
============================

When allocating memory to the requested shared structures, we allocate
space for each structure. In mmap'ed shared memory, the memory is
allocated against those structures only when those structures are
initialized.
Resizable shared memory structures are simply allocated maximum space
when that happens. The function which initializes the structure is
expected to initialize only the memory worth its initial size. When
resizing the structure memory is freed or allocated against the
reserved space depending upon the new size. This allows the structures
to be resized while keeping their starting address stable which is a
hard requirement in PostgreSQL.

Resizable shared memory feature depends upon the existence of function
madvise() and constants MADV_REMOVE and MADV_WRITE_POPULATE.

On the platforms which do not have these, we disable this feature at
compile time. The commit introduces a compile time flag
HAVE_RESIZABLE_SHMEM which is defined if MADV_REMOVE and
MADV_WRITE_POPULATE exist. We don't check the existence of madvise
separately, since the existence of the constants implies the existence
of the function.

HAVE_RESIZABLE_SHMEM is not defined in EXEC_BACKEND builds since
that's largely used for Windows where the APIs to free and allocate
memory from and to a given address space are not known to the author
right now. Given that PostgreSQL is used widely on Linux, providing
this feature on Linux covers benefits most of its users. Once we
figure out the required Windows APIs, we will support this feature on
Windows as well.

The feature is also not available when Sys-V shared memory is used
even on Linux since we do not know whether required Sys-V APIs exist;
mostly they don't. Since that combination is only available for
development and testing, not supporting the feature isn't going to
impact PostgreSQL users much.

Using HAVE_RESIZABLE_SHMEM we disable compiling the code related to
resizable shared memory structures on the platforms which do not
support the feature. But we also have run time checks to disable this
feature when Sys-V shared memory is used. In order to know whether a
given instance of a running server supports resizable structures, we
have introduced GUC have_resizable_shmem.

Following points are up for discussion
=============================

1. calculation of allocated_size of resizable structures
--------------------------------
For fixed sized shared memory structures, allocated_size is the size
of the aligned structure. Assuming that the whole structure is
initialized, it is also the memory allocated to the structure. Thus
summing all allocated_size's of the allocations gives a
nearly-accurate (considering page sized allocations) idea of the total
shared memory allocated. For a resizable structure, it's a bit more
complicated. We allocate maximum space required by the structure at
the beginning. At a given point in time, the memory page where the
next structure begins and the page which contains the end of the
structure at that point in time are allocated. The pages in-between
are not allocated. The memory allocated to that structure is the
{maximum size of the structure} - {total size of unallocated pages}. I
think setting allocated_size to the actually allocated memory is more
accurate than {current size of the structure} + {alignment} which does
not reflect the actual memory allocated to the structure. I would like
to know what others think.

2. maximum_size member in various structures and in pg_shmem_allocations view
-----------------------------------------------------------------------------
A resizable structure is requested by specifying non-zero maximum_size
in ShmemStructOpts. It gets copied to the maximum_size member in
ShmemStructDesc, ShmemIndexEnt. The question is for fixed-size
structures what should be the value maximum_size in those structures.
Setting it to the same value as the size member in the respective
structure is logical since their maximum size is the same as their
initial size. But if we do so, we need another member in
ShmemStructDesc and ShmemIndexEnt to indicate whether the structure is
resizable or not. Instead the patches set maximum_size to 0 for
fixed-size structures and non-zero for resizable structures. This way
we can check whether a structure is resizable or not by checking
whether its maximum_size is zero or not. pg_shmem_allocations view
also has a maximum_size column which has the similar characteristics.
I would like to know what others think.

3. allocated_space member in various structures and in pg_shmem_allocations view
-------------------------------------------------------------------------------
The patch adds a new member allocated_space to ShmemIndexEnt and
pg_shmem_allocations view. allocated_space to maximum_size is what
allocated_size is to size - it's the type aligned value of
maximum_size. But it also highlights the difference between the
address space allocation and the actual memory allocation. This
difference is crucial to resizable structures. However, unlike
maximum_size, we set it to a non-zero value, allocated_size, for
fixed-size structures as well since they are allocated the same amount
of space as their allocated_size. While this seems logically correct
to me, some may find maximum_size to be zero but allocated_space to be
non-zero for fixed-size structures a bit weird. I would like to know
what others think.

As a minor point, setting allocated_space to allocated_size makes the
calculations in pg_shmem_allocations() a bit easier. However, that can
be fixed trivially.

As a side question, do we want to allow users to specify minimum_size
in ShmemStructOpts for resizable structures? Resizing memory lower
than that would be prohibited. For fixed sized structures,
minimum_size would be same as size and also maximum_size. For now, it
seems only for the sanity checks, but it could be seen as a useful
safety feature. A difference in maximum_size and minimum_size would
indicate that the structure is resizable.

Considering 2 and 3 together, we have the following options
a. As implemented in patch and clarified in documentation.
b. Set maximum_size to size and allocated_space to allocated_size for
fixed-size structures, but add a new member to indicate whether the
structure is resizable or not.
c. Set maximum_size and allocated_space to zero for fixed-size
structures and explicitly mention it in the documentation.

4. to mprotect or not to mprotect
---------------------------------
If memory beyond the current size of a resizable structure is
accessed, it won't cause any segfault or bus error. When writing
memory will be simply allocated and when reading, it will return
zeroes if memory is not allocated yet. mprotect'ing the memory beyond
the current size of a resizable structure to PROT_NONE can prevent
accidental access to unallocated memory (sans page boundaries), but it
needs to be done in every backend process which requires a
synchronization mechanism beyond the scope of shmem.c. Hence the patch
does not use mprotect. A subsystem will require some higher level
synchronization mechanism between users of the structure and the
process which resizes it. That synchronization mechanism can be used
to mprotect the memory, if required. I have documented this, but I
would like to know whether we should provide an API in shmem.c to
mprotect.

6. Tests
-------
The patch adds a new test module resizable_shmem which tests the
resizable shared memory feature. Also it adds a test case to the
test_shmem module to make sure that the fixed-size shared memory
structures can not be resized. I think the resizable_shmem module
should be merged into test_shmem. But I have kept these two separate
for ease of review. Please let me know if you also think they should
be merged.

I have self-reviewed the tests a few times, fixing issues and
adjusting the test and module code. But it could help with some more
review. However, I wanted to get the patch out for review, given the
looming deadline. Similarly for the commit message.

I am adding this to CF so that it gets some CI coverage especially on
the platforms which do not support resizable shared memory.

-- 
Best Wishes,
Ashutosh Bapat

  55. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-05T09:06:27Z

    On Sun, 5 Apr 2026, 07:59 Ashutosh Bapat, <ashutosh.bapat.oss@gmail.com> wrote:
>
> On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > I will post my resizable shmem structures patch in a separate email in
> > this thread but continue to review your patches.
> >
>
> Attached is your patchset (0001 - 0014) + resizable shared memory
> structures patchset 0015.
>
> Resizable shared memory structures
> ============================
>
> When allocating memory to the requested shared structures, we allocate
> space for each structure. In mmap'ed shared memory, the memory is
> allocated against those structures only when those structures are
> initialized.
> Resizable shared memory structures are simply allocated maximum space
> when that happens. The function which initializes the structure is
> expected to initialize only the memory worth its initial size. When
> resizing the structure memory is freed or allocated against the
> reserved space depending upon the new size. This allows the structures
> to be resized while keeping their starting address stable which is a
> hard requirement in PostgreSQL.
>
> Resizable shared memory feature depends upon the existence of function
> madvise() and constants MADV_REMOVE and MADV_WRITE_POPULATE.
>
> On the platforms which do not have these, we disable this feature at
> compile time. The commit introduces a compile time flag
> HAVE_RESIZABLE_SHMEM which is defined if MADV_REMOVE and
> MADV_WRITE_POPULATE exist. We don't check the existence of madvise
> separately, since the existence of the constants implies the existence
> of the function.
>
> HAVE_RESIZABLE_SHMEM is not defined in EXEC_BACKEND builds since
> that's largely used for Windows where the APIs to free and allocate
> memory from and to a given address space are not known to the author
> right now. Given that PostgreSQL is used widely on Linux, providing
> this feature on Linux covers benefits most of its users. Once we
> figure out the required Windows APIs, we will support this feature on
> Windows as well.
>
> The feature is also not available when Sys-V shared memory is used
> even on Linux since we do not know whether required Sys-V APIs exist;
> mostly they don't. Since that combination is only available for
> development and testing, not supporting the feature isn't going to
> impact PostgreSQL users much.
>
> Using HAVE_RESIZABLE_SHMEM we disable compiling the code related to
> resizable shared memory structures on the platforms which do not
> support the feature. But we also have run time checks to disable this
> feature when Sys-V shared memory is used. In order to know whether a
> given instance of a running server supports resizable structures, we
> have introduced GUC have_resizable_shmem.

I'm not opposed to HAVE_RESIZABLE_SHMEM, but is it universal enough on
its platforms to make it part of the exposed ABI for Shmem? I think
that we should expose the same functions and structs, and just have
the shmem internals throw an error if the configuration used by the
user implies the user wants to update shmem sizing when the system
doesn't support it. That would avoid extensions having to recompile
between have/have not systems that have an otherwise compatible ABI;
especially when those extensions don't actually need the resizeable
part of the shmem system.

> Following points are up for discussion
> =============================
>
> 1. calculation of allocated_size of resizable structures
> --------------------------------
> The memory allocated to that structure is the
> {maximum size of the structure} - {total size of unallocated pages}. I
> think setting allocated_size to the actually allocated memory is more
> accurate than {current size of the structure} + {alignment} which does
> not reflect the actual memory allocated to the structure. I would like
> to know what others think.

I agree: For allocated_size, it should be the max size of the
structure (+alignment, if any), minus the total size of its
deallocated pages.

Nit: I think "reserved"/"space_reserved" is a better descriptor than
"allocated_space", as "allocated_space" could reasonably imply the
memory isn't available to the OS.

> 2. maximum_size member in various structures and in pg_shmem_allocations view
> -----------------------------------------------------------------------------
> A resizable structure is requested by specifying non-zero maximum_size
> in ShmemStructOpts. It gets copied to the maximum_size member in
> ShmemStructDesc, ShmemIndexEnt. The question is for fixed-size
> structures what should be the value maximum_size in those structures.
> Setting it to the same value as the size member in the respective
> structure is logical since their maximum size is the same as their
> initial size.

Note that currently, your patch rejects the case where resizeable
structs are initialized at their maximum size:

> +++ b/src/backend/storage/ipc/shmem.c

> +#ifdef HAVE_RESIZABLE_SHMEM
> +    if (options->maximum_size > 0 && options->size >= options->maximum_size)
> +        elog(ERROR, "resizable shared memory structure \"%s\" should have maximum size (%zd) greater than size (%zd)",
> +             options->name, options->maximum_size, options->size);

It'd need to check 'options->size  >  options->maximum_size' to allow
max-sized initialization to succeed here without erroring.

> But if we do so, we need another member in
> ShmemStructDesc and ShmemIndexEnt to indicate whether the structure is
> resizable or not. Instead the patches set maximum_size to 0 for
> fixed-size structures and non-zero for resizable structures. This way
> we can check whether a structure is resizable or not by checking
> whether its maximum_size is zero or not. pg_shmem_allocations view
> also has a maximum_size column which has the similar characteristics.
> I would like to know what others think.

I think that shmem allocations can set

.size for the initial size, and
.minimum_size/.maximum_size for configuring resizeability;

The latter fields can then be initialized with .size if they're 0.


> 3. allocated_space member in various structures and in pg_shmem_allocations view
> -------------------------------------------------------------------------------
> The patch adds a new member allocated_space to ShmemIndexEnt and
> pg_shmem_allocations view. allocated_space to maximum_size is what
> allocated_size is to size - it's the type aligned value of
> maximum_size. But it also highlights the difference between the
> address space allocation and the actual memory allocation. This
> difference is crucial to resizable structures. However, unlike
> maximum_size, we set it to a non-zero value, allocated_size, for
> fixed-size structures as well since they are allocated the same amount
> of space as their allocated_size. While this seems logically correct
> to me, some may find maximum_size to be zero but allocated_space to be
> non-zero for fixed-size structures a bit weird. I would like to know
> what others think.

I'd prefer to have consistent values; constant-sized structs are no
different from resizable structs whose min/max size equal their
current size. The only alternative that I think could be considered
correct is returning NULL for those, but zero is definitely wrong.

Note that returning min/max=size would also allow for better
aggregations on pg_shmem_allocations columns.

Note: if we expose minimum_size, we may also want to expose
min_allocated_size (i.e., the full reservation minus the size of
MADV_REMOVEd pages when the shmem allocation is min-sized).

> As a side question, do we want to allow users to specify minimum_size
> in ShmemStructOpts for resizable structures? Resizing memory lower
> than that would be prohibited.  For fixed sized structures,
> minimum_size would be same as size and also maximum_size.

I think it would be useful, if only to inform users and developers
about this in e.g. pg_shmem_allocations.

>  For now, it
> seems only for the sanity checks, but it could be seen as a useful
> safety feature. A difference in maximum_size and minimum_size would
> indicate that the structure is resizable.

I think that's the right approach.

> 4. to mprotect or not to mprotect
> ---------------------------------
> If memory beyond the current size of a resizable structure is
> accessed, it won't cause any segfault or bus error. When writing
> memory will be simply allocated and when reading, it will return
> zeroes if memory is not allocated yet. mprotect'ing the memory beyond
> the current size of a resizable structure to PROT_NONE can prevent
> accidental access to unallocated memory (sans page boundaries), but it
> needs to be done in every backend process which requires a
> synchronization mechanism beyond the scope of shmem.c. Hence the patch
> does not use mprotect.

It seems to me that the synchronization is a crucial component of
resizing; isn't it bad if shmem structs can suddenly without
synchronization contain zeroes?

> A subsystem will require some higher level
> synchronization mechanism between users of the structure and the
> process which resizes it. That synchronization mechanism can be used
> to mprotect the memory, if required. I have documented this, but I
> would like to know whether we should provide an API in shmem.c to
> mprotect.

I think we should; I think it would simplify and deduplicate external
code that needs to mark the pages PROT_NONE, and centralize OS page
calculations to within the shmem subsystem.
It'd also allow checks that validate that the pages marked with
PROT_NONE are 1) within a shmem allocation and 2) currently not in use
by that shmem allocation.

(Was there a point 5. for discussion? I can't find it)

(This is where I ran out of time for these questions, sorry I didn't
get to point 6)


Kind regards,

Matthias van de Meent

  56. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T10:03:49Z

    On 27/03/2026 02:51, Heikki Linnakangas wrote:
> On 25/03/2026 20:37, Robert Haas wrote:
>> On Sat, Mar 21, 2026 at 8:14 PM Heikki Linnakangas <hlinnaka@iki.fi> 
>> wrote:>> Shmem callbacks
>>> ---------------
>>>
>>> I separated the request/init/fn callbacks from the structs. There's now
>>> a concept of "shmem callbacks", which you register in _PG_init(). For
>>> example:
>>>
>>> static void pgss_shmem_request(void *arg);
>>> static void pgss_shmem_init(void *arg);
>>>
>>> static const ShmemCallbacks pgss_shmem_callbacks = {
>>>       .request_fn = pgss_shmem_request,
>>>       .init_fn = pgss_shmem_init,
>>>       .attach_fn = NULL, /* no special attach actions needed */
>>> };
>>
>> What's the advantage of coupling the functions together this way, vs.
>> just registering each callback individually?
> 
> One reason is to support allocations after postmaster startup. The 
> RegisterShmemCallbacks() call ties together all the resources requested 
> by the request_fn callback, with the the init_fn or attach_fn callbacks 
> that will later initialize/attach them. The init_fn/attach_fn callbacks 
> are called only after *all* the resources requested by the request_fn 
> callback have been initialized, and it holds a lock while doing all that.
> 
> If the callbacks were registered separately, shmem.c wouldn't know when 
> to call the init_fn/attach_fn. There's no problem during postmaster or 
> backend startup, because we run all init_fn or attach_fn callbacks in 
> the whole system, after requesting all the resources, but after startup, 
> you must only call the callbacks related to the newly-requested resources.
> 
> Aside from that after-startup allocation issue, though, IMO the 
> ShmemCallbacks struct makes it more clear that the callbacks are meant 
> to work together on the same resources.
> 
> One way to think of this is that all the resources requested by the 
> request_fn callback are implicitly part of the same "subsystem", and 
> need to be initialized/attached to together. We discussed that before, 
> and I still wonder if we should make that concept of a subsystem more 
> explicit. If we just renamed ShmemCallbacks to ShmemSubsystem, and give 
> each subsystem a name, it'd look like this:
> 
> static void pgss_shmem_request(void *arg);
> static void pgss_shmem_init(void *arg);
> 
> static const ShmemSubsystem pgss_shmem_subsystem = {
>      .name = "pg_stat_statements"
>      .request_fn = pgss_shmem_request,
>      .init_fn = pgss_shmem_init,
>      .attach_fn = NULL, /* no special attach actions needed */
> };
> 
> static void
> pgss_shmem_request(void *arg)
> {
>      ShmemRequestStruct(&pgssSharedStateDesc, &(ShmemRequestStructOpts) {
>           /*
>            * name is optional in this design, subsystem's name is used if
>            * not given
>            */
>          .name = "pg_stat_statements",
>          .size = sizeof(pgssSharedState),
>          .ptr = (void **) &pgss,
>      });
> }
> 
> static void
> pgss_shmem_init(void *arg)
> {
>      /* initialize contents of pgss */
>      ...
> }
> 
> void
> _PG_init(void)
> {
>      RegisterShmemSubsystem(&pgss_shmem_subsystem);
> }
> 
> 
> 
> Thinking how this might work without such a struct, registering the 
> callbacks separately, here's an alternative design:
> 
> static void pgss_shmem_request(void *arg);
> static void pgss_shmem_init(void *arg);
> 
> static void
> pgss_shmem_request(void *arg)
> {
>      ShmemRequestStruct(&pgssSharedStateDesc, &(ShmemRequestStructOpts) {
>          .name = "pg_stat_statements",
>          .size = sizeof(pgssSharedState),
>          .ptr = (void **) &pgss,
>      });
> 
>      ShmemRegisterInitCallback(&pgss_shmem_init);
>      /* no attach callback needed, but for illustration: */
>      ShmemRegisterInitCallback(&pgss_shmem_attach);
> }
> 
> static void
> pgss_shmem_init(void *arg)
> {
>      /* initialize contents of pgss */
>      ...
> }
> 
> void
> _PG_init(void)
> {
>      ShmemRegisterRequestCallback(&pgss_shmem_request);
> }
> 
> In this design, the ShmemRegisterRequestCallback() call still ties 
> together all the related resources. All the resources requested in the 
> request-callback are initialized together, and the fact that the init/ 
> attach callbacks are registered within the request callback associates 
> them with the resources. This feels a little Rube Goldbergian, with one 
> callback registering more callbacks, but would also work.

Thinking about this some more, we could also just pass the callback 
functions directly as arguments to the ShmemRegisterCallback() function, 
without the ShmemCallbacks struct. If they're all passed in one call, 
that still ties them together. The shmem.c implementation would probably 
still need the ShmemCallbacks struct, but that would be a detail 
internal to shmem.c. It would look like this:

static void pgss_shmem_request(void *arg);
static void pgss_shmem_init(void *arg);

static void
pgss_shmem_request(void *arg)
{
      ShmemRequestStruct(
          .name = "pg_stat_statements",
          .size = sizeof(pgssSharedState),
          .ptr = (void **) &pgss,
      );
}


static void
pgss_shmem_init(void *arg)
{
      /* initialize contents of pgss */
      ...
}


void
_PG_init(void)
{
      RegisterShmemSubsystem(pgss_shmem_request,
                             pgss_shmem_init,
                             NULL, /* no attach fn needed */
                             0, /* flags */);
}

This is pretty much the same as what's in the latest patch version, but 
a little less boilerplate as you don't need the ShmemCallbacks struct. 
The struct would be useful if we had needs to add lots of optional 
options in the future, but I don't think we have such needs.

- Heikki

  57. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-05T11:20:28Z

    On Sun, Apr 5, 2026 at 2:36 PM Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
>
> On Sun, 5 Apr 2026, 07:59 Ashutosh Bapat, <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
>
> I'm not opposed to HAVE_RESIZABLE_SHMEM, but is it universal enough on
> its platforms to make it part of the exposed ABI for Shmem? I think
> that we should expose the same functions and structs, and just have
> the shmem internals throw an error if the configuration used by the
> user implies the user wants to update shmem sizing when the system
> doesn't support it. That would avoid extensions having to recompile
> between have/have not systems that have an otherwise compatible ABI;
> especially when those extensions don't actually need the resizeable
> part of the shmem system.
>

I don't think I understand this fully. An extension may want to
support a structure in both modes - fixed as well as resizable
depending upon whether the latter is supported. If the structure has
maximum_size always the extension code needs to set it to 0 when the
resizable shared structure is not supported and set to actual
maximum_size when the resizable structure is supported. Without a
macro or some flag they can not do that. The flag/macro then becomes
part ABI for shmem. Am I correct? Since extension binaries need to be
built on different platforms anyway, that would automatically take
care of building with or without HAVE_RESIZABLE_SHMEM. I feel it makes
testing simpler since run time behaviour is fixed. Maybe I am missing
something. Maybe a code diff or some example platform might make it
more clear for me.

> > Following points are up for discussion
> > =============================

>
> Nit: I think "reserved"/"space_reserved" is a better descriptor than
> "allocated_space", as "allocated_space" could reasonably imply the
> memory isn't available to the OS.

Renamed it to reserved_space. New name is also less confusing with
allocation_size.


>
> Note that currently, your patch rejects the case where resizeable
> structs are initialized at their maximum size:
>
> > +++ b/src/backend/storage/ipc/shmem.c
>
> > +#ifdef HAVE_RESIZABLE_SHMEM
> > +    if (options->maximum_size > 0 && options->size >= options->maximum_size)
> > +        elog(ERROR, "resizable shared memory structure \"%s\" should have maximum size (%zd) greater than size (%zd)",
> > +             options->name, options->maximum_size, options->size);
>
> It'd need to check 'options->size  >  options->maximum_size' to allow
> max-sized initialization to succeed here without erroring.

good catch. FIxed in the attached patch.

>
> > But if we do so, we need another member in
> > ShmemStructDesc and ShmemIndexEnt to indicate whether the structure is
> > resizable or not. Instead the patches set maximum_size to 0 for
> > fixed-size structures and non-zero for resizable structures. This way
> > we can check whether a structure is resizable or not by checking
> > whether its maximum_size is zero or not. pg_shmem_allocations view
> > also has a maximum_size column which has the similar characteristics.
> > I would like to know what others think.
>
> I think that shmem allocations can set
>
> .size for the initial size, and
> .minimum_size/.maximum_size for configuring resizeability;
>
> The latter fields can then be initialized with .size if they're 0.
>
>
> > 3. allocated_space member in various structures and in pg_shmem_allocations view
> > -------------------------------------------------------------------------------
> > The patch adds a new member allocated_space to ShmemIndexEnt and
> > pg_shmem_allocations view. allocated_space to maximum_size is what
> > allocated_size is to size - it's the type aligned value of
> > maximum_size. But it also highlights the difference between the
> > address space allocation and the actual memory allocation. This
> > difference is crucial to resizable structures. However, unlike
> > maximum_size, we set it to a non-zero value, allocated_size, for
> > fixed-size structures as well since they are allocated the same amount
> > of space as their allocated_size. While this seems logically correct
> > to me, some may find maximum_size to be zero but allocated_space to be
> > non-zero for fixed-size structures a bit weird. I would like to know
> > what others think.
>
> I'd prefer to have consistent values; constant-sized structs are no
> different from resizable structs whose min/max size equal their
> current size. The only alternative that I think could be considered
> correct is returning NULL for those, but zero is definitely wrong.
>
> Note that returning min/max=size would also allow for better
> aggregations on pg_shmem_allocations columns.
>
> Note: if we expose minimum_size, we may also want to expose
> min_allocated_size (i.e., the full reservation minus the size of
> MADV_REMOVEd pages when the shmem allocation is min-sized).
>
> > As a side question, do we want to allow users to specify minimum_size
> > in ShmemStructOpts for resizable structures? Resizing memory lower
> > than that would be prohibited.  For fixed sized structures,
> > minimum_size would be same as size and also maximum_size.
>
> I think it would be useful, if only to inform users and developers
> about this in e.g. pg_shmem_allocations.
>
> >  For now, it
> > seems only for the sanity checks, but it could be seen as a useful
> > safety feature. A difference in maximum_size and minimum_size would
> > indicate that the structure is resizable.
>
> I think that's the right approach.


I also think that introducing minimum_size is useful. Let's hear from
Heikki before implementing it, in case he has a different opinion. I
am not sure about min_allocated_space though - what use do you see for
it. reserved_space is useful in pg_shmem_allocations() C function
itself and gives impact to the fully grown structure. What would
min_allocated_space give us? If at all it would be min_allocated_size
not space since reserved space will never change. But even that I am
not sure about.

>
> > 4. to mprotect or not to mprotect
> > ---------------------------------
> > If memory beyond the current size of a resizable structure is
> > accessed, it won't cause any segfault or bus error. When writing
> > memory will be simply allocated and when reading, it will return
> > zeroes if memory is not allocated yet. mprotect'ing the memory beyond
> > the current size of a resizable structure to PROT_NONE can prevent
> > accidental access to unallocated memory (sans page boundaries), but it
> > needs to be done in every backend process which requires a
> > synchronization mechanism beyond the scope of shmem.c. Hence the patch
> > does not use mprotect.
>
> It seems to me that the synchronization is a crucial component of
> resizing; isn't it bad if shmem structs can suddenly without
> synchronization contain zeroes?
>
> > A subsystem will require some higher level
> > synchronization mechanism between users of the structure and the
> > process which resizes it. That synchronization mechanism can be used
> > to mprotect the memory, if required. I have documented this, but I
> > would like to know whether we should provide an API in shmem.c to
> > mprotect.
>
> I think we should; I think it would simplify and deduplicate external
> code that needs to mark the pages PROT_NONE, and centralize OS page
> calculations to within the shmem subsystem.
> It'd also allow checks that validate that the pages marked with
> PROT_NONE are 1) within a shmem allocation and 2) currently not in use
> by that shmem allocation.

Reasonable. Let's wait for Heikki's opinion on this as well before
implementing it.

>
> (Was there a point 5. for discussion? I can't find it)

There is no point 5, just bad numbering.

>
> (This is where I ran out of time for these questions, sorry I didn't
> get to point 6)

CFBot did show some failures.

1. Makefile didn't define PGXS, fixed it.
2. Windows compiler didn't like #ifdef in the middle of function like
macro argument list C5101 and C2059. Used a conditional macro instead.
3. The test fails one one machine because RssShmem is consistently 8MB
higher than the allocated_size in all cases. I guess it is because of
huge page setting. Adding huge_pages = off to the test configuration.
I think the test can not rely on huge pages anyway since
allocated_size isn't aligned to huge page size.

--
Best Wishes,
Ashutosh Bapat

  58. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-05T14:08:09Z

    On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
>
> I will post my resizable shmem structures patch in a separate email in
> this thread but continue to review your patches.

I reviewed the SLRU patch. This is the first time I am looking at SLRU
code, so my review may not be sufficient. As far as I understand, the
patch faithfully copies the functionality from the old system to the
new system. I didn't find any issues there.

I think calls to SimpleLruRequest() reads much better than SimpleLruInit().

Both MultiXactShmemInit and MultiXactShmemAttach set
OldestMemberMXactId, OldestVisibleMXactId. In future if we add another
global variable to point to the shared memory, somebody needs to
remember to initialize it in both these functions. Maybe deduplicate
it with something like attached? Similarly for PredicateLock related
changes.

shmem_slru_init and shmem_slru_attach() also have the following
duplicate lines, which can be deduplicated in a similar fashion.
desc->shared = shared;
desc->nbanks = nbanks;
memcpy(&desc->options, options, sizeof(SlruOpts));

Including "access/slru.h" in shmem.h is circular inclusion. I am
wondering whether we need to create shmem_slru.h like shmem_hash.h to
handle shared memory APIs related to SLRU. Given that SLRU also has a
disk component, the bifurcation may not be straightforward. I haven't
looked into this aspect in detail.


--
Best Wishes,
Ashutosh Bapat

  59. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-05T14:16:51Z

    On Sun, Apr 5, 2026 at 4:50 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>

> 3. The test fails one one machine because RssShmem is consistently 8MB
> higher than the allocated_size in all cases. I guess it is because of
> huge page setting. Adding huge_pages = off to the test configuration.
> I think the test can not rely on huge pages anyway since
> allocated_size isn't aligned to huge page size.

Turning huge_pages = off didn't help. The test actually creates a
resizable shared memory structure which is 100s of MBs and adjusts
GUCs so that very minimum shared memory is allocated. This way the
resizable structure dominates the shared memory segment. Any small
variations in RssShmem because of parts of shared memory not accessed
by a backend can be ignored. Then it expects that the RssShmem of a
backend <= sum(allocated_size) from pg_shmem_allocations; usually
sum(allocated_size) - RssShmem ~= 2MB. That's not accurate but it's
the closest I can get to make sure that we do not over allocate memory
for resizable shared structures. There is something on
https://cirrus-ci.com/task/5501660157444096 which is mapping shared
memory worth 10MB, other than the main shared memory segment,
consistently in all the backends. Because of that RssShmem -
sum(allocated_size) is consistently ~= 8MB. I am not able to figure
out where that 10MB is coming from. If we could know that, we could
either disable the test on that machine or disable that allocation. On
all other CFBot VMs, the test is passing, including the platforms
where the feature is not supported.

-- 
Best Wishes,
Ashutosh Bapat

  60. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T15:07:23Z

    On 05/04/2026 02:17, Matthias van de Meent wrote:
> Formatting:
>> +    ShmemRequestHash(.name = "pg_stat_statements hash",
>> +                     .nelems = pgss_max,
>> +                     .hash_info.keysize = sizeof(pgssHashKey),
>> +                     .hash_info.entrysize = sizeof(pgssEntry),
>> +                     .hash_flags = HASH_ELEM | HASH_BLOBS,
>> +                     .ptr = &pgss_hash,
>> +        );
> (note that additional unit of indentation for the closing bracket)
> 
> Is this malformatting caused by pgindent? If so, could you see if
> there's a better way of defining ShmemRequestHash/Struct that doesn't
> have this indent as output?

Yeah, that's pgindent. Matter of taste, but I think that looks fine. An 
alternative is to put the closing bracket on the same line with the last 
argument and drop the trailing comma:

     ShmemRequestStruct(.name = "pg_stat_statements",
                        .size = sizeof(pgssSharedState),
                        .ptr = (void **) &pgss);

That looks OK to me too.

>> +    pgss->extent = 0;
>> +    pgss->n_writers = 0;
>> +    pgss->gc_count = 0;
>> +    pgss->stats.dealloc = 0;
> 
> Shmem is said to be zero-initialized, should we remove the manual
> zero-initialization?
> 
>> +    on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
> See my upthread comment about adding optional on_shmem_exit callbacks
> to ShmemCallbacks.
> 
> 0005: LGTM
> 
> 0006: I don't think it is a great idea to make the LwLock machinery
> the first to get allocation requests:
> It has the RequestNamedLWLockTranche infrastructure, which can only
> register new requests while process_shmem_requests_in_progress, and
> making it request its memory ahead of everything else is likely to
> cause an undersized tranche to be allocated.

Good catch. I think the easiest fix is to call process_shmem_requests() 
before ShmemCallRequestCallbacks() at postmaster startup. That's kind of 
how it was before: process_shmem_requests() was called before all the 
*ShmemSize() and *ShmemInit() functions in core.

> You could make sure that this isn't an issue by maintaining a flag
> in lwlock.c that's set when the shmem request is made (and reset on
> shmem exit), which must be false when RequestNamedLWLockTranche() is
> called, and if not then it should throw an error.
I'll change it so that the number of locks calculated in 
LWLockShmemRequest() is stored in a global variable, and 
LWLockShmemInit() has an Assert() to cross-checks with that. That 
catches the bug and seems like a good cross-check in general.

This isn't the only place where we need to pass information from the 
request callback to the init callback. I've used a global variable for 
that here, and also between ProcGlobalShmemRequest() and 
ProcGlobalShmemRequest(). An alternative might be to use the callback 
arg pointers that are currently unused, but I'm not sure how to make 
that ergonomic. The current 'arg' isn't very helpful for that, so 
perhaps the signatures should look like this instead:

static void
LWLockShmemRequest(Datum *init_arg)
{
     int         numLocks;

     numLocks = NUM_FIXED_LWLOCKS + NumLWLocksForNamedTranches();

     /* pass the calculated numLocks value to LWLockShmemInit() */
     *init_arg = Int32GetDatum(numLocks);

     ...
}

static void
LWLockShmemInit(Datum init_arg)
{
     int         numLocks = DatumGetIn32(numLocks);

     ...
}

If you need to pass more than a single Datum, you can allocate a struct 
and pass it via PointerGetDatum(). At that point global variables might 
feel simpler again though.

> 0010: Not looked at everything yet, but a few comment:
> 
>> +++ b/src/include/access/slru.h
> 
> With the changes in the signatures for most/all SLRU functions from a
> hidden-by-typedef pointer to a visible pointer type, maybe this could
> be an opportunity to swap them to `const SlruDesc *ctl` wherever
> possible? I don't think there are many backend-local changes that
> happen to SlruDescs once we've properly started the backend. I'm happy
> to provide an incremental patch if you'd like me to spend cycles on it
> if you're busy.

Yeah sounds like a good idea.

>> +++ b/src/backend/access/transam/clog.c
> 
>> +    SimpleLruRequest(.desc = &XactSlruDesc,
>> +                     .name = "transaction",
>> +                     .Dir = "pg_xact",
>> +                     .long_segment_names = false,
>> +
>> +                     .nslots = CLOGShmemBuffers(),
>> +                     .nlsns = CLOG_LSNS_PER_PAGE,
>> +
>> +                     .sync_handler = SYNC_HANDLER_CLOG,
>> +                     .PagePrecedes = CLOGPagePrecedes,
>> +                     .errdetail_for_io_error = clog_errdetail_for_io_error,
> 
> That awfully inconsistent field name styling is ... awful, but not
> this patch's fault. If something can be done about it in a cheap
> fashion in this patch, that'd be great, but I won't hold it against
> you if that's skipped.

:-D.

- Heikki

  61. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T15:39:37Z

    On 04/04/2026 19:32, Ashutosh Bapat wrote:
> test_shmem declares MODULE_big and OBJS which seems to be old
> fashioned, newer modules seem to be using MODULES. 

I don't think it's a matter of old or new. MODULE_big is used when you 
have multiple .o that are linked together into one .so file, while 
MODULES is used if each .o file is linked into a separate .so file. If 
there's only one .o file and .so file, then it doesn't really matter 
which you use, and I think we have examples of both.

> Also it should use NO_INSTALLCHECK.
> 
> /*
> * Alignment of the starting address. If not set, defaults to cacheline
> * boundary. Must be a power of two.
> */
> size_t alignment;
> 
> We don't seem to enforce the "must be a power of two" rule anywhere.
> We should at least validate it.

Will add.

- Heikki

  62. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T15:50:11Z

    On 05/04/2026 02:17, Matthias van de Meent wrote:
>> +    pgss->extent = 0;
>> +    pgss->n_writers = 0;
>> +    pgss->gc_count = 0;
>> +    pgss->stats.dealloc = 0;
> 
> Shmem is said to be zero-initialized, should we remove the manual
> zero-initialization?

Yeah, perhaps. We already had initialization like this in many places, 
while others relied on the implicit initialization. Some places even do 
just this:

void
LogicalDecodingCtlShmemInit(void)
{
	bool		found;

	LogicalDecodingCtl = ShmemInitStruct("Logical decoding control",
										 LogicalDecodingCtlShmemSize(),
										 &found);

	if (!found)
		MemSet(LogicalDecodingCtl, 0, LogicalDecodingCtlShmemSize());
}

I think there are two directions we could go here:

1. Document that the memory is zeroed, and you can rely on it. Remove 
silly initializations like that in LogicalDecodingCtlShmemInit(). In 
other places the explicitly zero-initialization might have documentation 
value though.

2. Require the init functions to explicitly zero the memory. Document it 
and add valgrind checks.

I'm inclined to go with 1. But in the name of avoiding scope creep, not 
as part of these patches.

- Heikki

  63. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T16:13:00Z

    On 26/03/2026 20:31, Daniel Gustafsson wrote:
>> On 22 Mar 2026, at 01:14, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> * The request_fn callback is called in postmaster startup, at the same stage as the old shmem_request callback was. But in EXEC_BACKEND mode, it's *also* called in each backend.
> 
> Should the request_fn be told, via an argument, from where it is called?  It
> can be figured out but it's cleaner if all implementations will do it in the
> same way.  I don't have a direct case in mind where it would be needed, but I
> was recently digging into SSL passphrase reloading which has failure cases
> precisely becasue of this so am thinking out loud to avoid similar problems
> here.

Hmm, you mean adding an argument along the lines of:

static void
pgss_shmem_request(void *arg, bool attaching)
{
     ...
}

Perhaps. The idea is that a request callback should generally do the 
exact same thing whether it's called from postmaster or from backend 
startup, though. I worry that an argument like that makes it too 
tempting to have different logic. That said, there are a couple of 
places where I'm using IsUnderPostmaster for that purpose. For example, 
I have this in lwlock.c (in latest version I'm currently working on that 
I haven't posted yet):

/* Size of MainLWLockArray.  Only valid in postmaster. */
static int	num_main_array_locks;

/*
  * Request shmem space for user-defined tranches and the main LWLock array.
  */
static void
LWLockShmemRequest(void *arg)
{
     size_t        size;

     /* Space for user-defined tranches */
     ShmemRequestStruct(.name = "LWLock tranches",
                        .size = sizeof(LWLockTrancheShmemData),
                        .ptr = (void **) &LWLockTranches,
         );

     /* Space for the LWLock array */
     if (!IsUnderPostmaster)
     {
         num_main_array_locks = NUM_FIXED_LWLOCKS + 
NumLWLocksForNamedTranches();
         size = num_main_array_locks * sizeof(LWLockPadded);
     }
     else
         size = SHMEM_ATTACH_UNKNOWN_SIZE;

     ShmemRequestStruct(.name = "Main LWLock array",
                        .size = size,
                        .ptr = (void **) &MainLWLockArray,
         );
}

- Heikki

  64. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-05T16:23:43Z

    On Sun, Apr 5, 2026 at 7:38 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> >
> > I will post my resizable shmem structures patch in a separate email in
> > this thread but continue to review your patches.
>
> I reviewed the SLRU patch. This is the first time I am looking at SLRU
> code, so my review may not be sufficient. As far as I understand, the
> patch faithfully copies the functionality from the old system to the
> new system. I didn't find any issues there.
>
> I think calls to SimpleLruRequest() reads much better than SimpleLruInit().
>
> Both MultiXactShmemInit and MultiXactShmemAttach set
> OldestMemberMXactId, OldestVisibleMXactId. In future if we add another
> global variable to point to the shared memory, somebody needs to
> remember to initialize it in both these functions. Maybe deduplicate
> it with something like attached? Similarly for PredicateLock related
> changes.

Sorry, I attached the wrong patch. Here's the right patch.

-- 
Best Wishes,
Ashutosh Bapat

  65. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T18:35:06Z

    On 05/04/2026 19:23, Ashutosh Bapat wrote:
>> Both MultiXactShmemInit and MultiXactShmemAttach set
>> OldestMemberMXactId, OldestVisibleMXactId. In future if we add another
>> global variable to point to the shared memory, somebody needs to
>> remember to initialize it in both these functions. Maybe deduplicate
>> it with something like attached? Similarly for PredicateLock related
>> changes.
> 
> Sorry, I attached the wrong patch. Here's the right patch.

Gotcha, yeah I've thought about that too. I even considered making 
ShmemInitRequested() automatically call all the attach callbacks after 
initialization, even in !EXEC_BACKEND builds. That way, you could put 
the backend-private steps only in the attach function, and have 
automatically be called in the postmaster too. I decided against it, 
because only few subsystems need the attach callback at all, and many of 
them need to do the "local" steps earlier in the init callback anyway. 
For example, XLOGShmemInit() uses the WALInsertLocks variable inside the 
function already.

In the end I decided it's OK as it is. I'm not too worried about the 
duplicated code in multixact.c and predicate.c, they fit in the same 
screen in an editor so it's pretty easy to see that they are duplicated 
for a reason. With more complicated logic it would be a different story.

- Heikki

  66. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T18:58:15Z

    On 03/04/2026 01:10, Matthias van de Meent wrote:
> While I do think it's an improvement over the current APIs, the
> improvement seems to be mostly concentrated in the RequestStruct/Hash
> department, with only marginal improvements in RegisterShmemCallbacks.
> I feel like it's missing the important part: I'd like
> direct-from-_PG_init() ShmemRequestStruct/Hash calls. If
> ShmemRequestStruct/Hash had a size callback as alternative to the size
> field (which would then be called after preload_libraries finishes)
> then that would be sufficient for most shmem allocations, and it'd
> simplify shmem management for most subsystems.
> We'd still need the shmem lifecycle hooks/RegisterShmemCallbacks to
> allow conditionally allocated shmem areas (e.g. those used in aio),
> but I think that, in general, we shouldn't need a separate callback
> function just to get started registering shmem structures.

I kind of started from that thought too, but the design has since 
evolved to what it is. Robert in particular was skeptical of that 
approach, and I think I've come around on most of his feedback.

A per-struct size callback isn't very ergonomic in places like 
LockManagerShmemRequest(), which derives the size of two different 
things from the same calculated value. You'd need to repeat the 
calculation for each one, or pass it through global variables or 
something. PredicateLockShmemInit() is another extreme example. It 
already has that problem to some extent, and already uses global 
variables (I'm all ears if you have suggestions to improve it!) but I 
feel that with a size callback this kind of stuff would get even harder.

Another reason is that it's good to have only one way of doing the 
initialization, instead of one simple way and a different way for more 
complex scenarios. If the simple way was vastly simpler, it might be 
worth it, but I don't think there's that much difference here.


BTW, I also considered another way of initializing structs for simple 
cases: instead of providing an init callback function, you could provide 
the struct contents directly in the ShmemRequestStruct() call. To pick a 
random example, slotsync.c currently looks like this:

static void
SlotSyncShmemRequest(void *arg)
{
     ShmemRequestStruct(.name = "Slot Sync Data",
                        .size = sizeof(SlotSyncCtxStruct),
                        .ptr = (void **) &SlotSyncCtx,
     );
}

static void
SlotSyncShmemInit(void *arg)
{
     memset(SlotSyncCtx, 0, sizeof(SlotSyncCtxStruct));
     SlotSyncCtx->pid = InvalidPid;
     SpinLockInit(&SlotSyncCtx->mutex);
}


But it could look like this instead:

static void
SlotSyncShmemRequest(void *arg)
{
     SlotSyncCtxStruct init_content;

     memset(SlotSyncCtx, 0, sizeof(SlotSyncCtxStruct));
     init_content.pid
     SpinLockInit(&SlotSyncCtx->mutex);

     ShmemRequestStruct(.name = "Slot Sync Data",
                        .size = sizeof(SlotSyncCtxStruct),
                        .ptr = (void **) &SlotSyncCtx,
                        .init_content = &init_content,
		);
}

That'd only work for small, simple structs, though. Since the initial 
contents would be copied in this model, it won't work for anything with 
pointers to itself, for example. And it's not much less code after all.

- Heikki

  67. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-05T19:05:04Z

    On Sun, 5 Apr 2026 at 13:20, Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Sun, Apr 5, 2026 at 2:36 PM Matthias van de Meent
> <boekewurm+postgres@gmail.com> wrote:
> >
> > On Sun, 5 Apr 2026, 07:59 Ashutosh Bapat, <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
> > > <ashutosh.bapat.oss@gmail.com> wrote:
> > > >
> >
> > I'm not opposed to HAVE_RESIZABLE_SHMEM, but is it universal enough on
> > its platforms to make it part of the exposed ABI for Shmem? I think
> > that we should expose the same functions and structs, and just have
> > the shmem internals throw an error if the configuration used by the
> > user implies the user wants to update shmem sizing when the system
> > doesn't support it. That would avoid extensions having to recompile
> > between have/have not systems that have an otherwise compatible ABI;
> > especially when those extensions don't actually need the resizeable
> > part of the shmem system.
> >
>
> I don't think I understand this fully. An extension may want to
> support a structure in both modes - fixed as well as resizable
> depending upon whether the latter is supported. If the structure has
> maximum_size always the extension code needs to set it to 0 when the
> resizable shared structure is not supported and set to actual
> maximum_size when the resizable structure is supported. Without a
> macro or some flag they can not do that. The flag/macro then becomes
> part ABI for shmem. Am I correct?

That's not quite what I meant.

With your patch, the size and field offsets in `struct
ShmemStructOpts` changes depending only on HAVE_RESIZABLE_SHMEM, as
does function's availability. This means that an extension that's
built without HAVE_RESIZABLE_SHMEM (an otherwise identical system)
can't correctly be loaded into a server that does have
HAVE_RESIZABLE_SHMEM defined - or at least it'll misbehave when it
tries to use the new shmem system without trying out resizeable areas.

If instead the fields used for definining resizable shmem areas (and
the relevant functions) are always defined, but with runtime checks to
make sure that in !HAVE_RESIZEABLE_SHMEM nobody tries to use the
resizing functionality, then that'd reduce the unchecked hidden
incompatibility; assuming that no extension manually does memory
management syscall operations on those shmem areas.

> Since extension binaries need to be
> built on different platforms anyway, that would automatically take
> care of building with or without HAVE_RESIZABLE_SHMEM.  I feel it makes
> testing simpler since run time behaviour is fixed. Maybe I am missing
> something. Maybe a code diff or some example platform might make it
> more clear for me.

I'm not entirely sure it would be automatic. Is it guaranteed that
HAVE_RESIZABLE_SHMEM won't change over the lifetime of any
distribution's platform? Because it's definitely not apparent to me
that rebuilding the new server version against an upgraded platform
(now possibly with HAVE_RESIZABLE_SHMEM) should also mean rebuilding
the extensions that have been built against a previous minor version
(without HAVE_RESIZABLE_SHMEM).

> > >  For now, it
> > > seems only for the sanity checks, but it could be seen as a useful
> > > safety feature. A difference in maximum_size and minimum_size would
> > > indicate that the structure is resizable.
> >
> > I think that's the right approach.
>
>
> I also think that introducing minimum_size is useful. Let's hear from
> Heikki before implementing it, in case he has a different opinion. I
> am not sure about min_allocated_space though - what use do you see for
> it. reserved_space is useful in pg_shmem_allocations() C function
> itself and gives impact to the fully grown structure. What would
> min_allocated_space give us? If at all it would be min_allocated_size
> not space since reserved space will never change. But even that I am
> not sure about.

I'd say it's mostly interesting for people looking at or debugging
shmem allocations. Which isn't a huge group of developers or DBAs, but
if we're exposing data like this, and are going to allow resizing,
then someone could see some benefits from this.

E.g., it may be useful to have the information to see how low the
currently running server can scale down its memory usage, so that the
admin can see whether a reboot is required if they want to allow it to
scale it down further (assuming there's a lower limit for allocations
- some shmem structs may have a lower scaling limit defined at
startup, while others may be able to scale linearly from 0 to 100)


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

  68. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T19:35:58Z

    On 04/04/2026 16:51, Matthias van de Meent wrote:
>> +++ b/src/include/storage/shmem.h
>> +/*
>> + * Shared memory is reserved and allocated in stages at postmaster startup,
>> + * and in EXEC_BACKEND mode, there's some extra work done to "attach" to them
>> + * at backend startup.  ShmemCallbacks holds callback functions that are
>> + * called at different stages.
>> + */
>> +typedef struct ShmemCallbacks
> 
> Maybe this should also have the opportunity for a (before_)shmem_exit callback?

Hmm, yeah, perhaps, but I'm going to skip that for now. We already have 
a mechanism for shmem-exit callbacks, and I'm not sure how that would 
plug into this. I think we can do that later if it turns out to be a 
good idea, and I don't think it changes the parts that's included in the 
patches now.

>> + * on-demaind in a backend.  If a subsystem sets this flag, the callbacks are
>> + * called immediately after registration, to initialize or attach to the
>> + * requested shared memory areas.
> 
> Ideally we only immediately call the callbacks if we're under
> postmaster, or in a standalone backend; we shouldn't allocate shmem
> for some preloaded libraries that set this flag, at least not ahead of
> loading all preload libraries.

Right, the SHMEM_CALLBACKS_ALLOW_AFTER_STARTUP flag doesn't do anything 
if called during shared_preload_libraries processing. I'll re-word the 
comment to clarify that.

> While it's mostly mechanical changes, it did make me notice the rather
> annoying allocation patterns by XLOGShmemRequest. It allocates various
> types of data in one go (which, in principle, is fine) but in doing so
> it adds its own alignment tricks etc, and I'm not super stoked about
> that. If time allows, could we clean that up?

I'm not going to cram it into these patches, but +1.

- Heikki

  69. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-05T19:58:52Z

    On Sun, 5 Apr 2026 at 17:07, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 05/04/2026 02:17, Matthias van de Meent wrote:
> > Is this malformatting caused by pgindent? If so, could you see if
> > there's a better way of defining ShmemRequestHash/Struct that doesn't
> > have this indent as output?
>
> Yeah, that's pgindent. Matter of taste, but I think that looks fine. An
> alternative is to put the closing bracket on the same line with the last
> argument and drop the trailing comma:
>
>      ShmemRequestStruct(.name = "pg_stat_statements",
>                         .size = sizeof(pgssSharedState),
>                         .ptr = (void **) &pgss);
>
> That looks OK to me too.

Then let's keep it as per the v11 patch with ugly closing indents --
hopefully someone will fix pgindent in the future, but that's not the
job of this patch.

> > You could make sure that this isn't an issue by maintaining a flag
> > in lwlock.c that's set when the shmem request is made (and reset on
> > shmem exit), which must be false when RequestNamedLWLockTranche() is
> > called, and if not then it should throw an error.
> I'll change it so that the number of locks calculated in
> LWLockShmemRequest() is stored in a global variable, and
> LWLockShmemInit() has an Assert() to cross-checks with that. That
> catches the bug and seems like a good cross-check in general.

Thanks for fixing this!

> > 0010: Not looked at everything yet, but a few comment:
> >
> >> +++ b/src/include/access/slru.h
> >
> > With the changes in the signatures for most/all SLRU functions from a
> > hidden-by-typedef pointer to a visible pointer type, maybe this could
> > be an opportunity to swap them to `const SlruDesc *ctl` wherever
> > possible? I don't think there are many backend-local changes that
> > happen to SlruDescs once we've properly started the backend. I'm happy
> > to provide an incremental patch if you'd like me to spend cycles on it
> > if you're busy.
>
> Yeah sounds like a good idea.

Attached 2 incremental patches:
0001 constifies the expected function arguments, which I propose to include; and
0002 adds 'type* const struct fields' in SlruShared.

0002 was not requested, but it looked feasible to at least try it out
in this subsystem. It might be interesting, but you're free to drop
it.


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

  70. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T20:06:38Z

    Here's patch version 12 [*]. I believe I've addressed all the feedback, 
and I feel this is in pretty good shape now. There hasn't been any big 
design changes lately.

One notable change is that I replaced the separate 
{request|init|attach}_fn_arg fields in ShmemCallbacks with a single 
'opaque_arg' field, and added a brief comment to it. You both commented 
on whether we need that at all, and maybe you're right that we don't, 
but at least it's now just one field rather than three. As before, 
callers can simply ignore it if they don't need it.

[*] also available at 
https://github.com/hlinnaka/postgres/tree/shmem-init-refactor-12

- Heikki

  71. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-05T23:28:47Z

    On 05/04/2026 23:06, Heikki Linnakangas wrote:
> Here's patch version 12 [*]. I believe I've addressed all the feedback, 
> and I feel this is in pretty good shape now. There hasn't been any big 
> design changes lately.
> 
> One notable change is that I replaced the separate {request|init|attach} 
> _fn_arg fields in ShmemCallbacks with a single 'opaque_arg' field, and 
> added a brief comment to it. You both commented on whether we need that 
> at all, and maybe you're right that we don't, but at least it's now just 
> one field rather than three. As before, callers can simply ignore it if 
> they don't need it.

After another round of comment cleanups and such, committed. Thanks!

- Heikki

  72. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-06T13:53:19Z

    On Mon, Apr 6, 2026 at 4:58 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 05/04/2026 23:06, Heikki Linnakangas wrote:
> > Here's patch version 12 [*]. I believe I've addressed all the feedback,
> > and I feel this is in pretty good shape now. There hasn't been any big
> > design changes lately.
> >
> > One notable change is that I replaced the separate {request|init|attach}
> > _fn_arg fields in ShmemCallbacks with a single 'opaque_arg' field, and
> > added a brief comment to it. You both commented on whether we need that
> > at all, and maybe you're right that we don't, but at least it's now just
> > one field rather than three. As before, callers can simply ignore it if
> > they don't need it.
>
> After another round of comment cleanups and such, committed. Thanks!

Thanks. Attached are rebased patches.

0001 is the same patch as submitted before to support resizable shared
memory structures.

0002 changes resizable_shmem_usage() in the resizable_shmem test
module to use /proc/self/smaps as suggested by Andres offline. Using
smaps the function estimates the actual memory mapped against the main
shared memory segment. Expecting this to fix failure reported on
https://cirrus-ci.com/task/5501660157444096.

0003 adds some more diagnostic about shared memory segments in the
server log to debug the failure in case it appears even with 0002

0004 addresses Matthias's comments in the discussion below

On Mon, Apr 6, 2026 at 12:35 AM Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
>
> On Sun, 5 Apr 2026 at 13:20, Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Sun, Apr 5, 2026 at 2:36 PM Matthias van de Meent
> > <boekewurm+postgres@gmail.com> wrote:
> > >
> > > On Sun, 5 Apr 2026, 07:59 Ashutosh Bapat, <ashutosh.bapat.oss@gmail.com> wrote:
> > > >
> > > > On Sun, Apr 5, 2026 at 11:18 AM Ashutosh Bapat
> > > > <ashutosh.bapat.oss@gmail.com> wrote:
> > > > >
> > >
> > > I'm not opposed to HAVE_RESIZABLE_SHMEM, but is it universal enough on
> > > its platforms to make it part of the exposed ABI for Shmem? I think
> > > that we should expose the same functions and structs, and just have
> > > the shmem internals throw an error if the configuration used by the
> > > user implies the user wants to update shmem sizing when the system
> > > doesn't support it. That would avoid extensions having to recompile
> > > between have/have not systems that have an otherwise compatible ABI;
> > > especially when those extensions don't actually need the resizeable
> > > part of the shmem system.
> > >
> >
> > I don't think I understand this fully. An extension may want to
> > support a structure in both modes - fixed as well as resizable
> > depending upon whether the latter is supported. If the structure has
> > maximum_size always the extension code needs to set it to 0 when the
> > resizable shared structure is not supported and set to actual
> > maximum_size when the resizable structure is supported. Without a
> > macro or some flag they can not do that. The flag/macro then becomes
> > part ABI for shmem. Am I correct?
>
> That's not quite what I meant.
>
> With your patch, the size and field offsets in `struct
> ShmemStructOpts` changes depending only on HAVE_RESIZABLE_SHMEM, as
> does function's availability. This means that an extension that's
> built without HAVE_RESIZABLE_SHMEM (an otherwise identical system)
> can't correctly be loaded into a server that does have
> HAVE_RESIZABLE_SHMEM defined - or at least it'll misbehave when it
> tries to use the new shmem system without trying out resizeable areas.
>
> If instead the fields used for definining resizable shmem areas (and
> the relevant functions) are always defined, but with runtime checks to
> make sure that in !HAVE_RESIZEABLE_SHMEM nobody tries to use the
> resizing functionality, then that'd reduce the unchecked hidden
> incompatibility; assuming that no extension manually does memory
> management syscall operations on those shmem areas.
>
> > Since extension binaries need to be
> > built on different platforms anyway, that would automatically take
> > care of building with or without HAVE_RESIZABLE_SHMEM.  I feel it makes
> > testing simpler since run time behaviour is fixed. Maybe I am missing
> > something. Maybe a code diff or some example platform might make it
> > more clear for me.
>
> I'm not entirely sure it would be automatic. Is it guaranteed that
> HAVE_RESIZABLE_SHMEM won't change over the lifetime of any
> distribution's platform? Because it's definitely not apparent to me
> that rebuilding the new server version against an upgraded platform
> (now possibly with HAVE_RESIZABLE_SHMEM) should also mean rebuilding
> the extensions that have been built against a previous minor version
> (without HAVE_RESIZABLE_SHMEM).
>

Please review changes in 0004 to see if they address your concerns.
The structures and functions are same irrespective of
HAVE_RESIZABLE_SHMEM. If HAVE_RESIZABLE_SHMEM is not defined, but
maximum_size > 0, then the request to add this structure will fail.
Thus on a server running with binary built with HAVE_RESIZABLE_SHMEM
undefined, maximum_size = 0 always. Many of the code blocks don't need
#ifdef HAVE_RESIZABLE_SHMEM blocks then. I think code this way is much
more readable. However binary will contain more instructions than
necessary - maybe the compiler can optimize those out.

> > > >  For now, it
> > > > seems only for the sanity checks, but it could be seen as a useful
> > > > safety feature. A difference in maximum_size and minimum_size would
> > > > indicate that the structure is resizable.
> > >
> > > I think that's the right approach.
> >
> >
> > I also think that introducing minimum_size is useful. Let's hear from
> > Heikki before implementing it, in case he has a different opinion. I
> > am not sure about min_allocated_space though - what use do you see for
> > it. reserved_space is useful in pg_shmem_allocations() C function
> > itself and gives impact to the fully grown structure. What would
> > min_allocated_space give us? If at all it would be min_allocated_size
> > not space since reserved space will never change. But even that I am
> > not sure about.
>
> I'd say it's mostly interesting for people looking at or debugging
> shmem allocations. Which isn't a huge group of developers or DBAs, but
> if we're exposing data like this, and are going to allow resizing,
> then someone could see some benefits from this.
>
> E.g., it may be useful to have the information to see how low the
> currently running server can scale down its memory usage, so that the
> admin can see whether a reboot is required if they want to allow it to
> scale it down further (assuming there's a lower limit for allocations
> - some shmem structs may have a lower scaling limit defined at
> startup, while others may be able to scale linearly from 0 to 100)

I guess, most of the time the lower limit will be a hard lower limit
which could not be changed, so I guess the usecases are pretty narrow.

0005 adds minimum_size member alongside maximum_size as per the
discussion starting [1]. I like the end result since maximum_size is
managed consistently across fixed-size and resizable structures and
also the condition to check whether a structure is resizable or fixed
is more natural/logical.

0006 adds support to add mprotect appropriately on the used and unused
part of a resizable structure, again per discussion starting [1]. This
is still a bit rough, but review comments are welcome.

I have kept these two patches separate from the main patch so that I
can remove them if others feel they are not worth including in the
feature.

[1] https://www.postgresql.org/message-id/CAExHW5stth2mdXh3ukn9rWJ+Pruoat+5tY3CYyM6KGqH2G30fQ@mail.gmail.com



--
Best Wishes,
Ashutosh Bapat

  73. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-07T10:06:25Z

    On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> I have kept these two patches separate from the main patch so that I
> can remove them if others feel they are not worth including in the
> feature.

Here are patches rebased on the latest HEAD. No conflicts just rebase.

Here are differences from the previous patchset.

o. There are two patches in this patchset now. a. 0001 which supports
resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
from the previous patchset. b. 0002 which is 0006 from the previous
patchset and adds support for protecting resizable shared memory
structures. 0003, which added diagnostics to investigate CFBot
failure, from the previous patchset is not required anymore since all
tests pass with CFBot.

o. I have merged 0002 into 0001 from the previous patchset since with
that patch all platforms are green on CFBot. The resizable shared
memory test now uses /proc/self/smaps instead of /proc/self/status to
find the amount of memory allocated in the main shared memory segment
of PostgreSQL.

o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
would be useful to protect against accidental shrinkage of the
resizable structures. It will help additional support for minimum
sizes of GUCs like shared_buffers. It also makes it easy and intuitive
to distinguish between fixed-size and resizable structures, and will
be useful to find the minimum size of the shared memory segment.

o. Merged 0005, which allows ABI compatibility between the binaries
which support resizable shared memory and those which don't, into
0001. Apart from ABI compatibility, the code has lesser #ifdef blocks
and thus easier to read and maintain.

I didn't find it useful to keep 0004 and 0005 separate since they were
interdependent and made review complicated and have higher chances of
being acceptable.

o. 0006 is still separate since I am not sure whether the
functionality is absolutely needed at this time. In an offlist
discussion, Andres mentioned that it is not strictly needed. The
subsystem that uses the resizable shared memory can implement their
own protection if required and integrate it in the subsystems specific
synchronization. But Matthias thinks different. The API to add
protection is platform dependent, so it's better to abstract it via
shmem.c. If we decide to accept this patch, we should merge it into
0001 before committing.

Also did some more cleanups and changed the name of the GUC
have_resizable_shmem to have_resizable_shared_memory since shmem is an
internal phrase.

I am looking at merging the resizable_shmem module into test_shmem module next.

-- 
Best Wishes,
Ashutosh Bapat

  74. Re: Better shared data structure management and resizable shared data structures

    Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> — 2026-04-07T12:24:28Z

    Heikki Linnakangas <hlinnaka@iki.fi> writes:

> Those are now committed, and here's a new version rebased over those
> changes.

I noticed this bit during my habitual morning skim of new commits:

> diff --git a/src/backend/utils/misc/injection_point.c b/src/backend/utils/misc/injection_point.c
> index c06b0e9b800..9981d6e212f 100644
> --- a/src/backend/utils/misc/injection_point.c
> +++ b/src/backend/utils/misc/injection_point.c
> @@ -17,6 +17,7 @@
>   */
>  #include "postgres.h"
>  
> +#include "storage/subsystems.h"
>  #include "utils/injection_point.h"
>  
>  #ifdef USE_INJECTION_POINTS
> @@ -109,6 +110,11 @@ typedef struct InjectionPointCacheEntry
>  
>  static HTAB *InjectionPointCache = NULL;
>  
> +#ifdef USE_INJECTION_POINTS
> +static void InjectionPointShmemRequest(void *arg);
> +static void InjectionPointShmemInit(void *arg);
> +#endif
> +

This is already inside an `#ifdef USE_INJECTION_POINTS` guard (in fact
visible at the end of the previous diff hunk), no need for another one.

- ilmari

  75. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-07T13:26:16Z

    On 07/04/2026 15:24, Dagfinn Ilmari Mannsåker wrote:
> Heikki Linnakangas <hlinnaka@iki.fi> writes:
> 
>> Those are now committed, and here's a new version rebased over those
>> changes.
> 
> I noticed this bit during my habitual morning skim of new commits:
> 
>> diff --git a/src/backend/utils/misc/injection_point.c b/src/backend/utils/misc/injection_point.c
>> index c06b0e9b800..9981d6e212f 100644
>> --- a/src/backend/utils/misc/injection_point.c
>> +++ b/src/backend/utils/misc/injection_point.c
>> @@ -17,6 +17,7 @@
>>    */
>>   #include "postgres.h"
>>   
>> +#include "storage/subsystems.h"
>>   #include "utils/injection_point.h"
>>   
>>   #ifdef USE_INJECTION_POINTS
>> @@ -109,6 +110,11 @@ typedef struct InjectionPointCacheEntry
>>   
>>   static HTAB *InjectionPointCache = NULL;
>>   
>> +#ifdef USE_INJECTION_POINTS
>> +static void InjectionPointShmemRequest(void *arg);
>> +static void InjectionPointShmemInit(void *arg);
>> +#endif
>> +
> 
> This is already inside an `#ifdef USE_INJECTION_POINTS` guard (in fact
> visible at the end of the previous diff hunk), no need for another one.

Fixed, thanks. I also noticed that the #include "storage/subsystems.h" 
can be moved inside the #ifdef block; fixed that too.

- Heikki

  76. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-07T14:19:28Z

    Hi Heikki,
CallShmemCallbacksAfterStartup() holds ShmemIndexLock while invoking
init_fn/attach_fn callbacks. That looks wrong. Before this commit,
init or attach code was not run with the lock held. Any reason the
lock is held while calling init and attach callbacks. Since these
function can come from extensions, we don't have control on what goes
in those functions, and thus looks problematic. Further, it will
serialize all the attach_fn executions across backends, since each
will be run under the lock. In my case, the init_fn was performing
ShmemIndex lookup which deadlocked. It's questionable whether init
function should lookup ShmemIndex but, it's not something that needs
to be prohibited either.

Here's patch fixing it.

--
Best Wishes,
Ashutosh Bapat

On Tue, Apr 7, 2026 at 6:56 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 07/04/2026 15:24, Dagfinn Ilmari Mannsåker wrote:
> > Heikki Linnakangas <hlinnaka@iki.fi> writes:
> >
> >> Those are now committed, and here's a new version rebased over those
> >> changes.
> >
> > I noticed this bit during my habitual morning skim of new commits:
> >
> >> diff --git a/src/backend/utils/misc/injection_point.c b/src/backend/utils/misc/injection_point.c
> >> index c06b0e9b800..9981d6e212f 100644
> >> --- a/src/backend/utils/misc/injection_point.c
> >> +++ b/src/backend/utils/misc/injection_point.c
> >> @@ -17,6 +17,7 @@
> >>    */
> >>   #include "postgres.h"
> >>
> >> +#include "storage/subsystems.h"
> >>   #include "utils/injection_point.h"
> >>
> >>   #ifdef USE_INJECTION_POINTS
> >> @@ -109,6 +110,11 @@ typedef struct InjectionPointCacheEntry
> >>
> >>   static HTAB *InjectionPointCache = NULL;
> >>
> >> +#ifdef USE_INJECTION_POINTS
> >> +static void InjectionPointShmemRequest(void *arg);
> >> +static void InjectionPointShmemInit(void *arg);
> >> +#endif
> >> +
> >
> > This is already inside an `#ifdef USE_INJECTION_POINTS` guard (in fact
> > visible at the end of the previous diff hunk), no need for another one.
>
> Fixed, thanks. I also noticed that the #include "storage/subsystems.h"
> can be moved inside the #ifdef block; fixed that too.
>
> - Heikki
>

  77. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-07T14:46:59Z

    On Tue, Apr 7, 2026 at 3:36 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > I have kept these two patches separate from the main patch so that I
> > can remove them if others feel they are not worth including in the
> > feature.
>
> Here are patches rebased on the latest HEAD. No conflicts just rebase.
>
> Here are differences from the previous patchset.
>
> o. There are two patches in this patchset now. a. 0001 which supports
> resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
> from the previous patchset. b. 0002 which is 0006 from the previous
> patchset and adds support for protecting resizable shared memory
> structures. 0003, which added diagnostics to investigate CFBot
> failure, from the previous patchset is not required anymore since all
> tests pass with CFBot.
>
> o. I have merged 0002 into 0001 from the previous patchset since with
> that patch all platforms are green on CFBot. The resizable shared
> memory test now uses /proc/self/smaps instead of /proc/self/status to
> find the amount of memory allocated in the main shared memory segment
> of PostgreSQL.
>
> o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
> would be useful to protect against accidental shrinkage of the
> resizable structures. It will help additional support for minimum
> sizes of GUCs like shared_buffers. It also makes it easy and intuitive
> to distinguish between fixed-size and resizable structures, and will
> be useful to find the minimum size of the shared memory segment.
>
> o. Merged 0005, which allows ABI compatibility between the binaries
> which support resizable shared memory and those which don't, into
> 0001. Apart from ABI compatibility, the code has lesser #ifdef blocks
> and thus easier to read and maintain.
>
> I didn't find it useful to keep 0004 and 0005 separate since they were
> interdependent and made review complicated and have higher chances of
> being acceptable.
>
> o. 0006 is still separate since I am not sure whether the
> functionality is absolutely needed at this time. In an offlist
> discussion, Andres mentioned that it is not strictly needed. The
> subsystem that uses the resizable shared memory can implement their
> own protection if required and integrate it in the subsystems specific
> synchronization. But Matthias thinks different. The API to add
> protection is platform dependent, so it's better to abstract it via
> shmem.c. If we decide to accept this patch, we should merge it into
> 0001 before committing.
>
> Also did some more cleanups and changed the name of the GUC
> have_resizable_shmem to have_resizable_shared_memory since shmem is an
> internal phrase.
>
> I am looking at merging the resizable_shmem module into test_shmem module next.

Here are patches with the test modules merged.

The merged module looks a bit rough to me and so does 0006. For
example, I am not sure whether calling ShmemStructProtect() from
init_fn is a good idea. See [1] for example. But init_fn is the last
chance for the subsystem to touch and setup the resizable structure
before it's opened to the wild. So, in the current infrastructure, I
don't see any better place to call ShmemStructProtect() either. If you
run tests after applying patch 0006, you will need to apply patch
attached to [1] as well; otherwise the test will hang.

[1] https://www.postgresql.org/message-id/CAExHW5uMQGvQH6GKaBZVtH4S9O13TwN+_0Vy1gUpAW=_T_AmRA@mail.gmail.com

-- 
Best Wishes,
Ashutosh Bapat

  78. Re: Better shared data structure management and resizable shared data structures

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2026-04-07T19:38:37Z

    On Tue, 7 Apr 2026 at 16:47, Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Apr 7, 2026 at 3:36 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > I have kept these two patches separate from the main patch so that I
> > > can remove them if others feel they are not worth including in the
> > > feature.
> >
> > Here are patches rebased on the latest HEAD. No conflicts just rebase.
> >
> > Here are differences from the previous patchset.
> >
> > o. There are two patches in this patchset now. a. 0001 which supports
> > resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
> > from the previous patchset. b. 0002 which is 0006 from the previous
> > patchset and adds support for protecting resizable shared memory
> > structures. 0003, which added diagnostics to investigate CFBot
> > failure, from the previous patchset is not required anymore since all
> > tests pass with CFBot.
> >
> > o. I have merged 0002 into 0001 from the previous patchset since with
> > that patch all platforms are green on CFBot. The resizable shared
> > memory test now uses /proc/self/smaps instead of /proc/self/status to
> > find the amount of memory allocated in the main shared memory segment
> > of PostgreSQL.
> >
> > o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
> > would be useful to protect against accidental shrinkage of the
> > resizable structures. It will help additional support for minimum
> > sizes of GUCs like shared_buffers. It also makes it easy and intuitive
> > to distinguish between fixed-size and resizable structures, and will
> > be useful to find the minimum size of the shared memory segment.

I was thinking more along the lines of attached (incremental) patch
0003 for min/max sizing. I'd say it has a slightly more natural API,
but YMMV.

-----

I also noticed that it's probably not correct to "just" check and
complain about the size of a resizable shmem segment when you attach:
Without coordination about which startup size the shmem segment should
have, how could you get the current size state correct? And
cross-process coordination of size information before shmem is
attached is not really possible, not when you may have to deal with a
very slow to start backend.

----

Attached also 0004, which makes some small adjustments to shmem.c's
resize checks.

Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

  79. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-07T19:48:17Z

    On 07/04/2026 17:46, Ashutosh Bapat wrote:
> Here are patches with the test modules merged.
> 
> The merged module looks a bit rough to me and so does 0006. For
> example, I am not sure whether calling ShmemStructProtect() from
> init_fn is a good idea. See [1] for example. But init_fn is the last
> chance for the subsystem to touch and setup the resizable structure
> before it's opened to the wild. So, in the current infrastructure, I
> don't see any better place to call ShmemStructProtect() either. If you
> run tests after applying patch 0006, you will need to apply patch
> attached to [1] as well; otherwise the test will hang.

I haven't really looked at these resizeable patches before, except for 
how they would fit with the new shmem allocation API, so I have some 
very basic, high-level design questions:

> +/*
> + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> + *
> + * The new size must be within [minimum_size, maximum_size].  If the structure
> + * is being shrunk, the memory pages that are no longer needed are freed. If
> + * the structure is being expanded, the memory pages that are needed for the
> + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> + * pages are allocated for a resizable structure.
> + */
> +void
> +ShmemResizeStruct(const char *name, Size new_size)

This interface only allows shrinking and growing the allocated region at 
the end, but the underlying mechanism is madvise(MADV_REMOVE) and 
madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e. 
freeing memory in the middle of a region. Do we gain anything by 
restricting ourselves to changing the size at the end? It seems to me 
that it could be handy to punch holes for some use cases.

What's the portability story? I understand that this is Linux-only at 
the moment, but what platforms can we support in the future, and what's 
the effort? I think BSD's have similar capabilities with plain mmap() 
and MADV_FREE if I read the man pages right. What about macOS and 
Windows? This doesn't necessarily need to be fully portable, if some 
OS's don't have the capabilities we need, but would be nice to know 
what's possible.

- Heikki

  80. Re: Better shared data structure management and resizable shared data structures

    Andres Freund <andres@anarazel.de> — 2026-04-07T20:09:25Z

    Hi,

On 2026-04-07 22:48:17 +0300, Heikki Linnakangas wrote:
> > +/*
> > + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> > + *
> > + * The new size must be within [minimum_size, maximum_size].  If the structure
> > + * is being shrunk, the memory pages that are no longer needed are freed. If
> > + * the structure is being expanded, the memory pages that are needed for the
> > + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> > + * pages are allocated for a resizable structure.
> > + */
> > +void
> > +ShmemResizeStruct(const char *name, Size new_size)
> 
> This interface only allows shrinking and growing the allocated region at the
> end, but the underlying mechanism is madvise(MADV_REMOVE) and
> madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e.
> freeing memory in the middle of a region. Do we gain anything by restricting
> ourselves to changing the size at the end? It seems to me that it could be
> handy to punch holes for some use cases.

Agreed. The hard part may be the "communication" with the user about how
granular the punches can be.  Because that will depend on things like
huge_pages, huge_page_size and may depend on what alignment you happened to
get.


> What's the portability story? I understand that this is Linux-only at the
> moment, but what platforms can we support in the future, and what's the
> effort? I think BSD's have similar capabilities with plain mmap() and
> MADV_FREE if I read the man pages right.

At least linux' MADV_FREE is only for private mappings. It's not clear in at
least freebsd's man page, but the described use case makes me suspect it may
be similar there.


> What about macOS and Windows? This doesn't necessarily need to be fully
> portable, if some OS's don't have the capabilities we need, but would be
> nice to know what's possible.

Looks like windows has OfferVirtualMemory
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-offervirtualmemory
but it's not clear to me if it actually does what we need when multiple
processes are attached.

I suspect it's going to be a lot easier once we're threaded...  The reason I
am ok with doing resizing this way before threading is because it's
architecturally pretty similar to what you'd want to do once threaded, so it's
not a huge dead end.  But I'm doubtful we'll find facilities that allow this
across processes in all operating systems...

Greetings,

Andres Freund

  81. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-08T03:49:26Z

    On Wed, Apr 8, 2026 at 1:08 AM Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
>
> On Tue, 7 Apr 2026 at 16:47, Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Tue, Apr 7, 2026 at 3:36 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
> > > <ashutosh.bapat.oss@gmail.com> wrote:
> > > >
> > > > I have kept these two patches separate from the main patch so that I
> > > > can remove them if others feel they are not worth including in the
> > > > feature.
> > >
> > > Here are patches rebased on the latest HEAD. No conflicts just rebase.
> > >
> > > Here are differences from the previous patchset.
> > >
> > > o. There are two patches in this patchset now. a. 0001 which supports
> > > resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
> > > from the previous patchset. b. 0002 which is 0006 from the previous
> > > patchset and adds support for protecting resizable shared memory
> > > structures. 0003, which added diagnostics to investigate CFBot
> > > failure, from the previous patchset is not required anymore since all
> > > tests pass with CFBot.
> > >
> > > o. I have merged 0002 into 0001 from the previous patchset since with
> > > that patch all platforms are green on CFBot. The resizable shared
> > > memory test now uses /proc/self/smaps instead of /proc/self/status to
> > > find the amount of memory allocated in the main shared memory segment
> > > of PostgreSQL.
> > >
> > > o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
> > > would be useful to protect against accidental shrinkage of the
> > > resizable structures. It will help additional support for minimum
> > > sizes of GUCs like shared_buffers. It also makes it easy and intuitive
> > > to distinguish between fixed-size and resizable structures, and will
> > > be useful to find the minimum size of the shared memory segment.
>
> I was thinking more along the lines of attached (incremental) patch
> 0003 for min/max sizing. I'd say it has a slightly more natural API,
> but YMMV.
>

Thanks for the proposal. There are some advantages and disadvantages
of that approach. Let me explain.

minimum_size = 0 seems more straightforward to me compared to the
introduction of SHMEM_RESIZE_TO_ZERO - a value other than 0 to mean 0.
That's confusing.

The thought to modify the options in place did cross my mind and I
started going that route. But soon realized that a. option is a caller
structure which is not designed to scribble upon, b. it is saved as a
request and used later. By scribbling upon it, we lose the intent of
the original request, thus the saved request may be susceptible to a
different interpretation in future. I would like to avoid scribbling
as much as possible. The code after scribbling doesn't look materially
improved than earlier.

I like the error handling refactoring, but need to pay close attention
to the details. I tried something similar that didn't work in all the
cases. I will try your changes in the next version.

0004 actually changes the error message we throw when the request is
opposite of the existing structure. Is that intentional? But I guess
some of it can be absorbed to simplify the code here. The macro
definition is confusing.

+#define CHECK_SIZE(size) \
+do { \
+ /* Check that the sizes in the index match the request. */ \
+ if (request->options->size != SHMEM_ATTACH_UNKNOWN_SIZE && \
+ index_entry->size != request->options->size) \
+ { \
+ ereport(ERROR, \
+ (errmsg("shared memory struct \"%s\" was created with" \
+ " different %s: existing %zu, requested %zu", \
+ name, CppAsString(size), index_entry->size, \
+ request->options->size))); \
+ } \
+} while (false)

Ideally size here should be in paranthesis. Its easy to confuse
request->options->size to mean request->options->size when it actually
means request->options->{maximum/minimum}_size. Is that right?
Possibly a static inline function where we pass corresponding members
of request->options and index_entry?

> -----
>
> I also noticed that it's probably not correct to "just" check and
> complain about the size of a resizable shmem segment when you attach:
> Without coordination about which startup size the shmem segment should
> have, how could you get the current size state correct? And
> cross-process coordination of size information before shmem is
> attached is not really possible, not when you may have to deal with a
> very slow to start backend.

SHMEM_ATTACH_UNKNOWN_SIZE can be used there. test_shmem module already
uses it that way.

-- 
Best Wishes,
Ashutosh Bapat

  82. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-08T05:20:53Z

    On Wed, Apr 8, 2026 at 1:39 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-04-07 22:48:17 +0300, Heikki Linnakangas wrote:
> > > +/*
> > > + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> > > + *
> > > + * The new size must be within [minimum_size, maximum_size].  If the structure
> > > + * is being shrunk, the memory pages that are no longer needed are freed. If
> > > + * the structure is being expanded, the memory pages that are needed for the
> > > + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> > > + * pages are allocated for a resizable structure.
> > > + */
> > > +void
> > > +ShmemResizeStruct(const char *name, Size new_size)
> >
> > This interface only allows shrinking and growing the allocated region at the
> > end, but the underlying mechanism is madvise(MADV_REMOVE) and
> > madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e.
> > freeing memory in the middle of a region. Do we gain anything by restricting
> > ourselves to changing the size at the end? It seems to me that it could be
> > handy to punch holes for some use cases.
>
> Agreed. The hard part may be the "communication" with the user about how
> granular the punches can be.  Because that will depend on things like
> huge_pages, huge_page_size and may depend on what alignment you happened to
> get.
>

We can extend it that way if there is a valid usecase. For now I kept
it simple for two reasons:

1. Buffer manager structures shrink and expand only at the end right
now. Longer note on buffer lookup table later. This effort started
with buffer resizing and didn't want to expand scope more than what's
needed.
2. Not all the approaches we tried to implement resizable shared
memory have the facility to free memory in the middle. Usually they
have a facility to shrink or expand at the end. If we offer ability to
free memory in the middle based on facilities on one platform, we will
face big hurdles when supporting other platforms. I think it's better
to avoid it when it's not needed.

 Buffer lookup table is fixed. It may benefit from punching holes in
the middle if we can somehow get pages worth of free entries together
somewhere in the middle. First it's not easy to perform such
compaction. But even if implement compaction, we can collect those
entries at the end instead of in the middle; the current API will
still be useful.

Is there any other usecase you are envisioning? I also think that it
will be better to introduce a new
ShmemFreeStructPart()/ShmemAllocStructPart() instead of the current
ShmemResizeStruct().

>
> > What's the portability story? I understand that this is Linux-only at the
> > moment, but what platforms can we support in the future, and what's the
> > effort? I think BSD's have similar capabilities with plain mmap() and
> > MADV_FREE if I read the man pages right.
>
> At least linux' MADV_FREE is only for private mappings. It's not clear in at
> least freebsd's man page, but the described use case makes me suspect it may
> be similar there.
>

looks so. FreeBSD also has fallocate with PUNCH_HOLES. We could use it
with fd created using memfd_create() on .and it will need
memfd_create(). I haven't checked whether that works.

>
> > What about macOS and Windows? This doesn't necessarily need to be fully
> > portable, if some OS's don't have the capabilities we need, but would be
> > nice to know what's possible.
>
> Looks like windows has OfferVirtualMemory
> https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-offervirtualmemory
> but it's not clear to me if it actually does what we need when multiple
> processes are attached.
>

Those APIs look similar to madvise+ MADV_REMOVE/MADV_WRITE_POPULATE,
with specific and cleaner interface. At least worth a try.

> I suspect it's going to be a lot easier once we're threaded...  The reason I
> am ok with doing resizing this way before threading is because it's
> architecturally pretty similar to what you'd want to do once threaded, so it's
> not a huge dead end.  But I'm doubtful we'll find facilities that allow this
> across processes in all operating systems...

check

-- 
Best Wishes,
Ashutosh Bapat

  83. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-04-21T07:40:14Z

    On 07/04/2026 17:19, Ashutosh Bapat wrote:
> Hi Heikki,
> CallShmemCallbacksAfterStartup() holds ShmemIndexLock while invoking
> init_fn/attach_fn callbacks. That looks wrong. Before this commit,
> init or attach code was not run with the lock held. Any reason the
> lock is held while calling init and attach callbacks. Since these
> function can come from extensions, we don't have control on what goes
> in those functions, and thus looks problematic. Further, it will
> serialize all the attach_fn executions across backends, since each
> will be run under the lock.

This was intentional, I added a note in the docs about it:

       When <function>RegisterShmemCallbacks()</function> is called after
       startup, it will immediately call the appropriate callbacks, 
depending
       on whether the requested memory areas were already initialized by
       another backend. The callbacks will be called while holding an 
internal
       lock, which prevents concurrent two backends from initializing the
       memory area concurrently.

That "internal lock" is ShmemIndexLock. I piggybacked on that since the 
code needs to acquire it anyway for the hash table lookups.

With the old ShmemInitStruct() interface, extensions needed to do the 
locking themselves, usually by holding AddinShmemInitLock.

(Now that I read that again, the grammar on the last sentence sounds 
awkward...)

> In my case, the init_fn was performing ShmemIndex lookup which
> deadlocked. It's questionable whether init function should lookup
> ShmemIndex but, it's not something that needs to be prohibited
> either.
Yeah I'm curious what the use case is. We could easily introduce another 
lock or reuse AddinShmemInitLock for this.

- Heikki

  84. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-04-21T16:05:56Z

    On Tue, Apr 21, 2026 at 1:10 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 07/04/2026 17:19, Ashutosh Bapat wrote:
> > Hi Heikki,
> > CallShmemCallbacksAfterStartup() holds ShmemIndexLock while invoking
> > init_fn/attach_fn callbacks. That looks wrong. Before this commit,
> > init or attach code was not run with the lock held. Any reason the
> > lock is held while calling init and attach callbacks. Since these
> > function can come from extensions, we don't have control on what goes
> > in those functions, and thus looks problematic. Further, it will
> > serialize all the attach_fn executions across backends, since each
> > will be run under the lock.
>
> This was intentional, I added a note in the docs about it:
>
>        When <function>RegisterShmemCallbacks()</function> is called after
>        startup, it will immediately call the appropriate callbacks,
> depending
>        on whether the requested memory areas were already initialized by
>        another backend. The callbacks will be called while holding an
> internal
>        lock, which prevents concurrent two backends from initializing the
>        memory area concurrently.
>
> That "internal lock" is ShmemIndexLock. I piggybacked on that since the
> code needs to acquire it anyway for the hash table lookups.
>

I had read this part, but didn't realize it's ShmemIndexLock. The
document and the code are placed far apart and the comments in the
code do not help connecting these two. The comment before
LWLockAcquire() call doesn't say anything about init functions.
/* Hold ShmemIndexLock while we allocate all the shmem entries */

> With the old ShmemInitStruct() interface, extensions needed to do the
> locking themselves, usually by holding AddinShmemInitLock.
>
> (Now that I read that again, the grammar on the last sentence sounds
> awkward...)
>

Given that the init_fn is called in only one backend which requests
the structures first, do we need a lock?

> > In my case, the init_fn was performing ShmemIndex lookup which
> > deadlocked. It's questionable whether init function should lookup
> > ShmemIndex but, it's not something that needs to be prohibited
> > either.
> Yeah I'm curious what the use case is. We could easily introduce another
> lock or reuse AddinShmemInitLock for this.
>

In case of resizable shared memory structures, I was adding mprotect
to make sure that the part of the shared address space which is
reserved but not used is protected from inadvertent access. The
mprotect is wrapped in a shmem API which fetches the ShmemIndex entry
of the shared structure, figures out the part of the address space to
protect using maximum_size and current size and calls mprotect
appropriately. To fetch the ShmemIndex entry it acquires a ShmemIndex
lock. The shmem API was supposed to be called from init_fn() and
attach_fn() to protect the address spaces as soon as the structure is
attached to. See patches attached to [1] for code.

[1] https://www.postgresql.org/message-id/CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com

-- 
Best Wishes,
Ashutosh Bapat

  85. Re: Better shared data structure management and resizable shared data structures

    Heikki Linnakangas <hlinnaka@iki.fi> — 2026-06-12T15:37:21Z

    On 21/04/2026 19:05, Ashutosh Bapat wrote:
> On Tue, Apr 21, 2026 at 1:10 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>>
>> On 07/04/2026 17:19, Ashutosh Bapat wrote:
>>> Hi Heikki,
>>> CallShmemCallbacksAfterStartup() holds ShmemIndexLock while invoking
>>> init_fn/attach_fn callbacks. That looks wrong. Before this commit,
>>> init or attach code was not run with the lock held. Any reason the
>>> lock is held while calling init and attach callbacks. Since these
>>> function can come from extensions, we don't have control on what goes
>>> in those functions, and thus looks problematic. Further, it will
>>> serialize all the attach_fn executions across backends, since each
>>> will be run under the lock.
>>
>> This was intentional, I added a note in the docs about it:
>>
>>         When <function>RegisterShmemCallbacks()</function> is called after
>>         startup, it will immediately call the appropriate callbacks,
>> depending
>>         on whether the requested memory areas were already initialized by
>>         another backend. The callbacks will be called while holding an
>> internal
>>         lock, which prevents concurrent two backends from initializing the
>>         memory area concurrently.
>>
>> That "internal lock" is ShmemIndexLock. I piggybacked on that since the
>> code needs to acquire it anyway for the hash table lookups.
> 
> I had read this part, but didn't realize it's ShmemIndexLock. The
> document and the code are placed far apart and the comments in the
> code do not help connecting these two. The comment before
> LWLockAcquire() call doesn't say anything about init functions.
> /* Hold ShmemIndexLock while we allocate all the shmem entries */
> 
>> With the old ShmemInitStruct() interface, extensions needed to do the
>> locking themselves, usually by holding AddinShmemInitLock.
>>
>> (Now that I read that again, the grammar on the last sentence sounds
>> awkward...)
> 
> Given that the init_fn is called in only one backend which requests
> the structures first, do we need a lock?

If two backends request the same structure concurrently, which one is 
"first"? That's what the lock determines.

It's not safe to release the lock before the init callback has finished. 
Otherwise, another backend might attach to the struct before it's fully 
initialized and read uninitialized values.

>>> In my case, the init_fn was performing ShmemIndex lookup which
>>> deadlocked. It's questionable whether init function should lookup
>>> ShmemIndex but, it's not something that needs to be prohibited
>>> either.
>> Yeah I'm curious what the use case is. We could easily introduce another
>> lock or reuse AddinShmemInitLock for this.
> 
> In case of resizable shared memory structures, I was adding mprotect
> to make sure that the part of the shared address space which is
> reserved but not used is protected from inadvertent access. The
> mprotect is wrapped in a shmem API which fetches the ShmemIndex entry
> of the shared structure, figures out the part of the address space to
> protect using maximum_size and current size and calls mprotect
> appropriately. To fetch the ShmemIndex entry it acquires a ShmemIndex
> lock. The shmem API was supposed to be called from init_fn() and
> attach_fn() to protect the address spaces as soon as the structure is
> attached to. See patches attached to [1] for code.
> 
> [1] https://www.postgresql.org/message-id/CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com

Ok. So if I understand correctly, holding ShmemIndexLock is not a actual 
problem per se, you just didn't expect it. Right?

I propose the attached to improve the wording a little on the docs, 
comments, and error message.

- Heikki

  86. Re: Better shared data structure management and resizable shared data structures

    Haoyu Huang <haoyu.huang.68@gmail.com> — 2026-06-12T15:51:16Z

    Hi all,

Wanted to introduce myself on this thread and share some related work —
with no intention of forking or redirecting what Ashutosh is driving here.

It was great catching up with Ashutosh, David Wein, and Heikki at PGConf
Vancouver. We had a working session on resizable shared buffers. It was
productive and a lot of fun. The outcome of the session is to surface our
work at Databricks on the same topic here.

At Databricks, we have a patch merged in our internal Postgres that enables
resizing shared_buffers without restart. It was inspired by Ashutosh's
earlier patch on this topic. Heikki and David reviewed it on our side. I'd
like to contribute the ideas  (and, where useful, the code) back upstream.
I want to be explicit that the current series is the path forward. I'd much
rather plug into that than propose a competing patchset. Happy to help with
review, testing, or specific pieces wherever it's most useful.

Please read the patch as input to the existing effort, not a
counter-proposal. Thanks for all the work on this so far.

Here are the major changes we made on top of Ashutosh's earlier patch

   1. Keep one mmap anonymous segment + madvise(MADV_POPULATE_WRITE /
   MADV_REMOVE) to allocate/free physical pages.
   2. SB variable names changed to use lowNBuffers, highNBuffers, and
   maxNBuffers. See the README for more details. We think that this simplifies
   the code significantly.
   3. Only the shrink needs to use proc signal barrier to coordinate with
   all other backends. The other cases are covered by a new
   AccessNBuffersLock.  The coordinator acquires exclusive lock on
   AccessNBuffersLock when it publishes new buffers. Other backends acquire
   the shared lock on AccessNBuffersLock when they loop through the buffer
   array based on the NBuffers value.
   4. The API to resize the shared buffer is `SELECT
   pg_resize_shared_buffers('new_size')`.



Thanks,
Haoyu


On Fri, Jun 12, 2026 at 8:37 AM Heikki Linnakangas <hlinnaka@iki.fi> wrote:

> On 21/04/2026 19:05, Ashutosh Bapat wrote:
> > On Tue, Apr 21, 2026 at 1:10 PM Heikki Linnakangas <hlinnaka@iki.fi>
> wrote:
> >>
> >> On 07/04/2026 17:19, Ashutosh Bapat wrote:
> >>> Hi Heikki,
> >>> CallShmemCallbacksAfterStartup() holds ShmemIndexLock while invoking
> >>> init_fn/attach_fn callbacks. That looks wrong. Before this commit,
> >>> init or attach code was not run with the lock held. Any reason the
> >>> lock is held while calling init and attach callbacks. Since these
> >>> function can come from extensions, we don't have control on what goes
> >>> in those functions, and thus looks problematic. Further, it will
> >>> serialize all the attach_fn executions across backends, since each
> >>> will be run under the lock.
> >>
> >> This was intentional, I added a note in the docs about it:
> >>
> >>         When <function>RegisterShmemCallbacks()</function> is called
> after
> >>         startup, it will immediately call the appropriate callbacks,
> >> depending
> >>         on whether the requested memory areas were already initialized
> by
> >>         another backend. The callbacks will be called while holding an
> >> internal
> >>         lock, which prevents concurrent two backends from initializing
> the
> >>         memory area concurrently.
> >>
> >> That "internal lock" is ShmemIndexLock. I piggybacked on that since the
> >> code needs to acquire it anyway for the hash table lookups.
> >
> > I had read this part, but didn't realize it's ShmemIndexLock. The
> > document and the code are placed far apart and the comments in the
> > code do not help connecting these two. The comment before
> > LWLockAcquire() call doesn't say anything about init functions.
> > /* Hold ShmemIndexLock while we allocate all the shmem entries */
> >
> >> With the old ShmemInitStruct() interface, extensions needed to do the
> >> locking themselves, usually by holding AddinShmemInitLock.
> >>
> >> (Now that I read that again, the grammar on the last sentence sounds
> >> awkward...)
> >
> > Given that the init_fn is called in only one backend which requests
> > the structures first, do we need a lock?
>
> If two backends request the same structure concurrently, which one is
> "first"? That's what the lock determines.
>
> It's not safe to release the lock before the init callback has finished.
> Otherwise, another backend might attach to the struct before it's fully
> initialized and read uninitialized values.
>
> >>> In my case, the init_fn was performing ShmemIndex lookup which
> >>> deadlocked. It's questionable whether init function should lookup
> >>> ShmemIndex but, it's not something that needs to be prohibited
> >>> either.
> >> Yeah I'm curious what the use case is. We could easily introduce another
> >> lock or reuse AddinShmemInitLock for this.
> >
> > In case of resizable shared memory structures, I was adding mprotect
> > to make sure that the part of the shared address space which is
> > reserved but not used is protected from inadvertent access. The
> > mprotect is wrapped in a shmem API which fetches the ShmemIndex entry
> > of the shared structure, figures out the part of the address space to
> > protect using maximum_size and current size and calls mprotect
> > appropriately. To fetch the ShmemIndex entry it acquires a ShmemIndex
> > lock. The shmem API was supposed to be called from init_fn() and
> > attach_fn() to protect the address spaces as soon as the structure is
> > attached to. See patches attached to [1] for code.
> >
> > [1]
> https://www.postgresql.org/message-id/CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com
>
> Ok. So if I understand correctly, holding ShmemIndexLock is not a actual
> problem per se, you just didn't expect it. Right?
>
> I propose the attached to improve the wording a little on the docs,
> comments, and error message.
>
> - Heikki
>

  87. Re: Better shared data structure management and resizable shared data structures

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2026-06-15T04:28:26Z

    On Fri, Jun 12, 2026 at 9:07 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>
> On 21/04/2026 19:05, Ashutosh Bapat wrote:
> > On Tue, Apr 21, 2026 at 1:10 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> >>
> >> On 07/04/2026 17:19, Ashutosh Bapat wrote:
> >>> Hi Heikki,
> >>> CallShmemCallbacksAfterStartup() holds ShmemIndexLock while invoking
> >>> init_fn/attach_fn callbacks. That looks wrong. Before this commit,
> >>> init or attach code was not run with the lock held. Any reason the
> >>> lock is held while calling init and attach callbacks. Since these
> >>> function can come from extensions, we don't have control on what goes
> >>> in those functions, and thus looks problematic. Further, it will
> >>> serialize all the attach_fn executions across backends, since each
> >>> will be run under the lock.
> >>
> >> This was intentional, I added a note in the docs about it:
> >>
> >>         When <function>RegisterShmemCallbacks()</function> is called after
> >>         startup, it will immediately call the appropriate callbacks,
> >> depending
> >>         on whether the requested memory areas were already initialized by
> >>         another backend. The callbacks will be called while holding an
> >> internal
> >>         lock, which prevents concurrent two backends from initializing the
> >>         memory area concurrently.
> >>
> >> That "internal lock" is ShmemIndexLock. I piggybacked on that since the
> >> code needs to acquire it anyway for the hash table lookups.
> >
> > I had read this part, but didn't realize it's ShmemIndexLock. The
> > document and the code are placed far apart and the comments in the
> > code do not help connecting these two. The comment before
> > LWLockAcquire() call doesn't say anything about init functions.
> > /* Hold ShmemIndexLock while we allocate all the shmem entries */
> >
> >> With the old ShmemInitStruct() interface, extensions needed to do the
> >> locking themselves, usually by holding AddinShmemInitLock.
> >>
> >> (Now that I read that again, the grammar on the last sentence sounds
> >> awkward...)
> >
> > Given that the init_fn is called in only one backend which requests
> > the structures first, do we need a lock?
>
> If two backends request the same structure concurrently, which one is
> "first"? That's what the lock determines.
>
> It's not safe to release the lock before the init callback has finished.
> Otherwise, another backend might attach to the struct before it's fully
> initialized and read uninitialized values.
>
> >>> In my case, the init_fn was performing ShmemIndex lookup which
> >>> deadlocked. It's questionable whether init function should lookup
> >>> ShmemIndex but, it's not something that needs to be prohibited
> >>> either.
> >> Yeah I'm curious what the use case is. We could easily introduce another
> >> lock or reuse AddinShmemInitLock for this.
> >
> > In case of resizable shared memory structures, I was adding mprotect
> > to make sure that the part of the shared address space which is
> > reserved but not used is protected from inadvertent access. The
> > mprotect is wrapped in a shmem API which fetches the ShmemIndex entry
> > of the shared structure, figures out the part of the address space to
> > protect using maximum_size and current size and calls mprotect
> > appropriately. To fetch the ShmemIndex entry it acquires a ShmemIndex
> > lock. The shmem API was supposed to be called from init_fn() and
> > attach_fn() to protect the address spaces as soon as the structure is
> > attached to. See patches attached to [1] for code.
> >
> > [1] https://www.postgresql.org/message-id/CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com
>
> Ok. So if I understand correctly, holding ShmemIndexLock is not a actual
> problem per se, you just didn't expect it. Right?
>
> I propose the attached to improve the wording a little on the docs,
> comments, and error message.

The patch helps to set the expectations right.

ShmemIndexLock is for protecting entries in ShmemIndex; I didn't
expect it to protect the Shared structures as well. I thought a shared
structure specific lock, which usually every shared structure is
expected to have, would protect its initialization and content. But I
see that I was wrong. Even those locks need to be initialized; so they
can't be used here. ShmemIndexLock works here with the proposed
comment changes.

--
Best Wishes,
Ashutosh Bapat