Horton
Menu

Re: Better shared data structure management and resizable shared data structures

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, Andres Freund <andres@anarazel.de>
Cc: chaturvedipalak1911@gmail.com
Date: 2026-02-13T12:03:12Z
Lists: pgsql-hackers 
On 13/02/2026 13:47, Ashutosh Bapat wrote:
> `man madvise` has this
>         MADV_REMOVE (since Linux 2.6.16)
>                Free  up a given range of pages and its associated
> backing store.  This is equivalent to punching a
>                hole in the corresponding byte range of the backing
> store (see fallocate(2)).  Subsequent  accesses
>                in the specified address range will see bytes containing zero.
> 
>                The  specified  address  range  must be mapped shared
> and writable.  This flag cannot be applied to
>                locked pages, Huge TLB pages, or VM_PFNMAP pages.
> 
>                In the initial implementation, only tmpfs(5) was
> supported MADV_REMOVE; but since  Linux  3.5,  any
>                filesystem  which  supports  the  fallocate(2)
> FALLOC_FL_PUNCH_HOLE mode also supports MADV_REMOVE.
>                Hugetlbfs fails with the error EINVAL and other
> filesystems fail with the error EOPNOTSUPP.
> 
> It says the flag can not be applied to Huge TLB pages. We won't be
> able to make resizable shared memory structures allocated with huge
> pages. That seems like a serious restriction.

Per https://man7.org/linux/man-pages/man2/madvise.2.html:

MADV_REMOVE (since Linux 2.6.16)
               ...

               Support for the Huge TLB filesystem was added in Linux
               v4.3.

> I may be misunderstanding something, but it seems like this is useful
> to free already allocated memory, not necessarily allocate more
> memory. I don't understand how a user would start with a larger
> reserved address space with only small portions of that space being
> backed by memory.

Hmm, I guess you'll need to use MAP_NORESERVE in the first mmap() call. 
to reserve address space for the maximum size, and then 
madvise(MADV_POPULATE_WRITE) using the initial size. Later, 
madvise(MADV_REMOVE) to shrink, and madvise(MADV_POPULATE_WRITE) to grow 
again.

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tidy up #ifdef USE_INJECTION_POINTS guards

  2. Convert all remaining subsystems to use the new shmem allocation API

  3. Convert buffer manager to use the new shmem allocation functions

  4. Add alignment option to ShmemRequestStruct()

  5. Convert AIO to use the new shmem allocation functions

  6. Convert SLRUs to use the new shmem allocation functions

  7. Refactor shmem initialization code in predicate.c

  8. Use the new shmem allocation functions in a few core subsystems

  9. Convert lwlock.c to use the new shmem allocation functions

  10. Introduce a registry of built-in shmem subsystems

  11. Convert pg_stat_statements to use the new shmem allocation functions

  12. Add a test module to test after-startup shmem allocations

  13. Introduce a new mechanism for registering shared memory areas

  14. Move some code from shmem.c and shmem.h

  15. Improve test_lwlock_tranches

  16. Test pg_stat_statements across crash restart

  17. Refactor PredicateLockShmemInit to not reuse var for different things

  18. Refactor ShmemIndex initialization

  19. Add a new shmem_request_hook hook.