Horton

Re: Changing shared_buffers without restart

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

View thread
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
To: Jack Ng <Jack.Ng@huawei.com>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>, Robert Haas <robertmhaas@gmail.com>, Ni Ku <jakkuniku@gmail.com>
Date: 2025-05-09T14:43:02Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Remove PG_MMAP_FLAGS from mem.h

  2. Improve runtime and output of tests for replication slots checkpointing.

  3. Revert support for improved tracking of nested queries

  4. Use exported symbols list on macOS for loadable modules as well

  5. Add support for basic NUMA awareness

  6. Avoid unnecessary copying of a string in pg_restore.c

  7. aio: Infrastructure for io_method=worker

  8. Improve InitShmemAccess() prototype

Attachments

On Wed, May 7, 2025 at 11:04 AM Jack Ng <Jack.Ng@huawei.com> wrote:

> > all the possible scenarios. But now I'm reworking it along the lines
> suggested
> > by Thomas, and will address those as well. Thanks!
>
> Thanks for the info, Dmitry.
> Just want to confirm my understanding of Thomas' suggestion and your
> discussions... I think the simpler and more portable solution goes
> something like the following?
>
> * For each BP resource segment (main, desc, buffers, etc):
>     1. create an anonymous file as backing
>     2. mmap a large reserved shared memory area with PROTO_READ/WRITE +
> MAP_NORESERVE using the anon fd
>     3. use ftruncate to back the in-use region (and maybe posix_fallocate
> too to avoid SIGBUS on alloc failure during first-touch), but no need to
> create a memory mapping for it
>     4. also no need to create a separate mapping for the reserved region
> (already covered by the mapping created in 2.)
>
> |-- Memory mapping (MAP_NORESERVE) for BUFFER --|
> |-- In-use region --|----- Reserved region -----|
>
> * During resize, simply calculate the new size and call ftruncate on each
> segment to adjust memory accordingly, no need to mmap/munmap or modify any
> memory mapping.
>
>
That's same as my understanding.


> I tried this approach with a test program (with huge pages), and both
> expand and shrink seem to work as expected --for shrink, the memory is
> freed right after the resize ftruncate.
>
> I thought I had shared a test program upthread, but I don't find it now.
Attached here. Can you please share your test program?

There are concerns around portability of this approach, though.


-- 
Best Wishes,
Ashutosh Bapat