Re: Changing shared_buffers without restart

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

To: Dmitry Dolgov <9erthalion6@gmail.com>

Cc: pgsql-hackers@postgresql.org, Robert Haas <robertmhaas@gmail.com>

Date: 2025-02-28T11:52:29Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Remove PG_MMAP_FLAGS from mem.h
- c100340729b6 19 (unreleased) landed
Improve runtime and output of tests for replication slots checkpointing.
- 4464fddf7b50 18.0 cited
Revert support for improved tracking of nested queries
- f85f6ab051b7 18.0 cited
Use exported symbols list on macOS for loadable modules as well
- 3feff3916ee1 18.0 cited
Add support for basic NUMA awareness
- 65c298f61fc7 18.0 cited
Avoid unnecessary copying of a string in pg_restore.c
- 5e1915439085 18.0 cited
aio: Infrastructure for io_method=worker
- 55b454d0e140 18.0 cited
Improve InitShmemAccess() prototype
- 2a7b2d97171d 18.0 landed

Attachments

0004-Introduce-pending-flag-for-GUC-assign-hooks-20250228.patch (text/x-patch) patch 0004
0003-Introduce-multiple-shmem-segments-for-share-20250228.patch (text/x-patch) patch 0003
0005-Allow-to-resize-shared-memory-without-resta-20250228.patch (text/x-patch) patch 0005
0002-Allow-placing-shared-memory-mapping-with-an-20250228.patch (text/x-patch) patch 0002
0001-Allow-to-use-multiple-shared-memory-mapping-20250228.patch (text/x-patch) patch 0001
0006-Use-anonymous-files-to-back-shared-memory-s-20250228.patch (text/x-patch) patch 0006
0010-WIP-Reinitialize-buffer-sync-strategy-20250228.patch (text/x-patch) patch 0010
0007-Fix-compilation-failures-in-previous-patche-20250228.patch (text/x-patch) patch 0007
0009-WIP-Support-shrinking-shared-buffers-20250228.patch (text/x-patch) patch 0009
0008-Add-TODOs-and-questions-about-previous-comm-20250228.patch (text/x-patch) patch 0008
pgbench-concurrent-resize-buffers.sh (application/x-shellscript)

On Thu, Feb 27, 2025 at 1:58 PM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
>
> > On Tue, Feb 25, 2025 at 10:52:05AM GMT, Dmitry Dolgov wrote:
> > > On Fri, Oct 18, 2024 at 09:21:19PM GMT, Dmitry Dolgov wrote:
> > > TL;DR A PoC for changing shared_buffers without PostgreSQL restart, via
> > > changing shared memory mapping layout. Any feedback is appreciated.
> >
> > Hi,
> >
> > Here is a new version of the patch, which contains a proposal about how to
> > coordinate shared memory resizing between backends. The rest is more or less
> > the same, a feedback about coordination is appreciated. It's a lot to read, but
> > the main difference is about:
>
> Just one note, there are still couple of compilation warnings in the
> code, which I haven't addressed yet. Those will go away in the next
> version.

PFA the patchset which implements shrinking shared buffers.
0001-0006 are same as the previous patchset
0007 fixes compilation warnings from previous patches - I think those
should be absorbed into their respective patches
0008 adds TODOs that need some code changes or at least need some
consideration. Some of them might point to the causes of Assertion
failures seen with this patch set.
0009 adds WIP support for shrinking shared buffers - I think this
should be absorbed into 0005
0010 WIP fix for Assertion failures seen from BgBufferSync() - I am
still investigating those.

I am using the attached script to shake the patch well.  It runs
pgbench and concurrently resizes the shared_buffers. I am seeing
Assertion failures when running the script in both cases, expanding
and shrinking the buffers. I am investigating "failed
Assert("strategy_delta >= 0")," next.

-- 
Best Wishes,
Ashutosh Bapat