Horton

Re: Changing shared_buffers without restart

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>

View thread
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
To: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: Tomas Vondra <tomas@vondra.me>, Thomas Munro <thomas.munro@gmail.com>, pgsql-hackers@postgresql.org, Jack Ng <Jack.Ng@huawei.com>, Ni Ku <jakkuniku@gmail.com>
Date: 2025-07-14T04:55:51Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Remove PG_MMAP_FLAGS from mem.h

  2. Improve runtime and output of tests for replication slots checkpointing.

  3. Revert support for improved tracking of nested queries

  4. Use exported symbols list on macOS for loadable modules as well

  5. Add support for basic NUMA awareness

  6. Avoid unnecessary copying of a string in pg_restore.c

  7. aio: Infrastructure for io_method=worker

  8. Improve InitShmemAccess() prototype

On Mon, Jul 14, 2025 at 12:07 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
>
> > On Sun, Jul 06, 2025 at 03:21:08PM +0200, Dmitry Dolgov wrote:
> > > On Sun, Jul 06, 2025 at 03:01:34PM +0200, Dmitry Dolgov wrote:
> > > * This way any backend between the ProcSignalBarriers will be able
> > >   proceed with whatever it's doing, and there is need to make sure it
> > >   will not access buffers that will soon disappear. A suggestion so far
> > >   was to get all backends agree to not allocate any new buffers in the
> > >   to-be-truncated range, but accessing already existing buffers that
> > >   will soon go away is a problem as well. As far as I can tell there is
> > >   no rock solid method to make sure a backend doesn't have a reference
> > >   to such a buffer somewhere (this was discussed earlier in thre
> > >   thread), meaning that either a backend has to wait or buffers have to
> > >   be checked every time on access.
> >
> > And sure enough, after I wrote this I've realized there should be no
> > such references after the buffer eviction and prohibiting new buffer
> > allocation. I still need to check it though, because not only buffers,
> > but other shared memory structures (which number depends on NBuffers)
> > will be truncated. But if they will also be handled by the eviction,
> > then maybe everything is just fine.
>
> Pondering more about this topic, I've realized there was one more
> problematic case mentioned by Robert early in the thread, which is
> relatively easy to construct:
>
> * When increasing shared buffers from NBuffers_small to NBuffers_large
>   it's possible that one backend already has applied NBuffers_large,
>   then allocated a buffer B from (NBuffer_small, NBuffers_large] and put
>   it into the buffer lookup table.
>
> * In the meantime another backend still has NBuffers_small, but got
>   buffer B from the lookup table.
>
> Currently it's being addressed via every backend waiting for each other,
> but I guess it could be as well managed via handling the freelist, so
> that only "available" buffers will be inserted into the lookup table.

I didn't get how can that be managed by freelist? Buffers are also
allocated through clocksweep, which needs to be managed as well.

> Long story short, in the next version of the patch I'll try to
> experiment with a simplified design: a simple function to trigger
> resizing, launching a coordinator worker, with backends not waiting for
> each other and buffers first allocated and then marked as "available to
> use".

Should all the backends wait between buffer allocation and them being
marked as "available"? I assume that marking them as available means
"declaring the new NBuffers". What about when shrinking the buffers?
Do you plan to make all the backends wait while the coordinator is
evicting buffers?

-- 
Best Wishes,
Ashutosh Bapat