Horton

Re: Some read stream improvements

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-02-27T10:09:40Z
Lists: pgsql-hackers 
On Thu, Feb 27, 2025 at 11:19 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Feb 26, 2025 at 10:55 PM Andres Freund <andres@anarazel.de> wrote:
> > I was working on expanding tests for AIO and as part of that wrote a test for
> > temp tables -- our coverage is fairly awful, there were many times during AIO
> > development where I knew I had trivially reachable temp table specific bugs
> > but all tests passed.
> >
> > The test for that does trigger the problem described above and is fixed by the
> > patches in this thread (which I included in the other thread):
>
> Thanks.  Alright, I'm assuming that you don't have any objections to
> the way I restyled that API, so I'm going to go ahead and push some of
> these shortly, and then follow up with a few newer patches that
> simplify and improve the look-ahead and advice control.  More very
> soon.

Ugh, I realised in another round of self-review that that version
could exceed the soft limit by a small amount if the registered
callback pins more buffers underneath it, so not pushed yet.  I think
I see how to fix that (namely the alternative design that a comment
already contemplated), more soon...

Commits

  1. read_stream: Fix overflow hazard with large shared buffers

  2. Support buffer forwarding in StartReadBuffers().

  3. Support buffer forwarding in read_stream.c.

  4. Simplify distance heuristics in read_stream.c.

  5. Improve read_stream.c advice for dense streams.

  6. Respect changing pin limits in read_stream.c.

  7. Improve buffer manager API for backend pin limits.

  8. Remove arbitrary cap on read_stream.c buffer queue.