Horton

Re: index prefetching

Thomas Munro <thomas.munro@gmail.com>

View thread
From: Thomas Munro <thomas.munro@gmail.com>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Tomas Vondra <tomas@vondra.me>, Nazir Bilal Yavuz <byavuz81@gmail.com>, Andres Freund <andres@anarazel.de>, Robert Haas <robertmhaas@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Georgios <gkokolatos@protonmail.com>, Konstantin Knizhnik <knizhnik@garret.ru>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2025-08-05T23:30:23Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. aio: io_uring: Trigger async processing for large IOs

  2. read stream: Split decision about look ahead for AIO and combining

  3. read_stream: Only increase read-ahead distance when waiting for IO

  4. read_stream: Prevent distance from decaying too quickly

  5. Reduce ExecSeqScan* code size using pg_assume()

  6. Fix rare bug in read_stream.c's split IO handling.

  7. Fix multiranges to behave more like dependent types.

  8. Add EXPLAIN (MEMORY) to report planner memory consumption

  9. Optimize nbtree backward scan boundary cases.

  10. Increment xactCompletionCount during subtransaction abort.

  11. Add nbtree Valgrind buffer lock checks.

  12. Add nbtree high key "continuescan" optimization.

  13. Reduce pinning and buffer content locking for btree scans.

  14. Teach btree to handle ScalarArrayOpExpr quals natively.

On Wed, Aug 6, 2025 at 9:35 AM Peter Geoghegan <pg@bowt.ie> wrote:
> On Tue, Aug 5, 2025 at 4:56 PM Tomas Vondra <tomas@vondra.me> wrote:
> > True, the complex patch could prefetch the leaf pages.

There must be a similar opportunity for parallel index scans.  It has
that "seize the scan" concept where parallel workers do one-at-a-time
locked linked list leapfrog.

> What I meant was that the complex patch can make up for the fact that
> direct I/O presumably won't ever have an equivalent to simple
> read-ahead. Just by having a very flexible prefetching implementation
> (and without any special sequential access heuristics ever being
> required).

I'm not so sure, there are certainly opportunities in different layers
of the system.  I'm going to dust off a couple of experimental patches
(stuff I talked to Peter about back in Athens), and try to describe
some other vague ideas Andres and I have bounced around over the past
few years when chatting about what you lose when you turn on direct
I/O.  Basically, the stuff that we can't fix with "precise" I/O
streaming as I like to call it, where it might still be interesting to
think about opportunities to do fuzzier speculative lookahead.  I'll
start a new thread.