Horton

Re: index prefetching

Peter Geoghegan <pg@bowt.ie>

View thread
From: Peter Geoghegan <pg@bowt.ie>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Tomas Vondra <tomas@vondra.me>, Andres Freund <andres@anarazel.de>, Nazir Bilal Yavuz <byavuz81@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Georgios <gkokolatos@protonmail.com>, Konstantin Knizhnik <knizhnik@garret.ru>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2025-08-12T22:50:58Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. aio: io_uring: Trigger async processing for large IOs

  2. read stream: Split decision about look ahead for AIO and combining

  3. read_stream: Only increase read-ahead distance when waiting for IO

  4. read_stream: Prevent distance from decaying too quickly

  5. Reduce ExecSeqScan* code size using pg_assume()

  6. Fix rare bug in read_stream.c's split IO handling.

  7. Fix multiranges to behave more like dependent types.

  8. Add EXPLAIN (MEMORY) to report planner memory consumption

  9. Optimize nbtree backward scan boundary cases.

  10. Increment xactCompletionCount during subtransaction abort.

  11. Add nbtree Valgrind buffer lock checks.

  12. Add nbtree high key "continuescan" optimization.

  13. Reduce pinning and buffer content locking for btree scans.

  14. Teach btree to handle ScalarArrayOpExpr quals natively.

Attachments

On Tue, Aug 12, 2025 at 5:22 PM Peter Geoghegan <pg@bowt.ie> wrote:
> There does seem to be something fishy going on with the patch here.  I can see
> strange inconsistencies in EXPLAIN ANALYZE output when the server is started
> with --debug_io_direct=data with the master, compared to what I see with the
> patch.

Attached is my working version of the patch, in case that helps anyone
with reproducing the problem.

Note that the nbtree changes are now included in this one
patch/commit. Definitely might make sense to revert to one patch per
index AM again later, but for now it's convenient to have one commit
that both adds the concept of amgetbatch, and removes nbtree's
btgettuple (since it bleeds into things like how indexam.c wants to do
mark and restore).

There are only fairly minor changes here. Most notably:

* Generalizes nbtree's _bt_drop_lock_and_maybe_pin, making it an
index-AM-generic thing I call index_batch_unlock.

Previous versions of this complex patch avoided the issue by always
holding on to a leaf page buffer pin, even when it wasn't truly
necessary (i.e. with plain index scans that use an MVCC snapshot).

It shouldn't be too hard to teach GiST to use index_batch_unlock to
continue dropping buffer pins on leaf pages, as before (with
gistgettuple). The hard part will be ordered GiST scans, and perhaps
every kind of GiST index-only scan (since in general index-only scans
cannot drop pins eagerly within index_batch_unlock, due to race
conditions with VACUUM concurrently setting VM bits all-visible).

* Replaces BufferMatches() with something a bit less invasive, which
works based on block numbers (not buffers).

* Various refinements to the way that nbtree deals with setting things
up using an existing batch.

In particular, the interface of _bt_readnextpage has been revised. It
now makes much more sense in a world where nbtree doesn't "own"
existing batches -- we no longer directly pass an existing batch to
_bt_readnextpage, and it no longer thinks it can clobber what is
actually an old batch.

-- 
Peter Geoghegan