Horton

Re: index prefetching

Peter Geoghegan <pg@bowt.ie>

View thread View on postgr.es
From: Peter Geoghegan <pg@bowt.ie>
To: Andres Freund <andres@anarazel.de>
Cc: Tomas Vondra <tomas@vondra.me>, Alexandre Felipe <o.alexandre.felipe@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, Nazir Bilal Yavuz <byavuz81@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Georgios <gkokolatos@protonmail.com>, Konstantin Knizhnik <knizhnik@garret.ru>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2026-04-01T22:50:15Z
Lists: pgsql-hackers

Attachments

On Wed, Apr 1, 2026 at 11:51 AM Peter Geoghegan <pg@bowt.ie> wrote:
> I plan on clarifying things in this area for v20. It will fully
> embrace the idea that IndexFetchTableData is just the table AM
> specific part of IndexScanDescData. This will allow me to move most of
> the batch-related calls currently in indexam.c over to heapam/the
> table AM. That will make it very clear that the table AM fully owns
> the batch ring buffer (there'll continue to be indexam.c calls to do
> things like take a mark, since that is implemented in a way that's
> already table AM agnostic).

Attached is v20, which does things this way. Now it's completely clear
that the scan's batch ring buffer is fully controlled by the table AM.

There are now only 2 remaining indexbatch.c functions that get called
from indexam.c. Since these are both trivial, highly generic
functions, restructuring things so that heapam would call these two
seemed unnecessary.

> One problem with the current design is that it still treats
> IndexFetchTableData as not just the table AM piece of
> IndexScanDescData. I now realize that it works that way purely so we
> can avoid changing anything about index_fetch_tuple. But, as you say,
> we really *should* do that anyway -- it shouldn't need its own
> IndexFetchTableData (or its own IndexScanDescData), since it literally
> doesn't have anything to do with index scans.

v20 also repurposes and renames index_fetch_tuple (the new name is
fetch_tid). This clarifies that it's a specialized function used only
by callers that inherently need to pass their own TID (the 2 remaining
constraint enforcement callers). These callers don't really perform an
index scan at all (the TID might come from an index, but clearly
_bt_check_unique isn't performing an index scan, by any reasonable
definition).

IndexFetchTableData is no longer used by fetch_tid or its callers. As
discussed, this change makes it possible to pass an IndexScanDescData
pointer (not a IndexFetchTableData pointer) through to heapam
functions like heapam_index_fetch_reset and heapam_index_fetch_end
(since doing so won't break index_fetch_tuple/fetch_tid, now that
their dependency on IndexFetchTableData has been broken).

One consequence of this refactoring is that we can no longer just call
heapam_index_fetch_reset from indexam.c when restoring a mark -- that
will reset the scan's batchringbuf, which isn't appropriate here (we
usually don't change the batchringbuf at all, in the happy path we
only need to override scanPos with markPos). indexam.c now calls a new
heapam_index_fetch_restrpos function (through its table AM shim)
instead -- heapam handles restoring the mark on its behalf.

The addition of this new heapam_index_fetch_restrpos function seems
strictly better to me. Restoring a mark in v19 meant resetting
xs_vm_items back to 1 -- which is only appropriate during a true
rescan. As in many other areas, the table AM can fully see what's
going on (it has all the relevant context), which allows it to do
exactly the right thing (layering that obscures useful information
about what's really going on is generally something we want to avoid
IMV).

--
Peter Geoghegan

Commits

  1. read stream: Split decision about look ahead for AIO and combining

  2. read_stream: Only increase read-ahead distance when waiting for IO

  3. aio: io_uring: Trigger async processing for large IOs

  4. heapam: Keep buffer pins across index scan resets.

  5. heapam: Track heap block in IndexFetchHeapData.

  6. Move heapam_handler.c index scan code to new file.

  7. Rename heapam_index_fetch_tuple argument for clarity.

  8. Optimize fast-path FK checks with batched index probes

  9. read_stream: Prevent distance from decaying too quickly

  10. read_stream: Issue IO synchronously while in fast path

  11. bufmgr: Return whether WaitReadBuffers() needed to wait

  12. aio: io_uring: Allow IO methods to check if IO completed in the background

  13. bufmgr: Make UnlockReleaseBuffer() more efficient

  14. Add fake LSN support to hash index AM.

  15. Make IndexScanInstrumentation a pointer in executor scan nodes.

  16. Use fake LSNs to improve nbtree dropPin behavior.

  17. Move fake LSN infrastructure out of GiST.

  18. Use simplehash for backend-private buffer pin refcounts.

  19. nbtree: Avoid allocating _bt_search stack.

  20. bufmgr: Fix use of wrong variable in GetPrivateRefCountEntrySlow()

  21. Conditional locking in pgaio_worker_submit_internal

  22. Reduce ExecSeqScan* code size using pg_assume()

  23. Fix rare bug in read_stream.c's split IO handling.

  24. Remove HeapBitmapScan's skip_fetch optimization

  25. Optimize nbtree backwards scans.

  26. Fix multiranges to behave more like dependent types.

  27. Add EXPLAIN (MEMORY) to report planner memory consumption

  28. Optimize nbtree backward scan boundary cases.

  29. Increment xactCompletionCount during subtransaction abort.

  30. Add nbtree Valgrind buffer lock checks.

  31. Add nbtree high key "continuescan" optimization.

  32. Reduce pinning and buffer content locking for btree scans.

  33. Teach btree to handle ScalarArrayOpExpr quals natively.