Horton

Re: index prefetching

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Tomas Vondra <tomas@vondra.me>, Robert Haas <robertmhaas@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Georgios <gkokolatos@protonmail.com>, Thomas Munro <thomas.munro@gmail.com>, Konstantin Knizhnik <knizhnik@garret.ru>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2025-07-16T22:18:39Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. aio: io_uring: Trigger async processing for large IOs

  2. read stream: Split decision about look ahead for AIO and combining

  3. read_stream: Only increase read-ahead distance when waiting for IO

  4. read_stream: Prevent distance from decaying too quickly

  5. Reduce ExecSeqScan* code size using pg_assume()

  6. Fix rare bug in read_stream.c's split IO handling.

  7. Fix multiranges to behave more like dependent types.

  8. Add EXPLAIN (MEMORY) to report planner memory consumption

  9. Optimize nbtree backward scan boundary cases.

  10. Increment xactCompletionCount during subtransaction abort.

  11. Add nbtree Valgrind buffer lock checks.

  12. Add nbtree high key "continuescan" optimization.

  13. Reduce pinning and buffer content locking for btree scans.

  14. Teach btree to handle ScalarArrayOpExpr quals natively.

Hi,

On 2025-07-16 17:47:53 -0400, Peter Geoghegan wrote:
> On Wed, Jul 16, 2025 at 5:41 PM Andres Freund <andres@anarazel.de> wrote:
> > I don't mean the index tids, but how the read stream is fed block numbers. In
> > the "complex" patch that's done by index_scan_stream_read_next(). And the
> > block number it returns is simply
> >
> >       return ItemPointerGetBlockNumber(tid);
> >
> > without the table AM having any way of influencing that. Which means that if
> > your table AM does not use the block number of the tid 1:1 as the real block
> > number, the fetched block will be completely bogus.
> 
> How is that handled when such a table AM uses the existing amgettuple
> interface?

There's no problem today - the indexams never use the tids to look up blocks
themselves. They're always passed to the tableam to do so (via
table_index_fetch_tuple() etc). I.e. the translation from TIDs to specific
blocks & buffers happens entirely inside the tableam, therefore the tableam
can choose to not use a 1:1 mapping or even to not use any buffers at all.


> I think that it shouldn't be hard to implement an opt-out
> of prefetching for such table AMs, so at least you won't fetch random
> garbage.

I don't think that's the right answer here. ISTM the layering in both patches
just isn't quite correct right now. The read stream shouldn't be "filled" with
table buffers by index code, it needs to be filled by tableam specific code.


> Right now, the amgetbatch interface is oriented around returning TIDs.
> Obviously it works that way because that's what heapam expects, and
> what amgettuple (which I'd like to replace with amgetbatch) does.

ISTM the right answer would be to allow the tableam to get the batches,
without indexam feeding the read stream.  That, perhaps not so coincidentally,
is also what's needed for batching heap page locking and and HOT search.

I think this means that it has to be the tableam that creates the read stream
and that does the work that's currently done in index_scan_stream_read_next(),
i.e. the translation from TID to whatever resources are required by the
tableam. Which presumably would include the tableam calling
index_batch_getnext().

Greetings,

Andres Freund