Re: Streaming I/O, vectored I/O (WIP)

Thomas Munro <thomas.munro@gmail.com>

From: Thomas Munro <thomas.munro@gmail.com>

To: Dilip Kumar <dilipbalaut@gmail.com>

Cc: Robert Haas <robertmhaas@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, pgsql-hackers <pgsql-hackers@postgresql.org>, Melanie Plageman <melanieplageman@gmail.com>, Andres Freund <andres@anarazel.de>

Date: 2024-03-12T13:02:35Z

Lists: pgsql-hackers

Attachments

random.txt (text/plain)
v7-0001-Provide-vectored-variant-of-ReadBuffer.patch (application/octet-stream) patch v7-0001
v7-0002-Provide-API-for-streaming-reads-of-relations.patch (application/octet-stream) patch v7-0002
v7-0003-Use-streaming-reads-in-pg_prewarm.patch (application/octet-stream) patch v7-0003
v7-0004-WIP-Use-streaming-reads-in-heapam-scans.patch (application/octet-stream) patch v7-0004
v7-0005-WIP-Use-streaming-reads-in-nbtree-vacuum-scan.patch (application/octet-stream) patch v7-0005
v7-0006-WIP-Use-streaming-reads-in-bitmap-heapscan.patch (application/octet-stream) patch v7-0006
v7-0007-WIP-pg_buffercache_invalidate.patch (application/octet-stream) patch v7-0007
random.sql (application/octet-stream)

On Tue, Mar 12, 2024 at 11:39 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> + actual_nblocks = operation->nblocks = *nblocks = i + 1;
> (Dilip: I think we should break after this?)

In the next loop, i < actual_nblocks is false so the loop terminates.
But yeah that was a bit obscure, so I have added an explicit break.

Here also is a new version also of the streaming_read.c patch.  This
is based on feedback from the bitmap heap scan thread, where Tomas and
Melanie noticed some problems when comparing effective_io_concurrency
= 0 and 1 against master.  The attached random.sql exercises a bunch
of random scans with different settings, and random.txt shows the
resulting system calls, unpatched vs patched.  Looking at it that way,
I was able to tweak the coding until I had the same behaviour as
master, except with fewer system calls wherever possible due to
coalescing or suppressing sequential advice.

Commits

Fix typos and incorrect type in read_stream.c
- 2ea4b2927722 17.0 landed
Use streaming I/O in pg_prewarm.
- 3a352df05e65 17.0 landed
Provide API for streaming relation data.
- b5a9b18cd0bc 17.0 landed
Provide vectored variant of ReadBuffer().
- 210622c60e1a 17.0 landed
Provide vectored variants of smgrread() and smgrwrite().
- 4908c5872059 17.0 landed
Provide multi-block smgrprefetch().
- b485ad7f07c8 17.0 landed
Provide vectored variants of FileRead() and FileWrite().
- 871fe4917e1e 17.0 landed
Provide helper for retrying partial vectored I/O.
- 0c6be59f5e34 17.0 landed
Optimize pg_readv/pg_pwritev single vector case.
- 15c9ac362993 17.0 landed
bufmgr: Support multiple in-progress IOs by using resowner
- 12f3867f5534 16.0 cited