Horton

Re: Streaming I/O, vectored I/O (WIP)

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>, Andres Freund <andres@anarazel.de>, Melanie Plageman <melanieplageman@gmail.com>
Date: 2024-05-24T22:00:21Z
Lists: pgsql-hackers 
On Wed, Nov 29, 2023 at 1:17 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> Done.  I like it, I just feel a bit bad about moving the p*v()
> replacement functions around a couple of times already!  I figured it
> might as well be static inline even if we use the fallback (= Solaris
> and Windows).

Just for the record, since I'd said things like the above a few times
while writing about this stuff: Solaris 11.4.69 has gained preadv()
and pwritev().  That's interesting because it means that there will
soon be no liive Unixoid operating systems left without them, and the
fallback code in src/include/port/pg_iovec.h will, in practice, be
only for Windows.  I wondered if that might have implications for how
we code or comment stuff like that, but it still seems to make sense
as we have it.

(I don't think Windows can have a real synchronous implementation; the
kernel knows how to do scatter/gather, a feature implemented
specifically for databases, but only in asynchronous ("overlapped") +
direct I/O mode, a difference I don't know how to hide at this level.
In later AIO work we should be able to use it as intended, but not by
pretending to be Unix like this.)

Commits

  1. Fix typos and incorrect type in read_stream.c

  2. Use streaming I/O in pg_prewarm.

  3. Provide API for streaming relation data.

  4. Provide vectored variant of ReadBuffer().

  5. Provide vectored variants of smgrread() and smgrwrite().

  6. Provide multi-block smgrprefetch().

  7. Provide vectored variants of FileRead() and FileWrite().

  8. Provide helper for retrying partial vectored I/O.

  9. Optimize pg_readv/pg_pwritev single vector case.

  10. bufmgr: Support multiple in-progress IOs by using resowner