Horton

Re: Streaming I/O, vectored I/O (WIP)

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Andres Freund <andres@anarazel.de>, pgsql-hackers <pgsql-hackers@postgresql.org>, Melanie Plageman <melanieplageman@gmail.com>
Date: 2023-12-11T09:12:05Z
Lists: pgsql-hackers

Attachments

On Sat, Dec 9, 2023 at 10:23 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> Ok, works for me.

I finished up making a few more improvements:

1.  I eventually figured out how to generalise
compute_remaining_iovec() (as I now call it) so that the existing
pg_pwritev_with_retry() in file_utils.c could also use it, so that's
now done in a patch of its own.

2.  FileReadV/FileWriteV patch:

 * further simplification of the traditional ENOSPC 'guess'
 * unconstify() changed to raw cast (pending [1])
 * fixed the DO_DB()-wrapped debugging code

3.  smgrreadv/smgrwritev patch:

 * improved ENOSPC handling
 * improve description of EOF and ENOSPC handling
 * fixed the sizes reported in dtrace static probes
 * fixed some words in the docs about that
 * changed error messages to refer to "blocks %u..%u"

4.  smgrprefetch-with-nblocks patch has no change, hasn't drawn any
comments hopefully because it is uncontroversial.

I'm planning to commit these fairly soon.

[1] https://www.postgresql.org/message-id/flat/CA%2BhUKGK3OXFjkOyZiw-DgL2bUqk9by1uGuCnViJX786W%2BfyDSw%40mail.gmail.com

Commits

  1. Fix typos and incorrect type in read_stream.c

  2. Use streaming I/O in pg_prewarm.

  3. Provide API for streaming relation data.

  4. Provide vectored variant of ReadBuffer().

  5. Provide vectored variants of smgrread() and smgrwrite().

  6. Provide multi-block smgrprefetch().

  7. Provide vectored variants of FileRead() and FileWrite().

  8. Provide helper for retrying partial vectored I/O.

  9. Optimize pg_readv/pg_pwritev single vector case.

  10. bufmgr: Support multiple in-progress IOs by using resowner