Horton

Re: Asynchronous and "direct" IO support for PostgreSQL.

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: pgsql-hackers@postgresql.org
Cc: Thomas Munro <thomas.munro@gmail.com>, David Rowley <dgrowleyml@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>
Date: 2021-09-01T05:56:59Z
Lists: pgsql-hackers

Attachments

Hi,

Attached is an updated patch AIO series. The major changes are:
- rebased onto master (Andres)
- lots of progress on posix AIO backend (Thomas)
- lots of progress towards a windows native AIO implementation - not yet quite
  merged (Thomas & David)
- considerably improved "worker" io_method (Thomas)
- some preliminary patches merged (Thomas) and thus dropped
- error handling overhaul, AIO references now use resource owners
- quite a few more localized bugfixes
- further CI improvements

Unfortunately there's a few tests that don't pass on windows. At least some of
those failures also happen on master - hence the alternative output file added
in the last commit.

Thanks to Thomas there's now a new wiki page for AIO support:
https://wiki.postgresql.org/wiki/AIO
It's currently mostly a shared todo list....

My own next steps are to try to get some of the preliminary patches merged
into master, and to address some of the higher level things that aren't yet
quite right with the AIO interface, and to split the "main" AIO patch into
smaller patches.

I hope that we soon send in a new version with native AIO support for
windows. I'm mostly interested in that to make sure that we get the shared
infrastructure right.

Melanie has some work improving bitmap heap scan AIO support and some IO stats
/ explain improvements.

I think a decent and reasonably simple example for the way the AIO interface
can be used to do faster IO is
v3-0028-aio-Use-AIO-in-nbtree-vacuum-scan.patch.gz which adds AIO for nbtree
vacuum. It's not perfectly polished, but I think it shows that it's not too
hard to add AIO usage to individual once the general infrastructure is in
place.

I've attached the code for posterity, but the series is large enough that I
don't think it makes sense to do that all that often... The code is at
https://github.com/anarazel/postgres/tree/aio

Greetings,

Andres Freund

Commits

  1. aio: Add README.md explaining higher level design

  2. bufmgr: Use AIO in StartReadBuffers()

  3. bufmgr: Implement AIO read support

  4. aio: Implement support for reads in smgr/md/fd

  5. aio: Add io_method=io_uring

  6. aio: Add io_method=worker

  7. aio: Infrastructure for io_method=worker

  8. aio: Add core asynchronous I/O infrastructure