Horton

Re: Direct I/O

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Greg Stark <stark@mit.edu>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>, Andrew Dunstan <andrew@dunslane.net>, Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>, Christoph Berg <myon@debian.org>, mikael.kjellstrom@gmail.com, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2023-05-04T00:10:46Z
Lists: pgsql-hackers 
On Wed, Apr 19, 2023 at 7:35 AM Greg Stark <stark@mit.edu> wrote:
> On Mon, 17 Apr 2023 at 17:45, Thomas Munro <thomas.munro@gmail.com> wrote:
> > (2) without a page cache, you really need to size your shared_buffers
> > adequately and we can't do that automatically.
>
> Well.... I'm more optimistic... That may not always be impossible.
> We've already added the ability to add more shared memory after
> startup. We could implement the ability to add or remove shared buffer
> segments after startup.

Yeah, there are examples of systems going back decades with multiple
buffer pools.  In some you can add more space later, and in some you
can also configure pools with different block sizes (imagine if you
could set your extremely OLTP tables to use 4KB blocks for reduced
write amplification and then perhaps even also promise that your
storage doesn't need FPIs for that size because you know it's
perfectly safe™, and imagine if you could set some big write-only
history tables to use 32KB blocks because some compression scheme
works better, etc), and you might also want different cache
replacement algorithms in different pools.  Complex and advanced stuff
no doubt and I'm not suggesting that's anywhere near a reasonable
thing for us to think about now (as a matter of fact in another thread
you can find me arguing for fully unifying our existing segregated
SLRU buffer pools with the one true buffer pool), but since we're
talking pie-in-the-sky ideas around the water cooler...

Commits

  1. Rename hook functions for debug_io_direct to match variable name.

  2. Rename io_direct to debug_io_direct.

  3. Skip the 004_io_direct.pl test if a pre-flight check fails.

  4. Use higher wal_level for 004_io_direct.pl.

  5. Skip \password TAP test on old IPC::Run versions

  6. Add io_direct setting (developer-only).

  7. Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.

  8. Add palloc_aligned() to allow aligned memory allocations

  9. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.