Horton

Re: Direct I/O

Nasby, Jim <nasbyj@amazon.com>

View thread View on postgr.es
From: Jim Nasby <nasbyj@amazon.com>
To: Thomas Munro <thomas.munro@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2022-11-04T19:47:31Z
Lists: pgsql-hackers 
On 11/1/22 2:36 AM, Thomas Munro wrote:

> Hi,
>
> Here is a patch to allow PostgreSQL to use $SUBJECT.  It is from the

This is exciting to see! There's two other items to add to the TODO list 
before this would be ready for production:

1) work_mem. This is a significant impediment to scaling shared buffers 
the way you'd want to.

2) Clock sweep. Specifically, currently the only thing that drives 
usage_count is individual backends running the clock hand. On large 
systems with 75% of memory going to shared_buffers, that becomes a very 
significant problem, especially when the backend running the clock sweep 
is doing so in order to perform an operation like a b-tree page split. I 
suspect it shouldn't be too hard to deal with this issue by just having 
bgwriter or another bgworker proactively ensuring some reasonable number 
of buffers with usage_count=0 exist.


One other thing to be aware of: overflowing as SLRU becomes a massive 
problem if there isn't a filesystem backing the SLRU. Obviously only an 
issue if you try and apply DIO to SLRU files.

Commits

  1. Rename hook functions for debug_io_direct to match variable name.

  2. Rename io_direct to debug_io_direct.

  3. Skip the 004_io_direct.pl test if a pre-flight check fails.

  4. Use higher wal_level for 004_io_direct.pl.

  5. Skip \password TAP test on old IPC::Run versions

  6. Add io_direct setting (developer-only).

  7. Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.

  8. Add palloc_aligned() to allow aligned memory allocations

  9. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.