Horton

Re: Direct I/O

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Andrew Dunstan <andrew@dunslane.net>, Heikki Linnakangas <hlinnaka@iki.fi>, Noah Misch <noah@leadboat.com>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Justin Pryzby <pryzby@telsasoft.com>, pgsql-hackers@postgresql.org
Date: 2023-04-09T04:52:10Z
Lists: pgsql-hackers 
Indeed, I can't reproduce this with (our) checksums on.  I also can't
reproduce it with O_DIRECT off.  I also can't reproduce it if I use
"mkdir pgdata && chattr +C pgdata && initdb -D pgdata" to have a
pgdata directory with copy-on-write and (their) checksums disabled.
But it reproduces quite easily with COW on (default behaviour) with
io_direct=data, debug_parallel_query=debug, create table as ...;
update ...; select count(*) ...; from that test.

Unfortunately my mental model of btrfs is extremely limited, basically
just "something a bit like ZFS".  FWIW I've been casually following
along with OpenZFS's ongoing O_DIRECT project, and I know that the
plan there is to make a temporary stable copy if checksums and other
features are on (a bit like PostgreSQL does for the same reason, as
you reminded us).  Time will tell how that works out but it *seems*
like all available modes would therefore work correctly for us, with
different tradeoffs (ie if you want the fastest zero-copy I/O, don't
use checksums, compression, etc).

Here, btrfs seems to be taking a different path that I can't quite
make out...  I see no warning/error about a checksum failure like [1],
and we apparently managed to read something other than a mix of the
old and new page contents (which, based on your hypothesis, should
just leave it indeterminate whether the hint bit changes were captured
or not, and the rest of the page should be stable, right).  It's like
the page time-travelled or got scrambled in some other way, but it
didn't tell us?  I'll try to dig further...

[1] https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/Gotchas.html#Direct_IO_and_CRCs

Commits

  1. Rename hook functions for debug_io_direct to match variable name.

  2. Rename io_direct to debug_io_direct.

  3. Skip the 004_io_direct.pl test if a pre-flight check fails.

  4. Use higher wal_level for 004_io_direct.pl.

  5. Skip \password TAP test on old IPC::Run versions

  6. Add io_direct setting (developer-only).

  7. Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.

  8. Add palloc_aligned() to allow aligned memory allocations

  9. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.