Horton

Re: Direct I/O

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Christoph Berg <myon@debian.org>, Thomas Munro <thomas.munro@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2023-04-12T05:48:54Z
Lists: pgsql-hackers

Attachments

On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg <myon@debian.org> wrote:
> > I'm hitting a panic in t_004_io_direct. The build is running on
> > overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird
> > combination but has worked well for building everything over the last
> > decade. On Debian unstable:
> >
> > PANIC:  could not open file "pg_wal/000000010000000000000001": Invalid argument

> ... I have a new idea:  perhaps it is possible to try
> to open a file with O_DIRECT from perl, and if it fails like that,
> skip the test.  Looking into that now.

I think I have that working OK.  Any Perl hackers want to comment on
my use of IO::File (copied from examples on the internet that showed
how to use O_DIRECT)?  I am not much of a perl hacker but according to
my package manager, IO/File.pm came with perl itself.  And the Fcntl
eval trick that I copied from File::stat, and the perl-critic
suppression that requires?

I tested this on OpenBSD, which has no O_DIRECT, so that tests the
first reason to skip.

Does it skip OK on your system, for the second reason?  Should we be
more specific about the errno?

As far as I know, the only systems on the build farm that should be
affected by this change are the illumos boxen (they have O_DIRECT,
unlike Solaris, but perl's $^O couldn't tell the difference between
Solaris and illumos, so they didn't previously run this test).

One thing I resisted the urge to do is invent PG_TEST_SKIP, a sort of
anti-PG_TEST_EXTRA.  I think I'd rather hear about it if there is a
system out there that passes the pre-flight check, but fails later on,
because we'd better investigate why.  That's basically the point of
shipping this "developer only" feature long before serious use of it.

Commits

  1. Rename hook functions for debug_io_direct to match variable name.

  2. Rename io_direct to debug_io_direct.

  3. Skip the 004_io_direct.pl test if a pre-flight check fails.

  4. Use higher wal_level for 004_io_direct.pl.

  5. Skip \password TAP test on old IPC::Run versions

  6. Add io_direct setting (developer-only).

  7. Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.

  8. Add palloc_aligned() to allow aligned memory allocations

  9. initdb: When running CREATE DATABASE, use STRATEGY = WAL_COPY.