Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. aio: Fix assertion, clarify README

  2. aio: Fix reference to outdated name

  3. aio: Fix possible state confusions due to interrupt processing

  4. aio: Improve debug logging around waiting for IOs

  5. aio: Fix crash potential for pg_aios views due to late state update

  6. Increase BAS_BULKREAD based on effective_io_concurrency

  7. localbuf: Add Valgrind buffer access instrumentation

  8. aio: Make AIO more compatible with valgrind

  9. aio: Avoid spurious coverity warning

  10. tests: Fix incompatibility of test_aio with *_FORCE_RELEASE

  11. tests: Cope with WARNINGs during failed CREATE DB on windows

  12. aio: Add errcontext for processing I/Os for another backend

  13. aio: Add README.md explaining higher level design

  14. aio: Minor comment improvements

  15. aio: Add test_aio module

  16. aio: Add pg_aios view

  17. docs: Add acronym and glossary entries for I/O and AIO

  18. Enable IO concurrency on all systems

  19. read_stream: Introduce and use optional batchmode support

  20. docs: Reframe track_io_timing related docs as wait time

  21. bufmgr: Use AIO in StartReadBuffers()

  22. bufmgr: Implement AIO read support

  23. aio: Add WARNING result status

  24. Let caller of PageIsVerified() control ignore_checksum_failure

  25. pgstat: Allow checksum errors to be reported in critical sections

  26. Add errhint_internal()

  27. localbuf: Track pincount in BufferDesc as well

  28. aio, bufmgr: Comment fixes/improvements

  29. Fix mis-attribution of checksum failure stats to the wrong database

  30. aio: Implement support for reads in smgr/md/fd

  31. aio: Add io_method=io_uring

  32. aio: Add liburing dependency

  33. aio: Rename pgaio_io_prep_* to pgaio_io_start_*

  34. aio: Pass result of local callbacks to ->report_return

  35. aio: Be more paranoid about interrupts

  36. Redefine max_files_per_process to control additionally opened files

  37. aio: Change prefix of PgAioResultStatus values to PGAIO_RS_

  38. bufmgr: Improve stats when a buffer is read in concurrently

  39. aio: Add io_method=worker

  40. aio: Infrastructure for io_method=worker

  41. aio: Add core asynchronous I/O infrastructure

  42. aio: Basic subsystem initialization

  43. tests: Expand temp table tests to some pin related matters

  44. localbuf: Introduce FlushLocalBuffer()

  45. localbuf: Introduce TerminateLocalBufferIO()

  46. localbuf: Fix dangerous coding pattern in GetLocalVictimBuffer()

  47. localbuf: Introduce StartLocalBufferIO()

  48. localbuf: Introduce InvalidateLocalBuffer()

  49. Allow lwlocks to be disowned

  50. Make jsonb casts to scalar types translate JSON null to SQL NULL.

  51. bufmgr/smgr: Don't cross segment boundaries in StartReadBuffers()

  52. Use aux process resource owner in walsender

  53. bufmgr: Return early in ScheduleBufferTagForWriteback() if fsync=off

  1. AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-01T06:27:50Z

    Hi,

It's been quite a while since the last version of the AIO patchset that I have
posted.  Of course parts of the larger project have since gone upstream [1].

A lot of time since the last versions was spent understanding the performance
characteristics of using AIO with WAL and understanding some other odd
performance characteristics I didn't understand. I think I mostly understand
that now and what the design implications for an AIO subsystem are.

The prototype I had been working on unfortunately suffered from a few design
issues that weren't trivial to fix.

The biggest was that each backend could essentially have hard references to
unbounded numbers of "AIO handles" and that these references prevented these
handles from being reused. Because "AIO handles" have to live in shared memory
(so other backends can wait on them, that IO workers can perform them, etc)
that's obviously an issue.  There was always a way to just run out of AIO
handles.  I went through quite a few iterations of a design for how to resolve
that - I think I finally got there.

Another significant issue was that when I wrote the AIO prototype,
bufmgr.c/smgr.c/md.c only issued IOs in BLCKSZ increments, with the AIO
subsystem merging them into larger IOs.  Thomas et al's work around streaming
read make bufmgr.c issue larger IOs - which is good for performance. But it
was surprisingly hard to fit into my older design.


It took me much longer than I had hoped to address these issues in
prototype. In the end I made progress by working on a rewriting the patchset
from scratch (well, with a bit of copy & paste).

The main reason I had previously implemented WAL AIO etc was to know the
design implications - but now that they're somewhat understood, I'm planning
to keep the patchset much smaller, with the goal of making it upstreamable.


While making v2 somewhat presentable I unfortunately found a few more design
issues - they're now mostly resolved, I think. But I only resolved the last
one a few hours ago, who knows what a few nights of sleeping on it will
bring. Unfortunately that prevented me from doing some of the polishing that I
had wanted to finish...


Because of the aforementioned move, I currently do not have access to my
workstation. I just have access to my laptop - which has enough thermal issues
to make benchmarks not particularly reliable.


So here are just a few teaser numbers, on an PCIe v4 NVMe SSD, note however
that this is with the BAS_BULKREAD size increased, with the default 256kB, we
can only keep one IO in flight at a time (due to io_combine_limit building
larger IOs) - we'll need to do something better than this, but that's yet
another separate discussion.


Workload: pg_prewarm('pgbench_accounts') of a scale 5k database, which is
bigger than memory:

                                time
master:                         59.097
aio v2.0, worker:               11.211
aio v2.0, uring *:              19.991
aio v2.0, direct, worker:       09.617
aio v2.0, direct, uring *:      09.802

Workload: SELECT sum(abalance) FROM pgbench_accounts;

                                0 workers       1 worker        2 workers       4 workers
master:                         65.753          33.246          21.095          12.918
aio v2.0, worker:               21.519          12.636          10.450          10.004
aio v2.0, uring*:               31.446          17.745          12.889          10.395
aio v2.0, uring**               23.497          13.824          10.881          10.589
aio v2.0, direct, worker:       22.377          11.989          09.915          09.772
aio v2.0, direct, uring*:       24.502          12.603          10.058          09.759

* the reason io_uring is slower is that workers effectively parallelize
  *memcpy, at the cost of increased CPU usage
** a simple heuristic to use IOSQE_ASYNC to force some parallelism of memcpys



Workload: checkpointing ~20GB of dirty data, mostly sequential:

                                time
master:                         10.209
aio v2.0, worker:               05.391
aio v2.0, uring:                04.593
aio v2.0, direct, worker:       07.745
aio v2.0, direct, uring:        03.351


To solve the issue with an unbounded number of AIO references there are few
changes compared to the prior approach:

1) Only one AIO handle can be "handed out" to a backend, without being
   defined. Previously the process of getting an AIO handle wasn't super
   lightweight, which made it appealing to cache AIO handles - which was one
   part of the problem for running out of AIO handles.

2) Nothing in a backend can force a "defined" AIO handle (i.e. one that is a
   valid operation) to stay around, it's always possible to execute the AIO
   operation and then reuse the handle.  This provides a forward guarantee, by
   ensuring that completing AIOs can free up handles (previously they couldn't
   be reused until the backend local reference was released).

3) Callbacks on AIOs are not allowed to error out anymore, unless it's ok to
   take the server down.

4) Obviously some code needs to know the result of AIO operation and be able
   to error out. To allow for that the issuer of an AIO can provide a pointer
   to local memory that'll receive the result of an AIO, including details
   about what kind of errors occurred (possible errors are e.g. a read failing
   or a buffer's checksum validation failing).


In the next few days I'll add a bunch more documentation and comments as well
as some better perf numbers (assuming my workstation survived...).

Besides that, I am planning to introduce "io_method=sync", which will just
execute IO synchrously. Besides that being a good capability to have, it'll
also make it more sensible to split off worker mode support into its own
commit(s).


Greetings,

Andres Freund


[1] bulk relation extension, streaming read
[2] personal health challenges, family health challenges and now moving from
    the US West Coast to the East Coast, ...

  2. Re: AIO v2.0

    Heikki Linnakangas <hlinnaka@iki.fi> — 2024-09-02T10:03:07Z

    On 01/09/2024 09:27, Andres Freund wrote:
> The main reason I had previously implemented WAL AIO etc was to know the
> design implications - but now that they're somewhat understood, I'm planning
> to keep the patchset much smaller, with the goal of making it upstreamable.

+1 on that approach.

> To solve the issue with an unbounded number of AIO references there are few
> changes compared to the prior approach:
> 
> 1) Only one AIO handle can be "handed out" to a backend, without being
>     defined. Previously the process of getting an AIO handle wasn't super
>     lightweight, which made it appealing to cache AIO handles - which was one
>     part of the problem for running out of AIO handles.
> 
> 2) Nothing in a backend can force a "defined" AIO handle (i.e. one that is a
>     valid operation) to stay around, it's always possible to execute the AIO
>     operation and then reuse the handle.  This provides a forward guarantee, by
>     ensuring that completing AIOs can free up handles (previously they couldn't
>     be reused until the backend local reference was released).
> 
> 3) Callbacks on AIOs are not allowed to error out anymore, unless it's ok to
>     take the server down.
> 
> 4) Obviously some code needs to know the result of AIO operation and be able
>     to error out. To allow for that the issuer of an AIO can provide a pointer
>     to local memory that'll receive the result of an AIO, including details
>     about what kind of errors occurred (possible errors are e.g. a read failing
>     or a buffer's checksum validation failing).
> 
> 
> In the next few days I'll add a bunch more documentation and comments as well
> as some better perf numbers (assuming my workstation survived...).

Yeah, a high-level README would be nice. Without that, it's hard to 
follow what "handed out" and "defined" above means for example.

A few quick comments the patches:

v2.0-0001-bufmgr-Return-early-in-ScheduleBufferTagForWrit.patch

+1, this seems ready to be committed right away.

v2.0-0002-Allow-lwlocks-to-be-unowned.patch

With LOCK_DEBUG, LWLock->owner will point to the backend that acquired 
the lock, but it doesn't own it anymore. That's reasonable, but maybe 
add a boolean to the LWLock to mark whether the lock is currently owned 
or not.

The LWLockReleaseOwnership() name is a bit confusing together with 
LWLockReleaseUnowned() and LWLockrelease(). From the names, you might 
think that they all release the lock, but LWLockReleaseOwnership() just 
disassociates it from the current process. Rename it to LWLockDisown() 
perhaps.

v2.0-0003-Use-aux-process-resource-owner-in-walsender.patch

+1. The old comment "We don't currently need any ResourceOwner in a 
walsender process" was a bit misleading, because the walsender did 
create the short-lived "base backup" resource owner, so it's nice to get 
that fixed.

v2.0-0008-aio-Skeleton-IO-worker-infrastructure.patch

My refactoring around postmaster.c child process handling will conflict 
with this [1]. Not in any fundamental way, but can I ask you to review 
those patch, please? After those patches, AIO workers should also have 
PMChild slots (formerly known as Backend structs).

[1] 
https://www.postgresql.org/message-id/a102f15f-eac4-4ff2-af02-f9ff209ec66f@iki.fi

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  3. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-03T14:29:07Z

    Hi,

On 2024-09-02 13:03:07 +0300, Heikki Linnakangas wrote:
> On 01/09/2024 09:27, Andres Freund wrote:
> > In the next few days I'll add a bunch more documentation and comments as well
> > as some better perf numbers (assuming my workstation survived...).
> 
> Yeah, a high-level README would be nice. Without that, it's hard to follow
> what "handed out" and "defined" above means for example.

Yea - I had actually written a bunch of that before, but then redesigns just
obsoleted most of it :(

FWIW, "handed out" is an IO handle acquired by code, which doesn't yet have an
operation associated with it. Once "defined" it actually could be - but isn't
yet - executed.


> A few quick comments the patches:
> 
> v2.0-0001-bufmgr-Return-early-in-ScheduleBufferTagForWrit.patch
> 
> +1, this seems ready to be committed right away.

Cool


> v2.0-0002-Allow-lwlocks-to-be-unowned.patch
> 
> With LOCK_DEBUG, LWLock->owner will point to the backend that acquired the
> lock, but it doesn't own it anymore. That's reasonable, but maybe add a
> boolean to the LWLock to mark whether the lock is currently owned or not.

Hm, not sure it's worth doing that...


> The LWLockReleaseOwnership() name is a bit confusing together with
> LWLockReleaseUnowned() and LWLockrelease(). From the names, you might think
> that they all release the lock, but LWLockReleaseOwnership() just
> disassociates it from the current process. Rename it to LWLockDisown()
> perhaps.

Yea, that makes sense.


> v2.0-0008-aio-Skeleton-IO-worker-infrastructure.patch
> 
> My refactoring around postmaster.c child process handling will conflict with
> this [1]. Not in any fundamental way, but can I ask you to review those
> patch, please? After those patches, AIO workers should also have PMChild
> slots (formerly known as Backend structs).

I'll try to do that soonish!

Greetings,

Andres Freund

  4. Re: AIO v2.0

    陈宗志 <baotiao@gmail.com> — 2024-09-04T17:37:34Z

    I hope there can be a high-level design document that includes a
description, high-level architecture, and low-level design.
This way, others can also participate in reviewing the code.
For example, which paths were modified in the AIO module? Is it the
path for writing WAL logs, or the path for flushing pages, etc.?

Also, I recommend keeping this patch as small as possible.
For example, the first step could be to introduce libaio only, without
considering io_uring, as that would make it too complex.

  5. Re: AIO v2.0

    David Rowley <dgrowleyml@gmail.com> — 2024-09-06T01:42:24Z

    On Sun, 1 Sept 2024 at 18:28, Andres Freund <andres@anarazel.de> wrote:
>                                 0 workers       1 worker        2 workers       4 workers
> master:                         65.753          33.246          21.095          12.918
> aio v2.0, worker:               21.519          12.636          10.450          10.004
> aio v2.0, uring*:               31.446          17.745          12.889          10.395
> aio v2.0, uring**               23.497          13.824          10.881          10.589
> aio v2.0, direct, worker:       22.377          11.989          09.915          09.772
> aio v2.0, direct, uring*:       24.502          12.603          10.058          09.759

I took this for a test drive on an AMD 3990x machine with a 1TB
Samsung 980 Pro SSD on PCIe 4.  I only tried io_method = io_uring, but
I did try with and without direct IO.

This machine has 64GB RAM and I was using ClickBench Q2 [1], which is
"SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits;"
(for some reason they use 0-based query IDs). This table is 64GBs
without indexes.

I'm seeing direct IO slower than buffered IO with smaller worker
counts. That's counter to what I would have expected as I'd have
expected the memcpys from the kernel space to be quite an overhead in
the buffered IO case. With larger worker counts the bottleneck is
certainly disk. The part that surprised me was that the bottleneck is
reached more quickly with buffered IO. I was seeing iotop going up to
5.54GB/s at higher worker counts.

times in milliseconds
workers    buffered    direct    cmp
0    58880    102852    57%
1    33622    53538    63%
2    24573    40436    61%
4    18557    27359    68%
8    14844    17330    86%
16    12491    12754    98%
32    11802    11956    99%
64    11895    11941    100%

Is there some other information I can provide to help this make sense?
(Or maybe it does already to you.)

David

[1] https://github.com/ClickHouse/ClickBench/blob/main/postgresql-tuned/queries.sql

  6. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-06T19:38:16Z

    Hi,

Attached is the next version of the patchset. Changes:

- added "sync" io method, the main benefit of that is that the main AIO commit
  doesn't need to include worker mode

- split worker and io_uring methods into their own commits

- added src/backend/storage/aio/README.md, explaining design constraints and
  the resulting design on a high level

- renamed LWLockReleaseOwnership as suggested by Heikki

- a bunch of small cleanups and improvements

There's plenty more to do, but I thought this would be a useful checkpoint.

Greetings,

Andres Freund

  7. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-06T19:47:35Z

    Hi,

On 2024-09-05 01:37:34 +0800, 陈宗志 wrote:
> I hope there can be a high-level design document that includes a
> description, high-level architecture, and low-level design.
> This way, others can also participate in reviewing the code.

Yep, that was already on my todo list. The version I just posted includes
that.


> For example, which paths were modified in the AIO module?
> Is it the path for writing WAL logs, or the path for flushing pages, etc.?

I don't think it's good to document this in a design document - that's just
bound to get out of date.

For now the patchset causes AIO to be used for

1) all users of read_stream.h, e.g. sequential scans

2) bgwriter / checkpointer, mainly to have way to exercise the write path. As
   mentioned in my email upthread, the code for that is in a somewhat rough
   shape as Thomas Munro is working on a more general abstraction for some of
   this.

The earlier patchset added a lot more AIO uses because I needed to know all
the design constraints. It e.g. added AIO use in WAL. While that allowed me to
learn a lot, it's not something that makes sense to continue working on for
now, as it requires a lot of work that's independent of AIO.  Thus I am
focusing on the above users for now.


> Also, I recommend keeping this patch as small as possible.

Yep. That's my goal (as mentioned upthread).


> For example, the first step could be to introduce libaio only, without
> considering io_uring, as that would make it too complex.

Currently the patchset doesn't contain libaio support and I am not planning to
work on using libaio. Nor do I think it makes sense for anybody else to do so
- libaio doesn't work for buffered IO, making it imo not particularly useful
for us.

The io_uring specific code isn't particularly complex / large compared to the
main AIO infrastructure.

Greetings,

Andres Freund

  8. Re: AIO v2.0

    Robert Pang <robertpang@google.com> — 2024-09-12T21:55:49Z

    Hi Andres

Thanks for the AIO patch update. I gave it a try and ran into a FATAL
in bgwriter when executing a benchmark.

2024-09-12 01:38:00.851 PDT [2780939] PANIC:  no more bbs
2024-09-12 01:38:00.854 PDT [2780473] LOG:  background writer process
(PID 2780939) was terminated by signal 6: Aborted
2024-09-12 01:38:00.854 PDT [2780473] LOG:  terminating any other
active server processes

I debugged a bit and found that BgBufferSync() is not capping the
batch size under io_bounce_buffers like BufferSync() for checkpoint.
Here is a small patch to fix it.

Best regards
Robert




On Fri, Sep 6, 2024 at 12:47 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2024-09-05 01:37:34 +0800, 陈宗志 wrote:
> > I hope there can be a high-level design document that includes a
> > description, high-level architecture, and low-level design.
> > This way, others can also participate in reviewing the code.
>
> Yep, that was already on my todo list. The version I just posted includes
> that.
>
>
> > For example, which paths were modified in the AIO module?
> > Is it the path for writing WAL logs, or the path for flushing pages, etc.?
>
> I don't think it's good to document this in a design document - that's just
> bound to get out of date.
>
> For now the patchset causes AIO to be used for
>
> 1) all users of read_stream.h, e.g. sequential scans
>
> 2) bgwriter / checkpointer, mainly to have way to exercise the write path. As
>    mentioned in my email upthread, the code for that is in a somewhat rough
>    shape as Thomas Munro is working on a more general abstraction for some of
>    this.
>
> The earlier patchset added a lot more AIO uses because I needed to know all
> the design constraints. It e.g. added AIO use in WAL. While that allowed me to
> learn a lot, it's not something that makes sense to continue working on for
> now, as it requires a lot of work that's independent of AIO.  Thus I am
> focusing on the above users for now.
>
>
> > Also, I recommend keeping this patch as small as possible.
>
> Yep. That's my goal (as mentioned upthread).
>
>
> > For example, the first step could be to introduce libaio only, without
> > considering io_uring, as that would make it too complex.
>
> Currently the patchset doesn't contain libaio support and I am not planning to
> work on using libaio. Nor do I think it makes sense for anybody else to do so
> - libaio doesn't work for buffered IO, making it imo not particularly useful
> for us.
>
> The io_uring specific code isn't particularly complex / large compared to the
> main AIO infrastructure.
>
> Greetings,
>
> Andres Freund
>
>

  9. Re: AIO v2.0

    Noah Misch <noah@leadboat.com> — 2024-09-16T14:43:49Z

    On Fri, Sep 06, 2024 at 03:38:16PM -0400, Andres Freund wrote:
> There's plenty more to do, but I thought this would be a useful checkpoint.

I find patches 1-5 are Ready for Committer.

> +typedef enum PgAioHandleState

This enum clarified a lot for me, so I wish I had read it before anything
else.  I recommend referring to it in README.md.  Would you also cover the
valid state transitions and which of them any backend can do vs. which are
specific to the defining backend?

> +{
> +	/* not in use */
> +	AHS_IDLE = 0,
> +
> +	/* returned by pgaio_io_get() */
> +	AHS_HANDED_OUT,
> +
> +	/* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
> +	AHS_DEFINED,
> +
> +	/* subjects prepare() callback has been called */
> +	AHS_PREPARED,
> +
> +	/* IO is being executed */
> +	AHS_IN_FLIGHT,

Let's align terms between functions and states those functions reach.  For
example, I recommend calling this state AHS_SUBMITTED, because
pgaio_io_prepare_submit() is the function reaching this state.
(Alternatively, use in_flight in the function name.)

> +
> +	/* IO finished, but result has not yet been processed */
> +	AHS_REAPED,
> +
> +	/* IO completed, shared completion has been called */
> +	AHS_COMPLETED_SHARED,
> +
> +	/* IO completed, local completion has been called */
> +	AHS_COMPLETED_LOCAL,
> +} PgAioHandleState;

> +void
> +pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
> +{
> +	PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
> +
> +	Assert(ioh->resowner);
> +
> +	ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
> +	ioh->resowner = NULL;
> +
> +	switch (ioh->state)
> +	{
> +		case AHS_IDLE:
> +			elog(ERROR, "unexpected");
> +			break;
> +		case AHS_HANDED_OUT:
> +			Assert(ioh == my_aio->handed_out_io || my_aio->handed_out_io == NULL);
> +
> +			if (ioh == my_aio->handed_out_io)
> +			{
> +				my_aio->handed_out_io = NULL;
> +				if (!on_error)
> +					elog(WARNING, "leaked AIO handle");
> +			}
> +
> +			pgaio_io_reclaim(ioh);
> +			break;
> +		case AHS_DEFINED:
> +		case AHS_PREPARED:
> +			/* XXX: Should we warn about this when is_commit? */

Yes.

> +			pgaio_submit_staged();
> +			break;
> +		case AHS_IN_FLIGHT:
> +		case AHS_REAPED:
> +		case AHS_COMPLETED_SHARED:
> +			/* this is expected to happen */
> +			break;
> +		case AHS_COMPLETED_LOCAL:
> +			/* XXX: unclear if this ought to be possible? */
> +			pgaio_io_reclaim(ioh);
> +			break;
> +	}

> +void
> +pgaio_io_ref_wait(PgAioHandleRef *ior)
> +{
> +	uint64		ref_generation;
> +	PgAioHandleState state;
> +	bool		am_owner;
> +	PgAioHandle *ioh;
> +
> +	ioh = pgaio_io_from_ref(ior, &ref_generation);
> +
> +	am_owner = ioh->owner_procno == MyProcNumber;
> +
> +
> +	if (pgaio_io_was_recycled(ioh, ref_generation, &state))
> +		return;
> +
> +	if (am_owner)
> +	{
> +		if (state == AHS_DEFINED || state == AHS_PREPARED)
> +		{
> +			/* XXX: Arguably this should be prevented by callers? */
> +			pgaio_submit_staged();

Agreed for AHS_DEFINED, if not both.  AHS_DEFINED here would suggest a past
longjmp out of pgaio_io_prepare() w/o a subxact rollback to cleanup.  Even so,
the next point might remove the need here:

> +void
> +pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op)
> +{
> +	Assert(ioh->state == AHS_HANDED_OUT);
> +	Assert(pgaio_io_has_subject(ioh));
> +
> +	ioh->op = op;
> +	ioh->state = AHS_DEFINED;
> +	ioh->result = 0;
> +
> +	/* allow a new IO to be staged */
> +	my_aio->handed_out_io = NULL;
> +
> +	pgaio_io_prepare_subject(ioh);
> +
> +	ioh->state = AHS_PREPARED;

As defense in depth, let's add a critical section from before assigning
AHS_DEFINED to here.  This code already needs to be safe for that (per
README.md).  When running outside a critical section, an ERROR in a subject
callback could leak the lwlock disowned in shared_buffer_prepare_common().  I
doubt there's a plausible way to reach that leak today, but future subject
callbacks could add risk over time.

> +if test "$with_liburing" = yes; then
> +  PKG_CHECK_MODULES(LIBURING, liburing)
> +fi

I used the attached makefile patch to build w/ liburing.

> +pgaio_uring_shmem_init(bool first_time)
> +{
> +	uint32		TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
> +	bool		found;
> +
> +	aio_uring_contexts = (PgAioUringContext *)
> +		ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
> +
> +	if (found)
> +		return;
> +
> +	for (int contextno = 0; contextno < TotalProcs; contextno++)
> +	{
> +		PgAioUringContext *context = &aio_uring_contexts[contextno];
> +		int			ret;
> +
> +		/*
> +		 * XXX: Probably worth sharing the WQ between the different rings,
> +		 * when supported by the kernel. Could also cause additional
> +		 * contention, I guess?
> +		 */
> +#if 0
> +		if (!AcquireExternalFD())
> +			elog(ERROR, "No external FD available");
> +#endif
> +		ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);

With EXEC_BACKEND, "make check PG_TEST_INITDB_EXTRA_OPTS=-cio_method=io_uring"
fails early:

2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG:  starting PostgreSQL 18devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 13.2.0-13) 13.2.0, 64-bit
2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG:  listening on Unix socket "/tmp/pg_regress-xgQOPH/.s.PGSQL.65312"
2024-09-15 12:46:08.203 PDT startup[2069423] LOG:  database system was shut down at 2024-09-15 12:46:07 PDT
2024-09-15 12:46:08.209 PDT client backend[2069425] [unknown] FATAL:  the database system is starting up
2024-09-15 12:46:08.222 PDT postmaster[2069397] LOG:  database system is ready to accept connections
2024-09-15 12:46:08.254 PDT autovacuum launcher[2069435] PANIC:  failed: -9/Bad file descriptor
2024-09-15 12:46:08.286 PDT client backend[2069444] [unknown] PANIC:  failed: -95/Operation not supported
2024-09-15 12:46:08.355 PDT client backend[2069455] [unknown] PANIC:  unexpected: -95/Operation not supported: No such file or directory
2024-09-15 12:46:08.370 PDT postmaster[2069397] LOG:  received fast shutdown request

I expect that's from io_uring_queue_init() stashing in shared memory a file
descriptor and mmap address, which aren't valid in EXEC_BACKEND children.
Reattaching descriptors and memory in each child may work, or one could just
block io_method=io_uring under EXEC_BACKEND.

> +pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
> +{
> +	struct io_uring *uring_instance = &my_shared_uring_context->io_uring_ring;
> +
> +	Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
> +
> +	for (int i = 0; i < num_staged_ios; i++)
> +	{
> +		PgAioHandle *ioh = staged_ios[i];
> +		struct io_uring_sqe *sqe;
> +
> +		sqe = io_uring_get_sqe(uring_instance);
> +
> +		pgaio_io_prepare_submit(ioh);
> +		pgaio_uring_sq_from_io(ioh, sqe);
> +	}
> +
> +	while (true)
> +	{
> +		int			ret;
> +
> +		pgstat_report_wait_start(WAIT_EVENT_AIO_SUBMIT);
> +		ret = io_uring_submit(uring_instance);
> +		pgstat_report_wait_end();
> +
> +		if (ret == -EINTR)
> +		{
> +			elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
> +			continue;
> +		}

Since io_uring_submit() is a wrapper around io_uring_enter(), this should also
retry on EAGAIN.  "man io_uring_enter" has:

    EAGAIN The kernel was unable to allocate memory for the request, or
    otherwise ran out of resources to handle it. The application should wait
    for some completions and try again.

> +FileStartWriteV(struct PgAioHandle *ioh, File file,
> +				int iovcnt, off_t offset,
> +				uint32 wait_event_info)
> +{
> +	int			returnCode;
> +	Vfd		   *vfdP;
> +
> +	Assert(FileIsValid(file));
> +
> +	DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
> +			   file, VfdCache[file].fileName,
> +			   (int64) offset,
> +			   iovcnt));
> +
> +	returnCode = FileAccess(file);
> +	if (returnCode < 0)
> +		return returnCode;
> +
> +	vfdP = &VfdCache[file];
> +
> +	/* FIXME: think about / reimplement  temp_file_limit */
> +
> +	pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
> +
> +	return 0;
> +}

FileStartWriteV() gets to state AHS_PREPARED, so let's align with the state
name by calling it FilePrepareWriteV (or FileWriteVPrepare or whatever).


For non-sync IO methods, I gather it's essential that a process other than the
IO definer be scanning for incomplete IOs and completing them.  Otherwise,
deadlocks like this would happen:

backend1 locks blk1 for non-IO reasons
backend2 locks blk2, starts AIO write
backend1 waits for lock on blk2 for non-IO reasons
backend2 waits for lock on blk1 for non-IO reasons

If that's right, in worker mode, the IO worker resolves that deadlock.  What
resolves it under io_uring?  Another process that happens to do
pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
make that happen systematically.  Could you add a mention of "deadlock" in the
comment at whichever code achieves that?


I could share more-tactical observations about patches 6-20, but they're
probably things you'd change without those observations.  Is there any
specific decision you'd like to settle before patch 6 exits WIP?

Thanks,
nm

  10. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-16T17:51:42Z

    Hi,

Thanks for the review!

On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> On Fri, Sep 06, 2024 at 03:38:16PM -0400, Andres Freund wrote:
> > There's plenty more to do, but I thought this would be a useful checkpoint.
>
> I find patches 1-5 are Ready for Committer.

Cool!


> > +typedef enum PgAioHandleState
>
> This enum clarified a lot for me, so I wish I had read it before anything
> else.  I recommend referring to it in README.md.

Makes sense.


> Would you also cover the valid state transitions and which of them any
> backend can do vs. which are specific to the defining backend?

Yea, we should. I earlier had something, but because details were still
changing it was hard to keep up2date.


> > +{
> > +	/* not in use */
> > +	AHS_IDLE = 0,
> > +
> > +	/* returned by pgaio_io_get() */
> > +	AHS_HANDED_OUT,
> > +
> > +	/* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
> > +	AHS_DEFINED,
> > +
> > +	/* subjects prepare() callback has been called */
> > +	AHS_PREPARED,
> > +
> > +	/* IO is being executed */
> > +	AHS_IN_FLIGHT,
>
> Let's align terms between functions and states those functions reach.  For
> example, I recommend calling this state AHS_SUBMITTED, because
> pgaio_io_prepare_submit() is the function reaching this state.
> (Alternatively, use in_flight in the function name.)

There used to be a separate SUBMITTED, but I removed it at some point as not
necessary anymore. Arguably it might be useful to re-introduce it so that
e.g. with worker mode one can tell the difference between the IO being queued
and the IO actually being processed.


> > +void
> > +pgaio_io_ref_wait(PgAioHandleRef *ior)
> > +{
> > +	uint64		ref_generation;
> > +	PgAioHandleState state;
> > +	bool		am_owner;
> > +	PgAioHandle *ioh;
> > +
> > +	ioh = pgaio_io_from_ref(ior, &ref_generation);
> > +
> > +	am_owner = ioh->owner_procno == MyProcNumber;
> > +
> > +
> > +	if (pgaio_io_was_recycled(ioh, ref_generation, &state))
> > +		return;
> > +
> > +	if (am_owner)
> > +	{
> > +		if (state == AHS_DEFINED || state == AHS_PREPARED)
> > +		{
> > +			/* XXX: Arguably this should be prevented by callers? */
> > +			pgaio_submit_staged();
>
> Agreed for AHS_DEFINED, if not both.  AHS_DEFINED here would suggest a past
> longjmp out of pgaio_io_prepare() w/o a subxact rollback to cleanup.

That, or not having submitted the IO.  One thing I've been thinking about as
being potentially helpful infrastructure is to have something similar to a
critical section, except that it asserts that one is not allowed to block or
forget submitting staged IOs.



> > +void
> > +pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op)
> > +{
> > +	Assert(ioh->state == AHS_HANDED_OUT);
> > +	Assert(pgaio_io_has_subject(ioh));
> > +
> > +	ioh->op = op;
> > +	ioh->state = AHS_DEFINED;
> > +	ioh->result = 0;
> > +
> > +	/* allow a new IO to be staged */
> > +	my_aio->handed_out_io = NULL;
> > +
> > +	pgaio_io_prepare_subject(ioh);
> > +
> > +	ioh->state = AHS_PREPARED;
>
> As defense in depth, let's add a critical section from before assigning
> AHS_DEFINED to here.  This code already needs to be safe for that (per
> README.md).  When running outside a critical section, an ERROR in a subject
> callback could leak the lwlock disowned in shared_buffer_prepare_common().  I
> doubt there's a plausible way to reach that leak today, but future subject
> callbacks could add risk over time.

Makes sense.


> > +if test "$with_liburing" = yes; then
> > +  PKG_CHECK_MODULES(LIBURING, liburing)
> > +fi
>
> I used the attached makefile patch to build w/ liburing.

Thanks, will incorporate.


> With EXEC_BACKEND, "make check PG_TEST_INITDB_EXTRA_OPTS=-cio_method=io_uring"
> fails early:

Right  - that's to be expected.

> 2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG:  starting PostgreSQL 18devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 13.2.0-13) 13.2.0, 64-bit
> 2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG:  listening on Unix socket "/tmp/pg_regress-xgQOPH/.s.PGSQL.65312"
> 2024-09-15 12:46:08.203 PDT startup[2069423] LOG:  database system was shut down at 2024-09-15 12:46:07 PDT
> 2024-09-15 12:46:08.209 PDT client backend[2069425] [unknown] FATAL:  the database system is starting up
> 2024-09-15 12:46:08.222 PDT postmaster[2069397] LOG:  database system is ready to accept connections
> 2024-09-15 12:46:08.254 PDT autovacuum launcher[2069435] PANIC:  failed: -9/Bad file descriptor
> 2024-09-15 12:46:08.286 PDT client backend[2069444] [unknown] PANIC:  failed: -95/Operation not supported
> 2024-09-15 12:46:08.355 PDT client backend[2069455] [unknown] PANIC:  unexpected: -95/Operation not supported: No such file or directory
> 2024-09-15 12:46:08.370 PDT postmaster[2069397] LOG:  received fast shutdown request
>
> I expect that's from io_uring_queue_init() stashing in shared memory a file
> descriptor and mmap address, which aren't valid in EXEC_BACKEND children.
> Reattaching descriptors and memory in each child may work, or one could just
> block io_method=io_uring under EXEC_BACKEND.

I think the latter option is saner - I don't think there's anything to be
gained by supporting io_uring in this situation. It's not like anybody will
use it for real-world workloads where performance matters. Nor would it be
useful fo portability testing.


> > +pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
> > +{

> > +		if (ret == -EINTR)
> > +		{
> > +			elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
> > +			continue;
> > +		}
>
> Since io_uring_submit() is a wrapper around io_uring_enter(), this should also
> retry on EAGAIN.  "man io_uring_enter" has:
>
>     EAGAIN The kernel was unable to allocate memory for the request, or
>     otherwise ran out of resources to handle it. The application should wait
>     for some completions and try again.

Hm. I'm not sure that makes sense. We only allow a limited number of IOs to be
in flight for each uring instance. That's different to a use of uring to
e.g. wait for incoming network data on thousands of sockets, where you could
have essentially unbounded amount of requests outstanding.

What would we wait for? What if we were holding a critical lock in that
moment? Would it be safe to just block for some completions? What if there's
actually no IO in progress?


> > +FileStartWriteV(struct PgAioHandle *ioh, File file,
> > +				int iovcnt, off_t offset,
> > +				uint32 wait_event_info)
> > +{
> > ...
>
> FileStartWriteV() gets to state AHS_PREPARED, so let's align with the state
> name by calling it FilePrepareWriteV (or FileWriteVPrepare or whatever).

Hm - that doesn't necessarily seem right to me. I don't think the caller
should assume that the IO will just be prepared and not already completed by
the time FileStartWriteV() returns - we might actually do the IO
synchronously.


> For non-sync IO methods, I gather it's essential that a process other than the
> IO definer be scanning for incomplete IOs and completing them.

Yep - it's something I've been fighting with / redesigning a *lot*.  Earlier
the AIO subsystem could transparently retry IOs, but that ends up being a
nightmare - or at least I couldn't find a way to not make it a
nightmare. There are two main complexities:

1) What if the IO is being completed in a critical section? We can't reopen
   the file in that situation. My initial fix for this was to defer retries,
   but that's problematic too:

2) Acquiring an IO needs to be able to guarantee forward progress. Because
   there's a limited number of IOs that means we need to be able to complete
   IOs while acquiring an IO. So we can't just keep the IO handle around -
   which in turn means that we'd need to save the state for retrying
   somewhere. Which would require some pre-allocated memory to save that
   state.

Thus I think it's actually better if we delegate retries to the callsites. I
was thinking that for partial reads of shared buffers we ought to not set
BM_IO_ERROR though...


> Otherwise, deadlocks like this would happen:

> backend1 locks blk1 for non-IO reasons
> backend2 locks blk2, starts AIO write
> backend1 waits for lock on blk2 for non-IO reasons
> backend2 waits for lock on blk1 for non-IO reasons
>
> If that's right, in worker mode, the IO worker resolves that deadlock.  What
> resolves it under io_uring?  Another process that happens to do
> pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> make that happen systematically.

Yea, it's code that I haven't forward ported yet. I think basically
LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
immediately acquire the lock and if the buffer has IO going on.


> I could share more-tactical observations about patches 6-20, but they're
> probably things you'd change without those observations.

Agreed.


> Is there any specific decision you'd like to settle before patch 6 exits
> WIP?

Patch 6 specifically? That I really mainly kept separate for review - it
doesn't seem particularly interesting to commit it earlier than 7, or do you
think differently?

In case you mean 6+7 or 6 to ~11, I can think of the following:

- I am worried about the need for bounce buffers for writes of checksummed
  buffers. That quickly ends up being a significant chunk of memory,
  particularly when using a small shared_buffers with a higher than default
  number of connection. I'm currently hacking up a prototype that'd prevent us
  from setting hint bits with just a share lock. I'm planning to start a
  separate thread about that.

- The header split doesn't yet quite seem right yet

- I'd like to implement retries in the later patches, to make sure that it
  doesn't have design implications

- Worker mode needs to be able to automatically adjust the number of running
  workers, I think - otherwise it's going to be too hard to tune.

- I think the PgAioHandles need to be slimmed down a bit - there's some design
  evolution visible that should not end up in the tree.

- I'm not sure that I like name "subject" for the different things AIO is
  performed for

- I am wondering if the need for pgaio_io_set_io_data_32() (to store the set
  of buffer ids that are affected by one IO) could be replaced by repurposing
  BufferDesc->freeNext or something along those lines. I don't like the amount
  of memory required for storing those arrays, even if it's not that much
  compared to needing space to store struct iovec[PG_IOV_MAX] for each AIO
  handle.

- I'd like to extend the test module to actually test more cases, it's too
  hard to reach some paths, particularly without [a lot] of users yet. That's
  not strictly a dependency of the earlier patches - since the initial patches
  can't actually do much in the way of IO.

- We shouldn't reserve AioHandles etc for io workers - but because different
  tpes of aux processes don't use a predetermined ProcNumber, that's not
  entirely trivial without adding more complexity.  I've actually wondered
  whether IO workes should be their own "top-level" kind of process, rather
  than an aux process. But that seems quite costly.

- Right now the io_uring mode has each backend's io_uring instance visible to
  each other process. That ends up using a fair number of FDs. That's OK from
  an efficiency perspective, but I think we'd need to add code to adjust the
  soft RLIMIT_NOFILE (it's set to 1024 on most distros because there are
  various programs that iterate over all possible FDs, causing significant
  slowdowns when the soft limit defaults to something high).  I earlier had a
  limited number of io_uring instances, but that added a fair amount of
  overhead because then submitting IO would require a lock.

  That again doesn't have to be solved as part of the earlier patches but
  might have some minor design impact.


Thanks again,

Andres Freund

  11. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-16T17:56:09Z

    Hi,

On 2024-09-12 14:55:49 -0700, Robert Pang wrote:
> Hi Andres
> 
> Thanks for the AIO patch update. I gave it a try and ran into a FATAL
> in bgwriter when executing a benchmark.
> 
> 2024-09-12 01:38:00.851 PDT [2780939] PANIC:  no more bbs
> 2024-09-12 01:38:00.854 PDT [2780473] LOG:  background writer process
> (PID 2780939) was terminated by signal 6: Aborted
> 2024-09-12 01:38:00.854 PDT [2780473] LOG:  terminating any other
> active server processes
> 
> I debugged a bit and found that BgBufferSync() is not capping the
> batch size under io_bounce_buffers like BufferSync() for checkpoint.
> Here is a small patch to fix it.

Good catch, thanks!


I am hoping (as described in my email to Noah a few minutes ago) that we can
get away from needing bounce buffers. They are a quite expensive solution to a
problem we made for ourselves...

Greetings,

Andres Freund

  12. Re: AIO v2.0

    Noah Misch <noah@leadboat.com> — 2024-09-17T18:08:19Z

    On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > On Fri, Sep 06, 2024 at 03:38:16PM -0400, Andres Freund wrote:

> > Reattaching descriptors and memory in each child may work, or one could just
> > block io_method=io_uring under EXEC_BACKEND.
> 
> I think the latter option is saner

Works for me.

> > > +pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
> > > +{
> 
> > > +		if (ret == -EINTR)
> > > +		{
> > > +			elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
> > > +			continue;
> > > +		}
> >
> > Since io_uring_submit() is a wrapper around io_uring_enter(), this should also
> > retry on EAGAIN.  "man io_uring_enter" has:
> >
> >     EAGAIN The kernel was unable to allocate memory for the request, or
> >     otherwise ran out of resources to handle it. The application should wait
> >     for some completions and try again.
> 
> Hm. I'm not sure that makes sense. We only allow a limited number of IOs to be
> in flight for each uring instance. That's different to a use of uring to
> e.g. wait for incoming network data on thousands of sockets, where you could
> have essentially unbounded amount of requests outstanding.
> 
> What would we wait for? What if we were holding a critical lock in that
> moment? Would it be safe to just block for some completions? What if there's
> actually no IO in progress?

I'd try the following.  First, scan for all IOs of all processes at
AHS_DEFINED and later, advancing them to AHS_COMPLETED_SHARED.  This might be
unsafe today, but discovering why it's unsafe likely will inform design beyond
EAGAIN returns.  I don't specifically know of a way it's unsafe.  Do just one
pass of that; there may be newer IOs in progress afterward.  If submit still
gets EAGAIN, sleep a bit and retry.  Like we do in pgwin32_open_handle(), fail
after a fixed number of iterations.  This isn't great if we hold a critical
lock, but it beats the alternative of PANIC on the first EAGAIN.

> > > +FileStartWriteV(struct PgAioHandle *ioh, File file,
> > > +				int iovcnt, off_t offset,
> > > +				uint32 wait_event_info)
> > > +{
> > > ...
> >
> > FileStartWriteV() gets to state AHS_PREPARED, so let's align with the state
> > name by calling it FilePrepareWriteV (or FileWriteVPrepare or whatever).
> 
> Hm - that doesn't necessarily seem right to me. I don't think the caller
> should assume that the IO will just be prepared and not already completed by
> the time FileStartWriteV() returns - we might actually do the IO
> synchronously.

Yes.  Even if it doesn't become synchronous IO, some other process may advance
the IO to AHS_COMPLETED_SHARED by the next wake-up of the process that defined
the IO.  Still, I think this shouldn't use the term "Start" while no state
name uses that term.  What else could remove that mismatch?

> > Is there any specific decision you'd like to settle before patch 6 exits
> > WIP?
> 
> Patch 6 specifically? That I really mainly kept separate for review - it

No.  I'll rephrase as "Is there any specific decision you'd like to settle
before the next cohort of patches exits WIP?"

> doesn't seem particularly interesting to commit it earlier than 7, or do you
> think differently?

No, I agree a lone commit of 6 isn't a win.  Roughly, the eight patches
6-9,12-15 could be a minimal attractive unit.  I've not thought through that
grouping much.

> In case you mean 6+7 or 6 to ~11, I can think of the following:
> 
> - I am worried about the need for bounce buffers for writes of checksummed
>   buffers. That quickly ends up being a significant chunk of memory,
>   particularly when using a small shared_buffers with a higher than default
>   number of connection. I'm currently hacking up a prototype that'd prevent us
>   from setting hint bits with just a share lock. I'm planning to start a
>   separate thread about that.

AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
Doing better is nice, but I don't consider this a blocker.  I recommend
dealing with the worry by reducing the limit initially (128 blocks?).  Can
always raise it later.

> - The header split doesn't yet quite seem right yet

I won't have a strong opinion on that one.  The aio.c/aio_io.c split did catch
my attention.  I made a note to check it again once those files get header
comments.

> - I'd like to implement retries in the later patches, to make sure that it
>   doesn't have design implications

Yes, that's a blocker to me.

> - Worker mode needs to be able to automatically adjust the number of running
>   workers, I think - otherwise it's going to be too hard to tune.

Changing that later wouldn't affect much else, so I'd not consider it a
blocker.  (The worst case is that we think the initial AIO release will be a
loss for most users, so we wrap it in debug_ terminology like we did for
debug_io_direct.  I'm not saying worker scaling will push AIO from one side of
that line to another, but that's why I'm fine with commits that omit
self-contained optimizations.)

> - I think the PgAioHandles need to be slimmed down a bit - there's some design
>   evolution visible that should not end up in the tree.

Okay.

> - I'm not sure that I like name "subject" for the different things AIO is
>   performed for

How about one of these six terms:

- listener, observer [if you view smgr as an observer of IOs in the sense of https://en.wikipedia.org/wiki/Observer_pattern]
- class, subclass, type, tag [if you view an SmgrIO as a subclass of an IO, in the object-oriented sense]

> - I am wondering if the need for pgaio_io_set_io_data_32() (to store the set
>   of buffer ids that are affected by one IO) could be replaced by repurposing
>   BufferDesc->freeNext or something along those lines. I don't like the amount
>   of memory required for storing those arrays, even if it's not that much
>   compared to needing space to store struct iovec[PG_IOV_MAX] for each AIO
>   handle.

Here too, changing that later wouldn't affect much else, so I'd not consider
it a blocker.

> - I'd like to extend the test module to actually test more cases, it's too
>   hard to reach some paths, particularly without [a lot] of users yet. That's
>   not strictly a dependency of the earlier patches - since the initial patches
>   can't actually do much in the way of IO.

Agreed.  Among the post-patch check-world coverage, which uncovered parts have
the most risk?

> - We shouldn't reserve AioHandles etc for io workers - but because different
>   tpes of aux processes don't use a predetermined ProcNumber, that's not
>   entirely trivial without adding more complexity.  I've actually wondered
>   whether IO workes should be their own "top-level" kind of process, rather
>   than an aux process. But that seems quite costly.

Here too, changing that later wouldn't affect much else, so I'd not consider
it a blocker.  Of these ones I'm calling non-blockers, which would you most
regret deferring?

> - Right now the io_uring mode has each backend's io_uring instance visible to
>   each other process. That ends up using a fair number of FDs. That's OK from
>   an efficiency perspective, but I think we'd need to add code to adjust the
>   soft RLIMIT_NOFILE (it's set to 1024 on most distros because there are
>   various programs that iterate over all possible FDs, causing significant

Agreed on raising the soft limit.  Docs and/or errhint() likely will need to
mention system configuration nonetheless, since some users will encounter
RLIMIT_MEMLOCK or /proc/sys/kernel/io_uring_disabled.

>   slowdowns when the soft limit defaults to something high).  I earlier had a
>   limited number of io_uring instances, but that added a fair amount of
>   overhead because then submitting IO would require a lock.
> 
>   That again doesn't have to be solved as part of the earlier patches but
>   might have some minor design impact.

How far do you see the design impact spreading on that one?

Thanks,
nm

  13. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-09-30T14:49:17Z

    Hi,

On 2024-09-17 11:08:19 -0700, Noah Misch wrote:
> > - I am worried about the need for bounce buffers for writes of checksummed
> >   buffers. That quickly ends up being a significant chunk of memory,
> >   particularly when using a small shared_buffers with a higher than default
> >   number of connection. I'm currently hacking up a prototype that'd prevent us
> >   from setting hint bits with just a share lock. I'm planning to start a
> >   separate thread about that.
> 
> AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
> Doing better is nice, but I don't consider this a blocker.  I recommend
> dealing with the worry by reducing the limit initially (128 blocks?).  Can
> always raise it later.

On storage that has nontrivial latency, like just about all cloud storage,
even 256 will be too low. Particularly for checkpointer.

Assuming 1ms latency - which isn't the high end of cloud storage latency - 256
blocks in flight limits you to <= 256MByte/s, even on storage that can have a
lot more throughput. With 3ms, which isn't uncommon, it's 85MB/s.

Of course this could be addressed by tuning, but it seems like something that
shouldn't need to be tuned by the majority of folks running postgres.


We also discussed the topic at https://postgr.es/m/20240925020022.c5.nmisch%40google.com
> ... neither BM_SETTING_HINTS nor keeping bounce buffers looks like a bad
> decision.  From what I've heard so far of the performance effects, if it were
> me, I would keep the bounce buffers.  I'd pursue BM_SETTING_HINTS and bounce
> buffer removal as a distinct project after the main AIO capability.  Bounce
> buffers have an implementation.  They aren't harming other design decisions.
> The AIO project is big, so I'd want to err on the side of not designating
> other projects as its prerequisites.

Given the issues that modifying pages while in flight causes, not just with PG
level checksums, but also filesystem level checksum, I don't feel like it's a
particularly promising approach.

However, I think this doesn't have to mean that the BM_SETTING_HINTS stuff has
to be completed before we can move forward with AIO. If I split out the write
portion from the read portion a bit further, the main AIO changes and the
shared-buffer read user can be merged before there's a dependency on the hint
bit stuff being done.

Does that seem reasonable?

Greetings,

Andres Freund

  14. Re: AIO v2.0

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2024-09-30T15:55:37Z

    On Mon, 30 Sept 2024 at 16:49, Andres Freund <andres@anarazel.de> wrote:
> On 2024-09-17 11:08:19 -0700, Noah Misch wrote:
> > > - I am worried about the need for bounce buffers for writes of checksummed
> > >   buffers. That quickly ends up being a significant chunk of memory,
> > >   particularly when using a small shared_buffers with a higher than default
> > >   number of connection. I'm currently hacking up a prototype that'd prevent us
> > >   from setting hint bits with just a share lock. I'm planning to start a
> > >   separate thread about that.
> >
> > AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
> > Doing better is nice, but I don't consider this a blocker.  I recommend
> > dealing with the worry by reducing the limit initially (128 blocks?).  Can
> > always raise it later.
>
> On storage that has nontrivial latency, like just about all cloud storage,
> even 256 will be too low. Particularly for checkpointer.
>
> Assuming 1ms latency - which isn't the high end of cloud storage latency - 256
> blocks in flight limits you to <= 256MByte/s, even on storage that can have a
> lot more throughput. With 3ms, which isn't uncommon, it's 85MB/s.

FYI, I think you're off by a factor 8, i.e. that would be 2GB/sec and
666MB/sec respectively, given a normal page size of 8kB and exactly
1ms/3ms full round trip latency:

1 page/1 ms * 8kB/page * 256 concurrency = 256 pages/ms * 8kB/page =
2MiB/ms ~= 2GiB/sec.
for 3ms divide by 3 -> ~666MiB/sec.

Kind regards,

Matthias van de Meent
Neon (https://neon.tech)

  15. Re: AIO v2.0

    Noah Misch <noah@leadboat.com> — 2024-09-30T19:29:53Z

    On Mon, Sep 30, 2024 at 10:49:17AM -0400, Andres Freund wrote:
> We also discussed the topic at https://postgr.es/m/20240925020022.c5.nmisch%40google.com
> > ... neither BM_SETTING_HINTS nor keeping bounce buffers looks like a bad
> > decision.  From what I've heard so far of the performance effects, if it were
> > me, I would keep the bounce buffers.  I'd pursue BM_SETTING_HINTS and bounce
> > buffer removal as a distinct project after the main AIO capability.  Bounce
> > buffers have an implementation.  They aren't harming other design decisions.
> > The AIO project is big, so I'd want to err on the side of not designating
> > other projects as its prerequisites.
> 
> Given the issues that modifying pages while in flight causes, not just with PG
> level checksums, but also filesystem level checksum, I don't feel like it's a
> particularly promising approach.
> 
> However, I think this doesn't have to mean that the BM_SETTING_HINTS stuff has
> to be completed before we can move forward with AIO. If I split out the write
> portion from the read portion a bit further, the main AIO changes and the
> shared-buffer read user can be merged before there's a dependency on the hint
> bit stuff being done.
> 
> Does that seem reasonable?

Yes.

  16. Re: AIO v2.0

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2024-11-18T12:19:58Z

     On Fri, Sep 6, 2024 at 9:38 PM Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> Attached is the next version of the patchset. (..)


Hi Andres,

Thank You for worth admiring persistence on this. Please do not take it as
criticism, just more like set of questions regarding the patchset v2.1 that
I finally got little time to play with:

0. Doesn't the v2.1-0011-aio-Add-io_uring-method.patch -> in
pgaio_uring_submit() -> io_uring_get_sqe() need a return value check ?
Otherwise we'll never know that SQ is full in theory, perhaps at least such
a check should be made with Assert() ? (I understand right now that we
allow just up to io_uring_queue_init(io_max_concurrency), but what happens
if:
a. previous io_uring_submit() failed for some reason and we do not have
free space for SQ?
b. (hypothetical) someday someone will try to make PG multithreaded and the
code starts using just one big queue, still without checking for
io_uring_get_sqe()?

1. In [0] you wrote that there's this high amount of FDs consumed for
io_uring (dangerously close to RLIMIT_NOFILE). I can attest that there are
many customers who are using extremely high max_connections (4k-5k, but
there outliers with 10k in the wild too) - so they won't even start - and I
have one doubt on the user-friendliness impact of this. I'm quite certain
it's going to be the same as with pgbouncer where one is forced to tweak
OS(systemd/pam/limits.conf/etc), but in PG we are better because PG tries
to preallocate and then close() a lot of FDs, so that's safer in runtime.
IMVHO even if we just consume e.g. say > 30% of FDs just for io_uring, the
max_files_per_process looses it's spirit a little bit and PG is going to
start loose efficiency too due to frequent open()/close() calls as fd cache
is too small. Tomas also complained about it some time ago in [1])

So maybe it would be good to introduce couple of sanity checks too (even
after setting higher limit):
- issue FATAL in case of using io_method = io_ring && max_connections would
be close to getrusage(RLIMIT_NOFILE)
- issue warning in case of using io_method = io_ring && we wouldnt have
even real 1k FDs free for handling relation FDs (detect something bad like:
getrusage(RLIMIT_NOFILE) <= max_connections + max_files_per_process)

2. In pgaio_uring_postmaster_child_init_local() there
"io_uring_queue_init(32,...)" - why 32? :) And also there's separate
io_uring_queue_init(io_max_concurrency) which seems to be derived from
AioChooseMaxConccurrency() which can go up to 64?

3. I find having two GUCs named literally the same
(effective_io_concurrency, io_max_concurrency). It is clear from IO_URING
perspective what is io_max_concurrency all about, but I bet having also
effective_io_concurrency in the mix is going to be a little confusing for
users (well, it is to me). Maybe that README.md could elaborate a little
bit on the relation between those two? Or maybe do you plan to remove
io_max_concurrency and bind it to effective_io_concurrency in future? To
add more fun , there's MAX_IO_CONCURRENCY nearby in v2.1-0014 too while the
earlier mentioned AioChooseMaxConccurrency() goes up to just 64

4. While we are at this, shouldn't the patch rename debug_io_direct to
simply io_direct so that GUCs are consistent in terms of naming?

5. It appears that pg_stat_io.reads seems to be not refreshed until they
query seems to be finished. While running a query for minutes with this
patchset, I've got:
              now              |  reads   | read_time
-------------------------------+----------+-----------
 2024-11-15 12:09:09.151631+00 | 15004271 |         0
[..]
 2024-11-15 12:10:25.241175+00 | 15004271 |         0
 2024-11-15 12:10:26.241179+00 | 15004271 |         0
 2024-11-15 12:10:27.241139+00 | 18250913 |         0

Or is that how it is supposed to work? Also pg_stat_io.read_time would be
something vague with io_uring/worker, so maybe zero is good here (?).
Otherwise we would have to measure time spent on waiting alone, but that
would force more instructions for calculating io times...

6. After playing with some basic measurements - which went fine, I wanted
to go test simple PostGIS even with sequential scans to see any
compatibility issues (AFAIR Thomas Munro on PGConfEU indicated as good
testing point), but before that I've tried to see what's the TOAST
performance alone with AIO+DIO (debug_io_direct=data). One issue I have
found is that DIO seems to be unusable until somebody will teach TOAST to
use readstreams, is that correct? Maybe I'm doing something wrong, but I
haven't seen any TOAST <-> readstreams topic:

-- 12MB table , 25GB toast
create table t (id bigint, t text storage external);
insert into t select i::bigint as id, repeat(md5(i::text),4000)::text as r
from generate_series(1,200000) s(i);
set max_parallel_workers_per_gather=0;
\timing
-- with cold caches: empty s_b, echo 3 > drop_caches
select sum(length(t)) from t;
  master    101897.823 ms (01:41.898)
  AIO          99758.399 ms (01:39.758)
  AIO+DIO 191479.079 ms (03:11.479)

hotpath was detoast_attr() -> toast_fetch_datum() ->
heap_fetch_toast_slice() -> systable_getnext_ordered() ->
index_getnext_slot() -> index_fetch_heap() -> heapam_index_fetch_tuple() ->
ReadBufferExtended -> AIO code.

The difference is that on cold caches with DIO gets 2x slowdown; with clean
s_b and so on:
* getting normal heap data seqscan: up to 285MB/s
* but TOASTs maxes out at 132MB/s when using io_uring+DIO

Not about patch itself, but questions about related stack functionality:
----------------------------------------------------------------------------------------------------


7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
on how to inspect real I/O calls requested to review if the code is issuing
sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
perhaps using some bpftrace / xfsslower is the best way to go ?

8. Not sure if that helps, but I've managed the somehow to hit the
impossible situation You describe in pgaio_uring_submit() "(ret !=
num_staged_ios)", but I had to push urings really hard into using futexes
and probably I've could made some error in coding too for that too occur
[3]. As it stands in that patch from my thread, it was not covered: /*
FIXME: fix ret != submitted ?! seems like bug?! */ (but i had that hit that
code-path pretty often with 6.10.x kernel)

9. Please let me know, what's the current up to date line of thinking about
this patchset: is it intended to be committed as v18 ? As a debug feature
or as non-debug feature? (that is which of the IO methods should be
scrutinized the most as it is going to be the new default - sync or worker?)

10. At this point, does it even make sense to give a try experimenty try to
pwritev2() with RWF_ATOMIC? (that thing is already in the open, but XFS is
going to cover it with 6.12.x apparently, but I could try with some -rcX)

-J.

p.s. I hope I did not ask stupid questions nor missed anything.

[0] -
https://www.postgresql.org/message-id/237y5rabqim2c2v37js53li6i34v2525y2baf32isyexecn4ic%40bqmlx5mrnwuf
- "Right now the io_uring mode has each backend's io_uring instance visible
to
  each other process.(..)"

[1] -
https://www.postgresql.org/message-id/flat/510b887e-c0ce-4a0c-a17a-2c6abb8d9a5c%40enterprisedb.com
- sentence after: "FWIW there's another bottleneck people may not realize
(..)"

[2] -
https://www.postgresql.org/message-id/flat/x3f32prdpgalmiieyialqtn53j5uvb2e4c47nvnjetkipq3zyk%40xk7jy7fnua6w#dbedc74f7d19abf40b90f2c348fe1778

[3] -
https://www.postgresql.org/message-id/CAKZiRmwrBjCbCJ433wV5zjvwt_OuY7BsVX12MBKiBu%2BeNZDm6g%40mail.gmail.com

  17. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-12-19T22:29:12Z

    Hi,

Sorry for loosing track of your message for this long, I saw it just now
because I was working on posting a new version.


On 2024-11-18 13:19:58 +0100, Jakub Wartak wrote:
>  On Fri, Sep 6, 2024 at 9:38 PM Andres Freund <andres@anarazel.de> wrote:
> Thank You for worth admiring persistence on this. Please do not take it as
> criticism, just more like set of questions regarding the patchset v2.1 that
> I finally got little time to play with:
>
> 0. Doesn't the v2.1-0011-aio-Add-io_uring-method.patch -> in
> pgaio_uring_submit() -> io_uring_get_sqe() need a return value check ?

Yea, it shouldn't ever happen, but it's worth adding a check.


> Otherwise we'll never know that SQ is full in theory, perhaps at least such
> a check should be made with Assert() ? (I understand right now that we
> allow just up to io_uring_queue_init(io_max_concurrency), but what happens
> if:
> a. previous io_uring_submit() failed for some reason and we do not have
> free space for SQ?

We'd have PANICed at that failure :)


> b. (hypothetical) someday someone will try to make PG multithreaded and the
> code starts using just one big queue, still without checking for
> io_uring_get_sqe()?

That'd not make sense - you'd still want to use separate rings, to avoid
contention.


> 1. In [0] you wrote that there's this high amount of FDs consumed for
> io_uring (dangerously close to RLIMIT_NOFILE). I can attest that there are
> many customers who are using extremely high max_connections (4k-5k, but
> there outliers with 10k in the wild too) - so they won't even start - and I
> have one doubt on the user-friendliness impact of this. I'm quite certain
> it's going to be the same as with pgbouncer where one is forced to tweak
> OS(systemd/pam/limits.conf/etc), but in PG we are better because PG tries
> to preallocate and then close() a lot of FDs, so that's safer in runtime.
> IMVHO even if we just consume e.g. say > 30% of FDs just for io_uring, the
> max_files_per_process looses it's spirit a little bit and PG is going to
> start loose efficiency too due to frequent open()/close() calls as fd cache
> is too small. Tomas also complained about it some time ago in [1])

My current thoughts around this are that we should generally, independent of
io_uring, increase the FD limit ourselves.

In most distros the soft ulimit is set to something like 1024, but the hard
limit is much higher.  The reason for that is that some applications try to
close all fds between 0 and RLIMIT_NOFILE - which takes a long time if
RLIMIT_NOFILE is high. By setting only the soft limit to a low value any
application needing higher limits can just opt into using more FDs.

On several of my machines the hard limit is 1073741816.



> So maybe it would be good to introduce couple of sanity checks too (even
> after setting higher limit):
> - issue FATAL in case of using io_method = io_ring && max_connections would
> be close to getrusage(RLIMIT_NOFILE)
> - issue warning in case of using io_method = io_ring && we wouldnt have
> even real 1k FDs free for handling relation FDs (detect something bad like:
> getrusage(RLIMIT_NOFILE) <= max_connections + max_files_per_process)

Probably still worth adding something like this, even if we were to do what I
am suggesting above.


> 2. In pgaio_uring_postmaster_child_init_local() there
> "io_uring_queue_init(32,...)" - why 32? :) And also there's separate
> io_uring_queue_init(io_max_concurrency) which seems to be derived from
> AioChooseMaxConccurrency() which can go up to 64?

Yea, that's probably not right.


> 3. I find having two GUCs named literally the same
> (effective_io_concurrency, io_max_concurrency). It is clear from IO_URING
> perspective what is io_max_concurrency all about, but I bet having also
> effective_io_concurrency in the mix is going to be a little confusing for
> users (well, it is to me). Maybe that README.md could elaborate a little
> bit on the relation between those two? Or maybe do you plan to remove
> io_max_concurrency and bind it to effective_io_concurrency in future?

io_max_concurrency is a hard maximum that needs to be set at server start,
because it requires allocating shared memory. Whereas effective_io_concurrency
can be changed on a per-session and per-tablespace
basis. I.e. io_max_concurrency is a hard upper limit for an entire backend,
whereas effective_io_concurrency controls how much one scan (or whatever does
prefetching) can issue.



>  To add more fun , there's MAX_IO_CONCURRENCY nearby in v2.1-0014 too while
> the earlier mentioned AioChooseMaxConccurrency() goes up to just 64

Yea, that should probably be disambiguated.


> 4. While we are at this, shouldn't the patch rename debug_io_direct to
> simply io_direct so that GUCs are consistent in terms of naming?

I used to have a patch like that in the series and it was a pain to
rebase...

I also suspect sure this is quite enough to make debug_io_direct quite
production ready, even if just considering io_direct=data. Without streaming
read use in heap + index VACUUM, RelationCopyStorage() and a few other places
the performance consequences of using direct IO can be, um, surprising.



> 5. It appears that pg_stat_io.reads seems to be not refreshed until they
> query seems to be finished. While running a query for minutes with this
> patchset, I've got:
>               now              |  reads   | read_time
> -------------------------------+----------+-----------
>  2024-11-15 12:09:09.151631+00 | 15004271 |         0
> [..]
>  2024-11-15 12:10:25.241175+00 | 15004271 |         0
>  2024-11-15 12:10:26.241179+00 | 15004271 |         0
>  2024-11-15 12:10:27.241139+00 | 18250913 |         0
>
> Or is that how it is supposed to work?

Currently the patch has a FIXME to add some IO statistics (I think I raised
that somewhere in this thread, too). It's not clear to me what IO time ought
to mean. I suspect the least bad answer is what you suggest:

> Also pg_stat_io.read_time would be something vague with io_uring/worker, so
> maybe zero is good here (?).  Otherwise we would have to measure time spent
> on waiting alone, but that would force more instructions for calculating io
> times...

I.e. we should track the amount of time spent waiting for IOs.

I don't think tracking time in worker or such would make much sense, that'd
often end up with reporting more IO time than a query took.


> 6. After playing with some basic measurements - which went fine, I wanted
> to go test simple PostGIS even with sequential scans to see any
> compatibility issues (AFAIR Thomas Munro on PGConfEU indicated as good
> testing point), but before that I've tried to see what's the TOAST
> performance alone with AIO+DIO (debug_io_direct=data).

It's worth noting that with the last posted version you needed to increase
effective_io_concurrency to something very high to see sensible
performance.

That's due to the way read_stream_begin_impl() limited the number of buffers
pinned to effective_io_concurrency * 4 - which, due to io_combine_limit, ends
up allowing only a single IO in flight in case of sequential blocks until
effective_io_concurrency is set to 8 or such.  I've adjusted that to some
degree now, but I think that might need a bit more sophistication.



> One issue I have found is that DIO seems to be unusable until somebody will
> teach TOAST to use readstreams, is that correct? Maybe I'm doing something
> wrong, but I haven't seen any TOAST <-> readstreams topic:

Hm, I suspect that aq read stream won't help a whole lot in manyq toast
cases. Unless you have particularly long toast datums, the time is going to be
dominated by the random accesses, as each toast datum is looked up in a
non-predictable way.

Generally, using DIO requires tuning shared buffers much more aggressively
than not using DIO, no amount of stream use will change that. Of course we
shoul try to reduce that "downside"...


I'm not sure if the best way to do prefetching toast chunks would be to rely
on more generalized index->table prefetching support, or to have dedicated code.


> -- 12MB table , 25GB toast
> create table t (id bigint, t text storage external);
> insert into t select i::bigint as id, repeat(md5(i::text),4000)::text as r
> from generate_series(1,200000) s(i);
> set max_parallel_workers_per_gather=0;
> \timing
> -- with cold caches: empty s_b, echo 3 > drop_caches
> select sum(length(t)) from t;
>   master    101897.823 ms (01:41.898)
>   AIO          99758.399 ms (01:39.758)
>   AIO+DIO 191479.079 ms (03:11.479)
>
> hotpath was detoast_attr() -> toast_fetch_datum() ->
> heap_fetch_toast_slice() -> systable_getnext_ordered() ->
> index_getnext_slot() -> index_fetch_heap() -> heapam_index_fetch_tuple() ->
> ReadBufferExtended -> AIO code.
>
> The difference is that on cold caches with DIO gets 2x slowdown; with clean
> s_b and so on:
> * getting normal heap data seqscan: up to 285MB/s
> * but TOASTs maxes out at 132MB/s when using io_uring+DIO

I started loading the data to try this out myself :).



> Not about patch itself, but questions about related stack functionality:
> ----------------------------------------------------------------------------------------------------
>
>
> 7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
> on how to inspect real I/O calls requested to review if the code is issuing
> sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
> perhaps using some bpftrace / xfsslower is the best way to go ?

I think we still want something like it, but I don't think it needs to be in
the initial commits.

There are kernel events that you can track using e.g. perf. Particularly
useful are
io_uring:io_uring_submit_req
io_uring:io_uring_complete



> 8. Not sure if that helps, but I've managed the somehow to hit the
> impossible situation You describe in pgaio_uring_submit() "(ret !=
> num_staged_ios)", but I had to push urings really hard into using futexes
> and probably I've could made some error in coding too for that too occur
> [3]. As it stands in that patch from my thread, it was not covered: /*
> FIXME: fix ret != submitted ?! seems like bug?! */ (but i had that hit that
> code-path pretty often with 6.10.x kernel)

I think you can hit that if you don't take care to limit the number of IOs
being submitted at once or if you're not consuming completions. If the
completion queue is full enough the kernel at some point won't allow more IOs
to be submitted.


> 9. Please let me know, what's the current up to date line of thinking about
> this patchset: is it intended to be committed as v18 ?

I'd love to get some of it into 18.  I don't quite know whether we can make it
happen and to what extent.


> As a debug feature or as non-debug feature? (that is which of the IO methods
> should be scrutinized the most as it is going to be the new default - sync
> or worker?)

I'd say initially worker, with a beta 1 or 2 checklist item to revise it.



> 10. At this point, does it even make sense to give a try experimenty try to
> pwritev2() with RWF_ATOMIC? (that thing is already in the open, but XFS is
> going to cover it with 6.12.x apparently, but I could try with some -rcX)

I don't think that's worth doing right now. There's too many dependencies and
it's going to be a while till the kernel support for that is widespread enough
to matter.

There's also the issue that, to my knowledge, outside of cloud environments
there's pretty much no hardware that actually reports power-fail atomicity
sizes bigger than a sector.


> p.s. I hope I did not ask stupid questions nor missed anything.

You did not!

Greetings,

Andres Freund

  18. Re: AIO v2.0

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-12-19T22:34:29Z

    Andres Freund <andres@anarazel.de> writes:
> My current thoughts around this are that we should generally, independent of
> io_uring, increase the FD limit ourselves.

I'm seriously down on that, because it amounts to an assumption that
we own the machine and can appropriate all its resources.  If ENFILE
weren't a thing, it'd be all right, but that is a thing.  We have no
business trying to consume resources the DBA didn't tell us we could.

			regards, tom lane

  19. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-12-20T00:54:10Z

    Hi,

On 2024-12-19 17:34:29 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > My current thoughts around this are that we should generally, independent of
> > io_uring, increase the FD limit ourselves.
>
> I'm seriously down on that, because it amounts to an assumption that
> we own the machine and can appropriate all its resources.  If ENFILE
> weren't a thing, it'd be all right, but that is a thing.  We have no
> business trying to consume resources the DBA didn't tell us we could.

Arguably the configuration *did* tell us, by having a higher hard limit...

I'm not saying that we should increase the limit without a bound or without a
configuration option, btw.

As I had mentioned, the problem with relying on increasing the soft limit that
is that it's not generally sensible to do so, because it causes a bunch of
binaries to do be weirdly slow.

Another reason to not increase the soft rlimit is that doing so can break
programs relying on select().

But opting into a higher rlimit, while obviously adhering to the hard limit
and perhaps some other config knob, seems fine?

Greetings,

Andres Freund

  20. Re: AIO v2.0

    Jelte Fennema-Nio <postgres@jeltef.nl> — 2024-12-20T17:27:13Z

    On Fri, 20 Dec 2024 at 01:54, Andres Freund <andres@anarazel.de> wrote:
> Arguably the configuration *did* tell us, by having a higher hard limit...
> <snip>
> But opting into a higher rlimit, while obviously adhering to the hard limit
> and perhaps some other config knob, seems fine?

Yes, totally fine. That's exactly the reasoning why the hard limit is
so much larger than the soft limit by default on systems with systemd:

https://0pointer.net/blog/file-descriptor-limits.html

  21. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2024-12-20T17:39:33Z

    Hi,

On 2024-12-20 18:27:13 +0100, Jelte Fennema-Nio wrote:
> On Fri, 20 Dec 2024 at 01:54, Andres Freund <andres@anarazel.de> wrote:
> > Arguably the configuration *did* tell us, by having a higher hard limit...
> > <snip>
> > But opting into a higher rlimit, while obviously adhering to the hard limit
> > and perhaps some other config knob, seems fine?
> 
> Yes, totally fine. That's exactly the reasoning why the hard limit is
> so much larger than the soft limit by default on systems with systemd:
> 
> https://0pointer.net/blog/file-descriptor-limits.html

Good link.

This isn't just relevant for using io_uring:

There obviously are several people working on threaded postgres. Even if we
didn't duplicate fd.c file descriptors between threads (we probably will, at
least initially), the client connection FDs alone will mean that we have a lot
more FDs open. Due to the select() issue the soft limit won't be increased
beyond 1024, requiring everyone to add a 'ulimit -n $somehighnumber' before
starting postgres on linux doesn't help anyone.

Greetings,

Andres Freund

  22. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-01T04:03:33Z

    Hi,

Attached is a new version of the AIO patchset.


The biggest changes are:

- The README has been extended with an overview of the API. I think it gives a
  good overview of how the API fits together.  I'd be very good to get
  feedback from folks that aren't as familiar with AIO, I can't really see
  what's easy/hard anymore.


- The read/write patches and the bounce buffer patches are split out, so that
  there's no dependency between the first few AIO patches and the "don't dirty
  while IO is going on" patcheset [1].


- Retries for partial IOs (i.e. short reads) are now implemented. Turned out
  to take all of three lines and adding one missing variable initialization.


- I added quite a lot of function-header and file-header comments. There's
  more to be done here, but see also the TODO section below.


- IO stats are now tracked. Specifically, the "time" for an IO is now the time
  spent waiting for an IO, as discussed around [2].  I haven't updated the
  docs yet.


- There now is a fastpath for executing AIO "synchronously", i.e. preparing an
  IO and immediately submitting it.


- Previously one needed very large effective_io_concurrency values to get
  sufficient asynchronous IO for sequential scans, as read_stream.c limited
  max_pinned_buffers to effective_io_concurrency * 4. Unless
  effective_io_concurrency was very high, that'd only allow a single IO to be
  in-flight, due to io_combine_limit buffers getting merged into one IO.

  Instead the pin limit is now capped by effective_io_concurrency *
  io_combine_limit.

  Right now that's part of one larger "hack up read_stream.c" commit, Thomas
  said he'd take a look at how to do this properly.  This is probably
  something we could and should commit separately.


- io_method = sync has been made more similar to the way IO happens today. In
  particular, we now continue to issue prefetch requests and the actual IO is
  done only within WaitReadBuffers().


- When using buffered IO with io_uring, there previously was a small
  regression, due to more IO happening in the process context with io_uring
  (instead of in a kernel thread). While one could argue that it's better to
  not increase CPU usage beyond one process, I don't find that sufficiently
  convincing. To work around that I added a heuritic that tells IO uring to
  execute IOs using it's worker infrastructure. That seems to have fixed this
  problem entirely.


- IO worker infrastructure was cleaned up


- I pushed a few minor preliminary commits a while ago


- lots of other smaller stuff



The biggest TODOs are:

- Right now the API between bufmgr.c and read_stream.c kind of necessitates
  that one StartReadBuffers() call actually can trigger multiple IOs, if
  one of the buffers was read in by another backend, before "this" backend
  called StartBufferIO().

  I think Thomas and I figured out a way to evolve the interface so that this
  isn't necessary anymore:

  We allow StartReadBuffers() to memorize buffers it pinned but didn't
  initiate IO on in the buffers[] argument. The next call to StartReadBuffers
  then doesn't have to repin thse buffers. That doesn't just solve the
  multiple-IOs for one "read operation" issue, it also make the - very common
  - case of a bunch of "buffer misses" followed by a "buffer hit" cleaner, the
  hit wouldn't be tracked in the same ReadBuffersOperation anymore.


- Right now bufmgr.h includes aio.h, because it needs to include a reference
  to the AIO's result in ReadBuffersOperation. Requiring a dynamic allocation
  would be noticeable overhead, so that's not an option. I think the best
  option here would be to introduce something like aio_types.h, so fewer
  things are included.


- There's no obvious way to tell "internal" function operating on an IO handle
  apart from functions that are expected to be called by the issuer of an IO.

  One way to deal with this would be to introduce a distinct "issuer IO
  reference" type.  I think that might be a good idea, it would also make it
  clearer that a good number of the functions can only be called by the
  issuer, before the IO is submitted.

  This would also make it easier to order functions more sensibly in aio.c, as
  all the issuer functions would be together.

  The functions on AIO handles that everyone can call already have a distinct
  type (PgAioHandleRef vs PgAioHandle*).


- While I've added a lot of comments, I only got so far adding them. More are
  needed.


- The naming around PgAioReturn, PgAioResult, PgAioResultStatus needs to be
  improved


- The debug logging functions are a bit of a mess, lots of very similar code
  in lots of places. I think AIO needs a few ereport() wrappers to make this
  easier.


- More tests are needed. None of our current test frameworks really makes this
  easy :(.


- Several folks asked for pg_stat_aio to come back, in "v1" that showed the
  set of currently in-flight AIOs.  That's not particularly hard - except
  that it doesn't really fit in the pg_stat_* namespace.


- I'm not sure that effective_io_concurrency as we have it right now really
  makes sense, particularly not with the current default values. But that's a
  mostly independent change.


Greetings,

Andres Freund

[1] https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf%40gcnactj4z56m
[2] https://postgr.es/m/tp63m6tcbi7mmsjlqgxd55sghhwvjxp3mkgeljffkbaujezvdl%40fvmdr3c6uhat

  23. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2025-01-06T16:28:39Z

    Hi,

On 2024-12-19 17:29:12 -0500, Andres Freund wrote:
> > Not about patch itself, but questions about related stack functionality:
> > ----------------------------------------------------------------------------------------------------
> >
> >
> > 7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
> > on how to inspect real I/O calls requested to review if the code is issuing
> > sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
> > perhaps using some bpftrace / xfsslower is the best way to go ?
> 
> I think we still want something like it, but I don't think it needs to be in
> the initial commits.

After I got this question from Thomas as well, I started hacking one up.

What information would you like to see?

Here's what I currently have:
┌─[ RECORD 1 ]───┬────────────────────────────────────────────────┐
│ pid            │ 358212                                         │
│ io_id          │ 2050                                           │
│ io_generation  │ 4209                                           │
│ state          │ COMPLETED_SHARED                               │
│ operation      │ read                                           │
│ offset         │ 509083648                                      │
│ length         │ 262144                                         │
│ subject        │ smgr                                           │
│ iovec_data_len │ 32                                             │
│ raw_result     │ 262144                                         │
│ result         │ OK                                             │
│ error_desc     │ (null)                                         │
│ subject_desc   │ blocks 1372864..1372895 in file "base/5/16388" │
│ flag_sync      │ f                                              │
│ flag_localmem  │ f                                              │
│ flag_buffered  │ t                                              │
├─[ RECORD 2 ]───┼────────────────────────────────────────────────┤
│ pid            │ 358212                                         │
│ io_id          │ 2051                                           │
│ io_generation  │ 4199                                           │
│ state          │ IN_FLIGHT                                      │
│ operation      │ read                                           │
│ offset         │ 511967232                                      │
│ length         │ 262144                                         │
│ subject        │ smgr                                           │
│ iovec_data_len │ 32                                             │
│ raw_result     │ (null)                                         │
│ result         │ UNKNOWN                                        │
│ error_desc     │ (null)                                         │
│ subject_desc   │ blocks 1373216..1373247 in file "base/5/16388" │
│ flag_sync      │ f                                              │
│ flag_localmem  │ f                                              │
│ flag_buffered  │ t                                              │


I didn't think that pg_stat_* was quite the right namespace, given that it
shows not stats, but the currently ongoing IOs.  I am going with pg_aios for
now, but I don't particularly like that.


I think we'll want a pg_stat_aio as well, tracking things like:

- how often the queue to IO workes was full
- how many times we submitted IO to the kernel (<= #ios with io_uring)
- how many times we asked the kernel for events (<= #ios with io_uring)
- how many times we had to wait for in-flight IOs before issuing more IOs

Greetings,

Andres Freund

  24. Re: AIO v2.2

    Noah Misch <noah@leadboat.com> — 2025-01-06T18:52:20Z

    Patches 1 and 2 are still Ready for Committer.

On Tue, Dec 31, 2024 at 11:03:33PM -0500, Andres Freund wrote:
> - The README has been extended with an overview of the API. I think it gives a
>   good overview of how the API fits together.  I'd be very good to get
>   feedback from folks that aren't as familiar with AIO, I can't really see
>   what's easy/hard anymore.

That's a helpful addition.  I've left inline comments on it, below.

> The biggest TODOs are:
> 
> - Right now the API between bufmgr.c and read_stream.c kind of necessitates
>   that one StartReadBuffers() call actually can trigger multiple IOs, if
>   one of the buffers was read in by another backend, before "this" backend
>   called StartBufferIO().
> 
>   I think Thomas and I figured out a way to evolve the interface so that this
>   isn't necessary anymore:
> 
>   We allow StartReadBuffers() to memorize buffers it pinned but didn't
>   initiate IO on in the buffers[] argument. The next call to StartReadBuffers
>   then doesn't have to repin thse buffers. That doesn't just solve the
>   multiple-IOs for one "read operation" issue, it also make the - very common
>   - case of a bunch of "buffer misses" followed by a "buffer hit" cleaner, the
>   hit wouldn't be tracked in the same ReadBuffersOperation anymore.

That sounds reasonable.

> - Right now bufmgr.h includes aio.h, because it needs to include a reference
>   to the AIO's result in ReadBuffersOperation. Requiring a dynamic allocation
>   would be noticeable overhead, so that's not an option. I think the best
>   option here would be to introduce something like aio_types.h, so fewer
>   things are included.

That sounds fine.  Header splits aren't going to be perfect, so I'd pick
something (e.g. your proposal here) and move on.

> - There's no obvious way to tell "internal" function operating on an IO handle
>   apart from functions that are expected to be called by the issuer of an IO.
> 
>   One way to deal with this would be to introduce a distinct "issuer IO
>   reference" type.  I think that might be a good idea, it would also make it
>   clearer that a good number of the functions can only be called by the
>   issuer, before the IO is submitted.

That's reasonable, albeit non-critical.

> - The naming around PgAioReturn, PgAioResult, PgAioResultStatus needs to be
>   improved

POSIX uses the word "result" for the consequences of a function (e.g. the
result of unlink() is readdir() no longer finding the link).  It uses the word
"return" for a memory value that describes a result.  In that usage, the
struct currently called PgAioResult would be a Return.  The struct currently
called PgAioReturn is PgAioResult plus the data to identify the IO.  Possible
name changes:

  PgAioResult -> PgAioReturn
  PgAioReturn -> PgAioReturnIdentified | PgAioReturnID | PgAioReturnTagged [I don't love these]
  PgAioResultStatus -> PgAioStatus | PgAioFill

That said, I don't dislike the existing names and would not have raised the
topic myself.

> - The debug logging functions are a bit of a mess, lots of very similar code
>   in lots of places. I think AIO needs a few ereport() wrappers to make this
>   easier.

May as well.

> - More tests are needed. None of our current test frameworks really makes this
>   easy :(.

Which testing gap do you find most concerning?  I'd be most interested in the
cases that would be undetected deadlocks under a naive design.  An example
appeared at the end of postgr.es/m/20240916144349.74.nmisch@google.com

> - Several folks asked for pg_stat_aio to come back, in "v1" that showed the
>   set of currently in-flight AIOs.  That's not particularly hard - except
>   that it doesn't really fit in the pg_stat_* namespace.

Later message
postgr.es/m/6vjl6jeaqvyhfbpgwziypwmhem2rwla4o5pgpuxwtg3o3o3jb5@evyzorb5meth is
considering the name pg_aios.  Works for me.

> --- a/src/backend/storage/aio/aio.c
> +++ b/src/backend/storage/aio/aio.c
> @@ -3,6 +3,28 @@
>   * aio.c
>   *    AIO - Core Logic
>   *
> + * For documentation about how AIO works on a higher level, including a
> + * schematic example, see README.md.
> + *
> + *
> + * AIO is a complicated subsystem. To keep things navigable it is split across
> + * a number of files:
> + *
> + * - aio.c - core AIO state handling
> + *
> + * - aio_init.c - initialization
> + *
> + * - aio_io.c - dealing with actual IO, including executing IOs synchronously
> + *
> + * - aio_subject.c - functionality related to executing IO for different
> + *   subjects
> + *
> + * - method_*.c - different ways of executing AIO
> + *
> + * - read_stream.c - helper for accessing buffered relation data with
> + *	 look-ahead
> + *

I felt like some list entries in this new header comment largely restated the
file name.  Here's how I'd write them to avoid that:

 * - method_*.c - different ways of executing AIO (e.g. worker process)
 * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
 * - aio_subject.c - callbacks at IO operation lifecycle events
 * - aio_init.c - per-fork and per-startup-process initialization
 * - aio.c - all other topics
 * - read_stream.c - helper for reading buffered relation data

> --- /dev/null
> +++ b/src/backend/storage/aio/README.md
> @@ -0,0 +1,413 @@
> +# Asynchronous & Direct IO

I would move "### Why Asynchronous IO" to here; that's good background before
getting into the example.  I might also move "### Why Direct / unbuffered IO"
to here.  For me as a reader, I'd benefit from seeing things in this order:

- "why"
- condensed usage example like manpage SYNOPSIS, comments and decls removed
- PgAioHandleState and discussion of valid transitions
- usage example as it is, with full comments
- the rest

In other words, like this:

# Asynchronous & Direct IO

## Motivation

### Why Asynchronous IO

[existing content moved from lower in the file]

## Synopsis

ioh = pgaio_io_get(CurrentResourceOwner, &ioret);
pgaio_io_get_ref(ioh, &ior);
pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
pgaio_io_set_io_data_32(ioh, (uint32 *) buffer, 1);
smgrstartreadv(ioh, operation->smgr, forknum, blkno,
               BufferGetBlock(buffer), 1);
pgaio_submit_staged();
pgaio_io_ref_wait(&ior);
if (ioret.result.status == ARS_ERROR)
    pgaio_result_log(aio_ret.result, &aio_ret.subject_data, ERROR);

## I/O Operation States & Transitions

[PgAioHandleState and its transitions]

## AIO Usage Example

[your content:]

> +
> +## AIO Usage Example
> +
> +In many cases code that can benefit from AIO does not directly have to
> +interact with the AIO interface, but can use AIO via higher-level
> +abstractions. See [Helpers](#helpers).
> +
> +In this example, a buffer will be read into shared buffers.
> +
> +```C
> +/*
> + * Result of the operation, only to be accessed in this backend.
> + */
> +PgAioReturn ioret;
> +
> +/*
> + * Acquire AIO Handle, ioret will get result upon completion.

Consider adding: from here to pgaio_submit_staged(), don't do [description of
the kind of unacceptable blocking operations].

> + * Once the IO handle has been handed of, it may not further be used, as the

s/of/off/

> +### IO can be started in critical sections
...
> +The need to be able to execute IO in critical sections has substantial design
> +implication on the AIO subsystem. Mainly because completing IOs (see prior
> +section) needs to be possible within a critical section, even if the
> +to-be-completed IO itself was not issued in a critical section. Consider
> +e.g. the case of a backend first starting a number of writes from shared
> +buffers and then starting to flush the WAL. Because only a limited amount of
> +IO can be in-progress at the same time, initiating the IO for flushing the WAL
> +may require to first finish executing IO executed earlier.

The last line's two appearances of the word "execute" read awkwardly to me,
and it's an opportunity to use PgAioHandleState terms.  Consider writing the
last line like "may first advance an existing IO from AHS_PREPARED to
AHS_COMPLETED_SHARED".

> +ASLR. This means that the shared memory cannot contain pointer to callbacks.

s/pointer/pointers/

> +### AIO Callbacks
...
> +In addition to completion, AIO callbacks also are called to "prepare" an
> +IO. This is, e.g., used to acquire buffer pins owned by the AIO subsystem for
> +IO to/from shared buffers, which is required to handle the case where the
> +issuing backend errors out and releases its own pins.

Reading this, it's not obvious to me how to reconcile "finishing an IO could
require pin acquisition" with "finishing an IO could happen in a critical
section".  Pinning a buffer in a critical section sounds bad.  I vaguely
recall understanding how it was okay as of my September review, but I've
already forgotten.  Can this text have a sentence making that explicit?

> +### AIO Subjects
> +
> +In addition to the completion callbacks describe above, each AIO Handle has
> +exactly one "subject". Each subject has some space inside an AIO Handle with
> +information specific to the subject and can provide callbacks to allow to
> +reopen the underlying file (required for worker mode) and to describe the IO
> +operation (used for debug logging and error messages).

Can this say roughly how to decide when to add a new subject?  Failing that,
can it give examples of what additional subjects might exist if certain
existing subsystems were to start using AIO?

> +### AIO Results
> +
> +As AIO completion callbacks
> +[are executed in critical sections](#io-can-be-started-in-critical-sections)
> +and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
> +completion callbacks cannot be used to, e.g., make the query that triggered an
> +IO ERROR out.
> +
> +To allow to react to failing IOs the issuing backend can pass a pointer to a
> +`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
> +`PgAioReturn` is filled with information about the IO. This includes
> +information about whether the IO was successful (as a value of
> +`PgAioResultStatus`) and enough information to raise an error in case of a
> +failure (via `pgaio_result_log()`, with the error details encoded in
> +`PgAioResult`).

Can this have a sentence on how this fits in bounded shmem, given the lack of
guarantees about a backend's responsiveness?  In other words, what makes it
okay to have requests take arbitrarily long to move from AHS_COMPLETED_SHARED
to AHS_COMPLETED_LOCAL?

Thanks,
nm

  25. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-06T21:40:26Z

    Hi,

On 2025-01-06 10:52:20 -0800, Noah Misch wrote:
> Patches 1 and 2 are still Ready for Committer.

I feel somewhat weird about pushing 0002 without a user, but I guess it's
still exercised, so it's probably fine...


> On Tue, Dec 31, 2024 at 11:03:33PM -0500, Andres Freund wrote:
> > - The README has been extended with an overview of the API. I think it gives a
> >   good overview of how the API fits together.  I'd be very good to get
> >   feedback from folks that aren't as familiar with AIO, I can't really see
> >   what's easy/hard anymore.
>
> That's a helpful addition.  I've left inline comments on it, below.

Cool!


> > - More tests are needed. None of our current test frameworks really makes this
> >   easy :(.
>
> Which testing gap do you find most concerning?

Most of it isn't even AIO specific...


- temporary tables are rather poorly tested in general:
  - e.g. trivial to exceed the number of buffers, but our tests don't reach that

- We have pretty no testing for IO errors. We have a bit of coverage due to
  src/bin/pg_amcheck/t/003_check.pl, but that's for errors originating in
  bufmgr.c itself.

- no real testing of StartBufferIO's etc wait paths

- no testing for BM_PIN_COUNT_WAITER


I e.g. just noticed that the error handling for AIO on temp tables was broken
- but our tests never reach that:

  The bug exists due to temp tables not differentiating between "backend" pins
  and a "global pincount" - which means that there's no real way for the AIO
  subsystem to have a reference separate from the backend local pin -
  CheckForLocalBufferLeaks() complains about any leftover pins. It seems to
  works in non-assert mode, but with assertions transaction abort asserts out.


> I'd be most interested in the
> cases that would be undetected deadlocks under a naive design.  An example
> appeared at the end of postgr.es/m/20240916144349.74.nmisch@google.com

That's a good one, yea.

I think I'll try to translate the regression tests I wrote into an isolation
test, I hope that'll make it a bit easier to cover more cases.

And then we'll need more injection points, I'm afraid :(.


> > - Several folks asked for pg_stat_aio to come back, in "v1" that showed the
> >   set of currently in-flight AIOs.  That's not particularly hard - except
> >   that it doesn't really fit in the pg_stat_* namespace.
>
> Later message
> postgr.es/m/6vjl6jeaqvyhfbpgwziypwmhem2rwla4o5pgpuxwtg3o3o3jb5@evyzorb5meth is
> considering the name pg_aios.  Works for me.

Cool.


> > --- a/src/backend/storage/aio/aio.c
> > +++ b/src/backend/storage/aio/aio.c
> > @@ -3,6 +3,28 @@
> >   * aio.c
> >   *    AIO - Core Logic
> >   *
> > + * For documentation about how AIO works on a higher level, including a
> > + * schematic example, see README.md.
> > + *
> > + *
> > + * AIO is a complicated subsystem. To keep things navigable it is split across
> > + * a number of files:
> > + *
> > + * - aio.c - core AIO state handling
> > + *
> > + * - aio_init.c - initialization
> > + *
> > + * - aio_io.c - dealing with actual IO, including executing IOs synchronously
> > + *
> > + * - aio_subject.c - functionality related to executing IO for different
> > + *   subjects
> > + *
> > + * - method_*.c - different ways of executing AIO
> > + *
> > + * - read_stream.c - helper for accessing buffered relation data with
> > + *	 look-ahead
> > + *
>
> I felt like some list entries in this new header comment largely restated the
> file name.  Here's how I'd write them to avoid that:

Thanks, adopting.


>  * - method_*.c - different ways of executing AIO (e.g. worker process)
>  * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
>  * - aio_subject.c - callbacks at IO operation lifecycle events
>  * - aio_init.c - per-fork and per-startup-process initialization

I don't particularly like "per-startup-process", because "global
initialization" really is separate (and precedes) from startup processes
startup. Maybe "per-server and per-backend initialization"?

>  * - aio.c - all other topics
>  * - read_stream.c - helper for reading buffered relation data

Did the order you listed the files have a system to it? If so, what is it?


> > --- /dev/null
> > +++ b/src/backend/storage/aio/README.md
> > @@ -0,0 +1,413 @@
> > +# Asynchronous & Direct IO
>
> I would move "### Why Asynchronous IO" to here; that's good background before
> getting into the example.

I moved the example back and forth when writing because different readers
would benefit from a different order and I couldn't quite decide.

So I'm happy to adjust based on your feedback...


> I might also move "### Why Direct / unbuffered IO" to here.  For me as a
> reader, I'd benefit from seeing things in this order:
>
> - "why"
> - condensed usage example like manpage SYNOPSIS, comments and decls removed
> - PgAioHandleState and discussion of valid transitions

Hm - why have PgAioHandleState and its states before the usage example?  Seems
like it'd be harder to understand that way.

> - usage example as it is, with full comments
> - the rest


> ## Synopsis
>
> ioh = pgaio_io_get(CurrentResourceOwner, &ioret);
> pgaio_io_get_ref(ioh, &ior);
> pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
> pgaio_io_set_io_data_32(ioh, (uint32 *) buffer, 1);
> smgrstartreadv(ioh, operation->smgr, forknum, blkno,
>                BufferGetBlock(buffer), 1);
> pgaio_submit_staged();
> pgaio_io_ref_wait(&ior);
> if (ioret.result.status == ARS_ERROR)
>     pgaio_result_log(aio_ret.result, &aio_ret.subject_data, ERROR);

Happy to add this, but I'm not entirely sure if that's really that useful to
have without commentary? The synopsis in manpages is helpful because it
provides the signature of various functions, but this wouldn't...



> > +
> > +## AIO Usage Example
> > +
> > +In many cases code that can benefit from AIO does not directly have to
> > +interact with the AIO interface, but can use AIO via higher-level
> > +abstractions. See [Helpers](#helpers).
> > +
> > +In this example, a buffer will be read into shared buffers.
> > +
> > +```C
> > +/*
> > + * Result of the operation, only to be accessed in this backend.
> > + */
> > +PgAioReturn ioret;
> > +
> > +/*
> > + * Acquire AIO Handle, ioret will get result upon completion.
>
> Consider adding: from here to pgaio_submit_staged(), don't do [description of
> the kind of unacceptable blocking operations].

Hm.  Strictly speaking it's fine to block here, depending on whether
StartBufferIO() was already called.  I'll clarify.


> > +### IO can be started in critical sections
> ...
> > +The need to be able to execute IO in critical sections has substantial design
> > +implication on the AIO subsystem. Mainly because completing IOs (see prior
> > +section) needs to be possible within a critical section, even if the
> > +to-be-completed IO itself was not issued in a critical section. Consider
> > +e.g. the case of a backend first starting a number of writes from shared
> > +buffers and then starting to flush the WAL. Because only a limited amount of
> > +IO can be in-progress at the same time, initiating the IO for flushing the WAL
> > +may require to first finish executing IO executed earlier.
>
> The last line's two appearances of the word "execute" read awkwardly to me,
> and it's an opportunity to use PgAioHandleState terms.  Consider writing the
> last line like "may first advance an existing IO from AHS_PREPARED to
> AHS_COMPLETED_SHARED".

It is indeed awkward.  I don't love referencing the state-constants here
though, somehow that feels like a reference-cycle ;). What about this:

> ... Consider
> e.g. the case of a backend first starting a number of writes from shared
> buffers and then starting to flush the WAL. Because only a limited amount of
> IO can be in-progress at the same time, initiating IO for flushing the WAL may
> require to first complete IO that was started earlier.



> > +### AIO Callbacks
> ...
> > +In addition to completion, AIO callbacks also are called to "prepare" an
> > +IO. This is, e.g., used to acquire buffer pins owned by the AIO subsystem for
> > +IO to/from shared buffers, which is required to handle the case where the
> > +issuing backend errors out and releases its own pins.
>
> Reading this, it's not obvious to me how to reconcile "finishing an IO could
> require pin acquisition" with "finishing an IO could happen in a critical
> section".  Pinning a buffer in a critical section sounds bad.  I vaguely
> recall understanding how it was okay as of my September review, but I've
> already forgotten.  Can this text have a sentence making that explicit?

Ah, yes, that's easy to misunderstand. The answer basically is that we don't
newly pin a buffer, we just increment the reference count by 1. That should
never fail.

How about:
> In addition to completion, AIO callbacks also are called to "prepare" an
> IO. This is, e.g., used to increase buffer reference counts to account for the
> AIO subsystem referencing the buffer, which is required to handle the case
> where the issuing backend errors out and releases its own pins while the IO is
> still ongoing.



> > +### AIO Subjects
> > +
> > +In addition to the completion callbacks describe above, each AIO Handle has
> > +exactly one "subject". Each subject has some space inside an AIO Handle with
> > +information specific to the subject and can provide callbacks to allow to
> > +reopen the underlying file (required for worker mode) and to describe the IO
> > +operation (used for debug logging and error messages).
>
> Can this say roughly how to decide when to add a new subject?

Hm, there obviously is some fuzziness. I was trying to get to some of that by
mentioning that the subject needs to know how to [re-]open a file and describe
the target of the IO in terms that make sense to the user.

E.g. smgr seemed to make sense as a subject as the smgr layer knows how to
open a file by delegating that to the layer below and the layer above just
knows about smgr, not md.c (or other potential smgr implementations).

The reason to keep this separate from the callbacks was that smgr IO going
through shared buffers, bypassing shared buffers and different smgr
implemenentations all could share the same subject implementation, even if
callbacks would differ between these use cases.


How about:

> I.e., if two different uses of AIO can describe the identity of the file being
> operated on the same way, it likely makes sense to use the same
> subject. E.g. different smgr implementations can describe IO with
> RelFileLocator, ForkNumber and BlockNumber and can thus share a subject. In
> contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
> and it would not make sense to use the same subject for smgr and WAL.




> Failing that, can it give examples of what additional subjects might exist
> if certain existing subsystems were to start using AIO?

I think the main ones I can think of are:

1) WAL logging

   This was implemented in v1. I'd guess that "real" WAL logging and
   initializing new WAL segments might use a different subject, but that's
   probably a question of taste.

2) "raw" file IO, for things that don't use the smgr abstraction. I could
   e.g. imagine using AIO in COPY to read / write the FROM/TO file or to
   implement CREATE DATABASE ... STRATEGY file_copy with AIO.

   This was used in v1, e.g. to implement the initial data directory sync
   after a crash. We do that on a filesystem level, not going through smgr
   etc.

3) FE/BE network IO



> > +### AIO Results
> > +
> > +As AIO completion callbacks
> > +[are executed in critical sections](#io-can-be-started-in-critical-sections)
> > +and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
> > +completion callbacks cannot be used to, e.g., make the query that triggered an
> > +IO ERROR out.
> > +
> > +To allow to react to failing IOs the issuing backend can pass a pointer to a
> > +`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
> > +`PgAioReturn` is filled with information about the IO. This includes
> > +information about whether the IO was successful (as a value of
> > +`PgAioResultStatus`) and enough information to raise an error in case of a
> > +failure (via `pgaio_result_log()`, with the error details encoded in
> > +`PgAioResult`).
>
> Can this have a sentence on how this fits in bounded shmem, given the lack of
> guarantees about a backend's responsiveness?  In other words, what makes it
> okay to have requests take arbitrarily long to move from AHS_COMPLETED_SHARED
> to AHS_COMPLETED_LOCAL?

I agree this should be explained somewhere - but not sure this is the best
place.

The reason it's ok is that each backend has a limited number of AIO handles
and if it runs out of IO handles we'll a) check if any IOs can be reclaimed b)
wait for the oldest IO to finish.


Thanks for the review!

Andres Freund

  26. Re: AIO v2.2

    Heikki Linnakangas <hlinnaka@iki.fi> — 2025-01-07T15:09:58Z

    On 01/01/2025 06:03, Andres Freund wrote:
> Hi,
> 
> Attached is a new version of the AIO patchset.

I haven't gone through it all yet, but some comments below.

> The biggest changes are:
> 
> - The README has been extended with an overview of the API. I think it gives a
>    good overview of how the API fits together.  I'd be very good to get
>    feedback from folks that aren't as familiar with AIO, I can't really see
>    what's easy/hard anymore.

Thanks, the README is super helpful! I was overwhelmed by all the new 
concepts before, now it all makes much more sense.

Now that it's all laid out more clearly, I see how many different 
concepts and states there really are:

- For a single IO, there is an "IO handle", "IO references", and an "IO 
return". You first allocate an IO handle (PgAioHandle), and then you get 
a reference (PgAioHandleRef) and an "IO return" (PgAioReturn) struct for it.

- An IO handle has eight different states (PgAioHandleState).

I'm sure all those concepts exist for a reason. But still I wonder: can 
we simplify?

pgaio_io_get() and pgaio_io_release() are a bit asymmetric, I'd suggest 
pgaio_io_acquire() or similar. "get" also feels very innocent, even 
though it may wait for previous IO to finish. Especially when 
pgaio_io_get_ref() actually is innocent.

> typedef enum PgAioHandleState
> {
> 	/* not in use */
> 	AHS_IDLE = 0,
> 
> 	/* returned by pgaio_io_get() */
> 	AHS_HANDED_OUT,
> 
> 	/* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
> 	AHS_DEFINED,
> 
> 	/* subject's prepare() callback has been called */
> 	AHS_PREPARED,
> 
> 	/* IO has been submitted and is being executed */
> 	AHS_IN_FLIGHT,
> 
> 	/* IO finished, but result has not yet been processed */
> 	AHS_REAPED,
> 
> 	/* IO completed, shared completion has been called */
> 	AHS_COMPLETED_SHARED,
> 
> 	/* IO completed, local completion has been called */
> 	AHS_COMPLETED_LOCAL,
> } PgAioHandleState;

Do we need to distinguish between DEFINED and PREPARED? At quick glance, 
those states are treated the same. (The comment refers to 
pgaio_io_start_*() functions, but there's no such thing)

I didn't quite understand the point of the prepare callbacks. For 
example, when AsyncReadBuffers() calls smgrstartreadv(), the 
shared_buffer_readv_prepare() callback will be called. Why doesn't 
AsyncReadBuffers() do the "prepare" work itself directly; why does it 
need to be in a callback? I assume it's somehow related to error 
handling, but I didn't quite get it. Perhaps an "abort" callback that'd 
be called on error, instead of a "prepare" callback, would be better?

There are some synonyms used in the code: I think "in-flight" and 
"submitted" mean the same thing. And "prepared" and "staged". I'd 
suggest picking just one term for each concept.

I didn't understand the COMPLETED_SHARED and COMPLETED_LOCAL states. 
does a single IO go through both states, or are the mutually exclusive? 
At quick glance, I don't actually see any code that would set the 
COMPLETED_LOCAL state; is it dead code?

REAPED feels like a bad name. It sounds like a later stage than 
COMPLETED, but it's actually vice versa.

I'm a little surprised that the term "IO request" isn't used anywhere. I 
have no concrete suggestion, but perhaps that would be a useful term.

> - Retries for partial IOs (i.e. short reads) are now implemented. Turned out
>    to take all of three lines and adding one missing variable initialization.

:-)

> - There's no obvious way to tell "internal" function operating on an IO handle
>    apart from functions that are expected to be called by the issuer of an IO.
> 
>    One way to deal with this would be to introduce a distinct "issuer IO
>    reference" type.  I think that might be a good idea, it would also make it
>    clearer that a good number of the functions can only be called by the
>    issuer, before the IO is submitted.
> 
>    This would also make it easier to order functions more sensibly in aio.c, as
>    all the issuer functions would be together.
> 
>    The functions on AIO handles that everyone can call already have a distinct
>    type (PgAioHandleRef vs PgAioHandle*).

Hmm, yeah I think you might be onto something here.

Could pgaio_io_get() return an PgAioHandleRef directly, so that the 
issuer would never see a raw PgAioHandle ?


Finally, attached are a couple of typos and other trivial suggestions.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  27. Re: AIO v2.2

    Heikki Linnakangas <hlinnaka@iki.fi> — 2025-01-07T16:08:51Z

    On LWLockDisown():

> +/*
> + * Stop treating lock as held by current backend.
> + *
> + * After calling this function it's the callers responsibility to ensure that
> + * the lock gets released, even in case of an error. This only is desirable if
> + * the lock is going to be released in a different process than the process
> + * that acquired it.
> + *
> + * Returns the mode in which the lock was held by the current backend.

Returning the lock mode feels a bit ad hoc..

> + * NB: This will leave lock->owner pointing to the current backend (if
> + * LOCK_DEBUG is set). We could add a separate flag indicating that, but it
> + * doesn't really seem worth it.

Hmm. I won't insist, but I feel it probably would be worth it. This is 
only in LOCK_DEBUG mode so there's no performance penalty in non-debug 
builds, and when you do compile with LOCK_DEBUG you probably appreciate 
any extra information.

> + * NB: This does not call RESUME_INTERRUPTS(), but leaves that responsibility
> + * of the caller.
> + */

That feels weird. The only caller outside lwlock.c does call 
RESUME_INTERRUPTS() immediately.

Perhaps it'd make for a better external interface if LWLockDisown() did 
call RESUME_INTERRUPTS(), and there was a separate internal version that 
didn't. And it might make more sense for the external version to return 
'void' while we're at it. Returning a value that the caller ignores is 
harmless, of course, but it feels a bit weird. It makes you wonder what 
you're supposed to do with it.

> +	{
> +		{"io_method", PGC_POSTMASTER, RESOURCES_MEM,
> +			gettext_noop("Selects the method of asynchronous I/O to use."),
> +			NULL
> +		},
> +		&io_method,
> +		DEFAULT_IO_METHOD, io_method_options,
> +		NULL, assign_io_method, NULL
> +	},
> +

The description is a bit funny because synchronous I/O is one of the 
possible methods.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  28. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-07T16:11:45Z

    Hi,

On 2025-01-07 17:09:58 +0200, Heikki Linnakangas wrote:
> On 01/01/2025 06:03, Andres Freund wrote:
> > Hi,
> > 
> > Attached is a new version of the AIO patchset.
> 
> I haven't gone through it all yet, but some comments below.

Thanks!


> > The biggest changes are:
> > 
> > - The README has been extended with an overview of the API. I think it gives a
> >    good overview of how the API fits together.  I'd be very good to get
> >    feedback from folks that aren't as familiar with AIO, I can't really see
> >    what's easy/hard anymore.
> 
> Thanks, the README is super helpful! I was overwhelmed by all the new
> concepts before, now it all makes much more sense.
> 
> Now that it's all laid out more clearly, I see how many different concepts
> and states there really are:
> 
> - For a single IO, there is an "IO handle", "IO references", and an "IO
> return". You first allocate an IO handle (PgAioHandle), and then you get a
> reference (PgAioHandleRef) and an "IO return" (PgAioReturn) struct for it.
> 
> - An IO handle has eight different states (PgAioHandleState).
> 
> I'm sure all those concepts exist for a reason. But still I wonder: can we
> simplify?

Probably, but it's not exactly obvious to me where.

The difference between a handle and a reference is useful right now, to have
some separation between the functions that can be called by anyone (taking a
PgAioHandleRef) and only by the issuer (PgAioHandle). That might better be
solved by having a PgAioHandleIssuerRef ref or something.

Having PgAioReturn be separate from the AIO handle turns out to be rather
crucial, otherwise it's very hard to guarantee "forward progress",
i.e. guarantee that pgaio_io_get() will return something without blocking
forever.



> pgaio_io_get() and pgaio_io_release() are a bit asymmetric, I'd suggest
> pgaio_io_acquire() or similar. "get" also feels very innocent, even though
> it may wait for previous IO to finish. Especially when pgaio_io_get_ref()
> actually is innocent.

WFM.


> > typedef enum PgAioHandleState
> > {
> > 	/* not in use */
> > 	AHS_IDLE = 0,
> > 
> > 	/* returned by pgaio_io_get() */
> > 	AHS_HANDED_OUT,
> > 
> > 	/* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
> > 	AHS_DEFINED,
> > 
> > 	/* subject's prepare() callback has been called */
> > 	AHS_PREPARED,
> > 
> > 	/* IO has been submitted and is being executed */
> > 	AHS_IN_FLIGHT,
> > 
> > 	/* IO finished, but result has not yet been processed */
> > 	AHS_REAPED,
> > 
> > 	/* IO completed, shared completion has been called */
> > 	AHS_COMPLETED_SHARED,
> > 
> > 	/* IO completed, local completion has been called */
> > 	AHS_COMPLETED_LOCAL,
> > } PgAioHandleState;
> 
> Do we need to distinguish between DEFINED and PREPARED?

I found it to be rather confusing if it's not possible to tell if some action
(like the prepare callback) has already happened, or not.  It's useful to be
able look at an IO in a backtrace or such and see exactly in what state it is
in.

In v1 I had several of the above states managed as separate boolean variables
- that turned out to be a huge mess, it's a lot easier to understand if
there's a single strictly monotonically increasing state.


> At quick glance, those states are treated the same. (The comment refers to
> pgaio_io_start_*() functions, but there's no such thing)

They're called pgaio_io_prep_{readv,writev} now, updated the comment.


> I didn't quite understand the point of the prepare callbacks. For example,
> when AsyncReadBuffers() calls smgrstartreadv(), the
> shared_buffer_readv_prepare() callback will be called. Why doesn't
> AsyncReadBuffers() do the "prepare" work itself directly; why does it need
> to be in a callback?

One big part of it is "ownership" - while the IO isn't completely "assembled",
we can release all buffer pins etc in case of an error. But if the error
happens just after the IO was staged, we can't - the buffer is still
referenced by the IO. For that the AIO subystem needs to take its own pins
etc.  Initially the prepare callback didn't exist, the code in
AsyncReadBuffers() was a lot more complicated before it.


> I assume it's somehow related to error handling, but I didn't quite get
> it. Perhaps an "abort" callback that'd be called on error, instead of a
> "prepare" callback, would be better?

I don't think an error callback would be helpful - the whole thing is that we
basically need claim ownership of all IO related resources IFF the IO is
staged. Not before (because then the IO not getting staged would mean we have
a resource leak), not after (because we might error out and thus not keep
e.g. buffers pinned).


> There are some synonyms used in the code: I think "in-flight" and
> "submitted" mean the same thing.

Fair.  I guess in my mind the process of moving an IO into flight is
"submitting" and the state of not having been submitted but not yet having
completed is being in flight. But that's probably not useful.


> And "prepared" and "staged". I'd suggest picking just one term for each
> concept.

Agreed.


> I didn't understand the COMPLETED_SHARED and COMPLETED_LOCAL states. does a
> single IO go through both states, or are the mutually exclusive? At quick
> glance, I don't actually see any code that would set the COMPLETED_LOCAL
> state; is it dead code?

It's dead code right now. I've made it dead and undead a couple times
:/. Unfortunately I think I need to revive it to make some corner cases with
temporary tables work (AIO for temp table is executed via IO uring, another
backend waits for *another* IO executed via that IO uring instance and reaps
the completion -> we can't update the local buffer state in the shared
completion callback).


> REAPED feels like a bad name. It sounds like a later stage than COMPLETED,
> but it's actually vice versa.

What would you call having gotten "completion notifications" from the kernel,
but not having processed them?



> > - There's no obvious way to tell "internal" function operating on an IO handle
> >    apart from functions that are expected to be called by the issuer of an IO.
> > 
> >    One way to deal with this would be to introduce a distinct "issuer IO
> >    reference" type.  I think that might be a good idea, it would also make it
> >    clearer that a good number of the functions can only be called by the
> >    issuer, before the IO is submitted.
> > 
> >    This would also make it easier to order functions more sensibly in aio.c, as
> >    all the issuer functions would be together.
> > 
> >    The functions on AIO handles that everyone can call already have a distinct
> >    type (PgAioHandleRef vs PgAioHandle*).
> 
> Hmm, yeah I think you might be onto something here.

I'll give it a try.


> Could pgaio_io_get() return an PgAioHandleRef directly, so that the issuer
> would never see a raw PgAioHandle ?

Don't think that would be helpful - that way there'd be no difference at all
anymore between what functions any backend can call and what the issuer can
do.


>
> Finally, attached are a couple of typos and other trivial suggestions.

Integrating...


Thanks!

Andres

  29. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-07T16:32:10Z

    Hi,

On 2025-01-07 18:08:51 +0200, Heikki Linnakangas wrote:
> On LWLockDisown():
> 
> > +/*
> > + * Stop treating lock as held by current backend.
> > + *
> > + * After calling this function it's the callers responsibility to ensure that
> > + * the lock gets released, even in case of an error. This only is desirable if
> > + * the lock is going to be released in a different process than the process
> > + * that acquired it.
> > + *
> > + * Returns the mode in which the lock was held by the current backend.
> 
> Returning the lock mode feels a bit ad hoc..

It seemed useful to me, that way callers could verify that the released lock
level is actually what it expected. What do we gain by hiding this information
anyway?

Orthogonal: I think it was a mistake that LWLockRelease() didn't require the
to-be-releaased lock mode to be passed in...


> > + * NB: This will leave lock->owner pointing to the current backend (if
> > + * LOCK_DEBUG is set). We could add a separate flag indicating that, but it
> > + * doesn't really seem worth it.
> 
> Hmm. I won't insist, but I feel it probably would be worth it. This is only
> in LOCK_DEBUG mode so there's no performance penalty in non-debug builds,
> and when you do compile with LOCK_DEBUG you probably appreciate any extra
> information.

I actually thought it'd be more useful if it stays pointing to the 'original
owner'.

When you say "it" would be worth it, you mean resetting owner, or adding a
flag indicating that it's a disowned lock?



> > + * NB: This does not call RESUME_INTERRUPTS(), but leaves that responsibility
> > + * of the caller.
> > + */
> 
> That feels weird. The only caller outside lwlock.c does call
> RESUME_INTERRUPTS() immediately.

Yea, I didn't feel happy with it either. It just seemed that the cure (a
separate function, or a parameter indicating whether interrupts should be
resumed) was as bad as the disease.


> Perhaps it'd make for a better external interface if LWLockDisown() did call
> RESUME_INTERRUPTS(), and there was a separate internal version that didn't.

Hm, that seems more complicated than it's worth.  I'd either leave it as-is,
or add a parameter to LWLockDisown to indicate if interrupts should be
resumed.


> And it might make more sense for the external version to return 'void' while
> we're at it. Returning a value that the caller ignores is harmless, of
> course, but it feels a bit weird. It makes you wonder what you're supposed
> to do with it.

This one I disagree with, I think it makes a lot of sense to return the lock
mode of the lock you just disowned.

Doubtful it matters, but the compiler can trivially optimize that out for the
lwlock.c callers.


> > +	{
> > +		{"io_method", PGC_POSTMASTER, RESOURCES_MEM,
> > +			gettext_noop("Selects the method of asynchronous I/O to use."),
> > +			NULL
> > +		},
> > +		&io_method,
> > +		DEFAULT_IO_METHOD, io_method_options,
> > +		NULL, assign_io_method, NULL
> > +	},
> > +
> 
> The description is a bit funny because synchronous I/O is one of the
> possible methods.

Hah.  How about:

"Selects the method of, potentially asynchronous, IO execution."?

Greetings,

Andres Freund

  30. Re: AIO v2.2

    Noah Misch <noah@leadboat.com> — 2025-01-07T19:10:59Z

    On Mon, Jan 06, 2025 at 04:40:26PM -0500, Andres Freund wrote:
> On 2025-01-06 10:52:20 -0800, Noah Misch wrote:
> > On Tue, Dec 31, 2024 at 11:03:33PM -0500, Andres Freund wrote:

> - We have pretty no testing for IO errors.

Yes, that's remained a gap.  I've wondered how much to address this via
targeted tests of specific sites vs. fuzzing, iterative fault injection, or
some other approach closer to brute force.

> > I'd be most interested in the
> > cases that would be undetected deadlocks under a naive design.  An example
> > appeared at the end of postgr.es/m/20240916144349.74.nmisch@google.com
> 
> That's a good one, yea.
> 
> I think I'll try to translate the regression tests I wrote into an isolation
> test, I hope that'll make it a bit easier to cover more cases.
> 
> And then we'll need more injection points, I'm afraid :(.

Sounds good.

> >  * - method_*.c - different ways of executing AIO (e.g. worker process)
> >  * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
> >  * - aio_subject.c - callbacks at IO operation lifecycle events
> >  * - aio_init.c - per-fork and per-startup-process initialization
> 
> I don't particularly like "per-startup-process", because "global
> initialization" really is separate (and precedes) from startup processes
> startup. Maybe "per-server and per-backend initialization"?

That works for me.  I wrote "per-startup-process" because it can happen more
than once in a postmaster that reaches "all server processes terminated;
reinitializing".  That said, there's little risk of "per-server" giving folks
a materially wrong idea.

> >  * - aio.c - all other topics
> >  * - read_stream.c - helper for reading buffered relation data
> 
> Did the order you listed the files have a system to it? If so, what is it?

The rough idea was to avoid forward references:

 * - method_*.c - different ways of executing AIO (e.g. worker process)
makes sense without other background
 * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
refers to methods, so listed after methods
 * - aio_subject.c - callbacks at IO operation lifecycle events
refers to IO ops, so listed after aio_io.c
 * - aio_init.c - per-fork and per-startup-process initialization
no surprise that this code will exist somewhere, so list it lower to deemphasize it
 * - aio.c - all other topics
default route, hence last
 * - read_stream.c - helper for reading buffered relation data
could just as easily come first, not last
could be under a distinct heading like "Recommended abstractions:" 

> > I'd benefit from seeing things in this order:
> >
> > - "why"
> > - condensed usage example like manpage SYNOPSIS, comments and decls removed
> > - PgAioHandleState and discussion of valid transitions
> 
> Hm - why have PgAioHandleState and its states before the usage example?  Seems
> like it'd be harder to understand that way.

I usually look at the data structures before the code that manipulates them.
(Similarly, I look at the map before the directions.)  I wouldn't mind it
appearing after the usage example, since order preferences do vary.

> > - usage example as it is, with full comments
> > - the rest
> 
> 
> > ## Synopsis
> >
> > ioh = pgaio_io_get(CurrentResourceOwner, &ioret);
> > pgaio_io_get_ref(ioh, &ior);
> > pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
> > pgaio_io_set_io_data_32(ioh, (uint32 *) buffer, 1);
> > smgrstartreadv(ioh, operation->smgr, forknum, blkno,
> >                BufferGetBlock(buffer), 1);
> > pgaio_submit_staged();
> > pgaio_io_ref_wait(&ior);
> > if (ioret.result.status == ARS_ERROR)
> >     pgaio_result_log(aio_ret.result, &aio_ret.subject_data, ERROR);
> 
> Happy to add this, but I'm not entirely sure if that's really that useful to
> have without commentary? The synopsis in manpages is helpful because it
> provides the signature of various functions, but this wouldn't...

I'm not sure either.  Let's drop that idea.

> > > +### IO can be started in critical sections
> > ...
> > > +The need to be able to execute IO in critical sections has substantial design
> > > +implication on the AIO subsystem. Mainly because completing IOs (see prior
> > > +section) needs to be possible within a critical section, even if the
> > > +to-be-completed IO itself was not issued in a critical section. Consider
> > > +e.g. the case of a backend first starting a number of writes from shared
> > > +buffers and then starting to flush the WAL. Because only a limited amount of
> > > +IO can be in-progress at the same time, initiating the IO for flushing the WAL
> > > +may require to first finish executing IO executed earlier.
> >
> > The last line's two appearances of the word "execute" read awkwardly to me,
> > and it's an opportunity to use PgAioHandleState terms.  Consider writing the
> > last line like "may first advance an existing IO from AHS_PREPARED to
> > AHS_COMPLETED_SHARED".
> 
> It is indeed awkward.  I don't love referencing the state-constants here
> though, somehow that feels like a reference-cycle ;). What about this:
> 
> > ... Consider
> > e.g. the case of a backend first starting a number of writes from shared
> > buffers and then starting to flush the WAL. Because only a limited amount of
> > IO can be in-progress at the same time, initiating IO for flushing the WAL may
> > require to first complete IO that was started earlier.

That's non-awkward.  I like specific state names here since "complete" could
mean AHS_COMPLETED_SHARED or AHS_COMPLETED_LOCAL, and it matters here.  If the
state names changed so AHS_COMPLETED_LOCAL dropped the word "complete", that
too would solve it.

> > > +### AIO Callbacks
> > ...
> > > +In addition to completion, AIO callbacks also are called to "prepare" an
> > > +IO. This is, e.g., used to acquire buffer pins owned by the AIO subsystem for
> > > +IO to/from shared buffers, which is required to handle the case where the
> > > +issuing backend errors out and releases its own pins.
> >
> > Reading this, it's not obvious to me how to reconcile "finishing an IO could
> > require pin acquisition" with "finishing an IO could happen in a critical
> > section".  Pinning a buffer in a critical section sounds bad.  I vaguely
> > recall understanding how it was okay as of my September review, but I've
> > already forgotten.  Can this text have a sentence making that explicit?
> 
> Ah, yes, that's easy to misunderstand. The answer basically is that we don't
> newly pin a buffer, we just increment the reference count by 1. That should
> never fail.
> 
> How about:
> > In addition to completion, AIO callbacks also are called to "prepare" an
> > IO. This is, e.g., used to increase buffer reference counts to account for the
> > AIO subsystem referencing the buffer, which is required to handle the case
> > where the issuing backend errors out and releases its own pins while the IO is
> > still ongoing.

Perfect.

> > > +### AIO Subjects
> > > +
> > > +In addition to the completion callbacks describe above, each AIO Handle has
> > > +exactly one "subject". Each subject has some space inside an AIO Handle with
> > > +information specific to the subject and can provide callbacks to allow to
> > > +reopen the underlying file (required for worker mode) and to describe the IO
> > > +operation (used for debug logging and error messages).
> >
> > Can this say roughly how to decide when to add a new subject?
> 
> Hm, there obviously is some fuzziness. I was trying to get to some of that by
> mentioning that the subject needs to know how to [re-]open a file and describe
> the target of the IO in terms that make sense to the user.
> 
> E.g. smgr seemed to make sense as a subject as the smgr layer knows how to
> open a file by delegating that to the layer below and the layer above just
> knows about smgr, not md.c (or other potential smgr implementations).
> 
> The reason to keep this separate from the callbacks was that smgr IO going
> through shared buffers, bypassing shared buffers and different smgr
> implemenentations all could share the same subject implementation, even if
> callbacks would differ between these use cases.
> 
> 
> How about:
> 
> > I.e., if two different uses of AIO can describe the identity of the file being
> > operated on the same way, it likely makes sense to use the same
> > subject. E.g. different smgr implementations can describe IO with
> > RelFileLocator, ForkNumber and BlockNumber and can thus share a subject. In
> > contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
> > and it would not make sense to use the same subject for smgr and WAL.

Sounds good to include.

> > Can this have a sentence on how this fits in bounded shmem, given the lack of
> > guarantees about a backend's responsiveness?  In other words, what makes it
> > okay to have requests take arbitrarily long to move from AHS_COMPLETED_SHARED
> > to AHS_COMPLETED_LOCAL?
> 
> I agree this should be explained somewhere - but not sure this is the best
> place.
> 
> The reason it's ok is that each backend has a limited number of AIO handles
> and if it runs out of IO handles we'll a) check if any IOs can be reclaimed b)
> wait for the oldest IO to finish.

Reading it again today, that topic may already have adequate coverage.

  31. Re: AIO v2.2

    Robert Haas <robertmhaas@gmail.com> — 2025-01-07T19:59:58Z

    On Tue, Jan 7, 2025 at 11:11 AM Andres Freund <andres@anarazel.de> wrote:
> The difference between a handle and a reference is useful right now, to have
> some separation between the functions that can be called by anyone (taking a
> PgAioHandleRef) and only by the issuer (PgAioHandle). That might better be
> solved by having a PgAioHandleIssuerRef ref or something.

To me, those names don't convey that. I would perhaps call the thing
that supports issuer-only operations a "PgAio" and the thing other
people can use a "PgAioHandle". Or "PgAioRequest" and "PgAioHandle" or
something like that. With PgAioHandleRef, IMHO you've got two words
that both imply a layer of indirection -- "handle" and "ref" -- which
doesn't seem quite as nice, because then the other thing --
"PgAioHandle" still sort of implies
one layer of indirection and the whole thing seems a bit less clear.

(I say all of this having looked at nothing, so feel free to ignore me
if that doesn't sound coherent.)

> > REAPED feels like a bad name. It sounds like a later stage than COMPLETED,
> > but it's actually vice versa.
>
> What would you call having gotten "completion notifications" from the kernel,
> but not having processed them?

The Linux kernel calls those zombie processes, so we could call it a
ZOMBIE state, but that seems like it might be a bit of inside
baseball. I do agree with Heikki that REAPED sounds later than
COMPLETED, because you reap zombie processes by collecting their exit
status. Maybe you could have AHS_COMPLETE or AHS_IO_COMPLETE for the
state where the I/O is done but there's still completion-related work
to be done, and then the other state could be AHS_DONE or AHS_FINISHED
or AHS_FINAL or AHS_REAPED or something.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  32. Re: AIO v2.2

    Heikki Linnakangas <hlinnaka@iki.fi> — 2025-01-07T20:09:56Z

    On 07/01/2025 18:11, Andres Freund wrote:
> The difference between a handle and a reference is useful right now, to have
> some separation between the functions that can be called by anyone (taking a
> PgAioHandleRef) and only by the issuer (PgAioHandle). That might better be
> solved by having a PgAioHandleIssuerRef ref or something.
> 
> Having PgAioReturn be separate from the AIO handle turns out to be rather
> crucial, otherwise it's very hard to guarantee "forward progress",
> i.e. guarantee that pgaio_io_get() will return something without blocking
> forever.

Right, yeah, I can see that.

>>> typedef enum PgAioHandleState
>>> {
>>> 	/* not in use */
>>> 	AHS_IDLE = 0,
>>>
>>> 	/* returned by pgaio_io_get() */
>>> 	AHS_HANDED_OUT,
>>>
>>> 	/* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
>>> 	AHS_DEFINED,
>>>
>>> 	/* subject's prepare() callback has been called */
>>> 	AHS_PREPARED,
>>>
>>> 	/* IO has been submitted and is being executed */
>>> 	AHS_IN_FLIGHT,
>>>
>>> 	/* IO finished, but result has not yet been processed */
>>> 	AHS_REAPED,
>>>
>>> 	/* IO completed, shared completion has been called */
>>> 	AHS_COMPLETED_SHARED,
>>>
>>> 	/* IO completed, local completion has been called */
>>> 	AHS_COMPLETED_LOCAL,
>>> } PgAioHandleState;
>>
>> Do we need to distinguish between DEFINED and PREPARED?
> 
> I found it to be rather confusing if it's not possible to tell if some action
> (like the prepare callback) has already happened, or not.  It's useful to be
> able look at an IO in a backtrace or such and see exactly in what state it is
> in.

I see.

> In v1 I had several of the above states managed as separate boolean variables
> - that turned out to be a huge mess, it's a lot easier to understand if
> there's a single strictly monotonically increasing state.

Agreed on that

>> I didn't quite understand the point of the prepare callbacks. For example,
>> when AsyncReadBuffers() calls smgrstartreadv(), the
>> shared_buffer_readv_prepare() callback will be called. Why doesn't
>> AsyncReadBuffers() do the "prepare" work itself directly; why does it need
>> to be in a callback?
> 
> One big part of it is "ownership" - while the IO isn't completely "assembled",
> we can release all buffer pins etc in case of an error. But if the error
> happens just after the IO was staged, we can't - the buffer is still
> referenced by the IO. For that the AIO subystem needs to take its own pins
> etc.  Initially the prepare callback didn't exist, the code in
> AsyncReadBuffers() was a lot more complicated before it.
> 
> 
>> I assume it's somehow related to error handling, but I didn't quite get
>> it. Perhaps an "abort" callback that'd be called on error, instead of a
>> "prepare" callback, would be better?
> 
> I don't think an error callback would be helpful - the whole thing is that we
> basically need claim ownership of all IO related resources IFF the IO is
> staged. Not before (because then the IO not getting staged would mean we have
> a resource leak), not after (because we might error out and thus not keep
> e.g. buffers pinned).

Hmm. The comments say that when you call smgrstartreadv(), the IO handle 
may no longer be modified, as the IO may be executed immediately. What 
if we changed that so that it never submits the IO, only adds the 
necessary callbacks to it?

In that world, when smgrstartreadv() returns, the necessary details and 
completion callbacks have been set in the IO handle, but the caller can 
still do more preparation before the IO is submitted. The caller must 
ensure that it gets submitted, however, so no erroring out in that state.

Currently the call stack looks like this:

AsyncReadBuffers()
-> smgrstartreadv()
   -> mdstartreadv()
     -> FileStartReadV()
       -> pgaio_io_prep_readv()
         -> shared_buffer_readv_prepare() (callback)
         <- (return)
       <- (return)
     <- (return)
   <- (return)
<- (return)

I'm thinking that the prepare work is done "on the way up" instead:

AsyncReadBuffers()
-> smgrstartreadv()
   -> mdstartreadv()
     -> FileStartReadV()
       -> pgaio_io_prep_readv()
       <- (return)
     <- (return)
   <- (return)
-> shared_buffer_readv_prepare()
<- (return)

Attached is a patch to demonstrate concretely what I mean.

This adds a new pgaio_io_stage() step to the issuer, and the issuer 
needs to call the prepare functions explicitly, instead of having them 
as callbacks. Nominally that's more steps, but IMHO it's better to be 
explicit. The same actions were happening previously too, it was just 
hidden in the callback. I updated the README to show that too.

I'm not wedded to this, but it feels a little better to me.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  33. Re: AIO v2.0

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-01-08T14:04:39Z

    On Mon, Jan 6, 2025 at 5:28 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2024-12-19 17:29:12 -0500, Andres Freund wrote:
> > > Not about patch itself, but questions about related stack functionality:
> > > ----------------------------------------------------------------------------------------------------
> > >
> > >
> > > 7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
> > > on how to inspect real I/O calls requested to review if the code is issuing
> > > sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
> > > perhaps using some bpftrace / xfsslower is the best way to go ?
> >
> > I think we still want something like it, but I don't think it needs to be in
> > the initial commits.
>
> After I got this question from Thomas as well, I started hacking one up.
>
> What information would you like to see?
>
> Here's what I currently have:
..
> ├─[ RECORD 2 ]───┼────────────────────────────────────────────────┤
> │ pid            │ 358212                                         │
> │ io_id          │ 2051                                           │
> │ io_generation  │ 4199                                           │
> │ state          │ IN_FLIGHT                                      │
> │ operation      │ read                                           │
> │ offset         │ 511967232                                      │
> │ length         │ 262144                                         │
> │ subject        │ smgr                                           │
> │ iovec_data_len │ 32                                             │
> │ raw_result     │ (null)                                         │
> │ result         │ UNKNOWN                                        │
> │ error_desc     │ (null)                                         │
> │ subject_desc   │ blocks 1373216..1373247 in file "base/5/16388" │
> │ flag_sync      │ f                                              │
> │ flag_localmem  │ f                                              │
> │ flag_buffered  │ t                                              │

Cool! It's more than enough for me in future, thanks!

> I didn't think that pg_stat_* was quite the right namespace, given that it
> shows not stats, but the currently ongoing IOs.  I am going with pg_aios for
> now, but I don't particularly like that.

If you are looking for other proposals:
* pg_aios_progress ? (to follow pattern of pg_stat_copy|vaccuum_progress?)
* pg_debug_aios ?
* pg_debug_io ?

> I think we'll want a pg_stat_aio as well, tracking things like:
>
> - how often the queue to IO workes was full
> - how many times we submitted IO to the kernel (<= #ios with io_uring)
> - how many times we asked the kernel for events (<= #ios with io_uring)
> - how many times we had to wait for in-flight IOs before issuing more IOs

If I could dream of one thing that would be 99.9% percentile of IO
response times in milliseconds for different classes of I/O traffic
(read/write/flush). But it sounds like it would be very similiar to
pg_stat_io and potentially would have to be
per-tablespace/IO-traffic(subject)-type too. AFAIU pg_stat_io has
improper structure to have that there.

BTW: before trying to even start to compile that AIO v2.2* and
responding to the previous review, what are You looking interested to
hear the most about it so that it adds some value ? Any workload
specific measurements? just general feedback, functionality gaps?
Integrity/data testing with stuff like dm-dust, dm-flakey, dm-delay to
try the error handling routines? Some kind of AIO <-> standby/recovery
interactions?

* - btw, Date: 2025-01-01 04:03:33 - I saw what you did there! so
let's officially recognize the 2025 as the year of AIO in PG, as it
was 1st message :D

-J.

  34. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2025-01-08T20:58:50Z

    Hi,

On 2025-01-08 15:04:39 +0100, Jakub Wartak wrote:
> On Mon, Jan 6, 2025 at 5:28 PM Andres Freund <andres@anarazel.de> wrote:
> > I didn't think that pg_stat_* was quite the right namespace, given that it
> > shows not stats, but the currently ongoing IOs.  I am going with pg_aios for
> > now, but I don't particularly like that.
>
> If you are looking for other proposals:
> * pg_aios_progress ? (to follow pattern of pg_stat_copy|vaccuum_progress?)
> * pg_debug_aios ?
> * pg_debug_io ?

I think pg_aios is better than those, if not by much.  Seems others are ok
with that name too. And we easily can evolve it later.


> > I think we'll want a pg_stat_aio as well, tracking things like:
> >
> > - how often the queue to IO workes was full
> > - how many times we submitted IO to the kernel (<= #ios with io_uring)
> > - how many times we asked the kernel for events (<= #ios with io_uring)
> > - how many times we had to wait for in-flight IOs before issuing more IOs
>
> If I could dream of one thing that would be 99.9% percentile of IO
> response times in milliseconds for different classes of I/O traffic
> (read/write/flush). But it sounds like it would be very similiar to
> pg_stat_io and potentially would have to be
> per-tablespace/IO-traffic(subject)-type too.

Yea, that's a significant project on its own. It's not that cheap to compute
reasonably accurate percentiles and we have no infrastructure for doing so
right now.


> AFAIU pg_stat_io has improper structure to have that there.

Hm, not obvious to me why? It might make the view a bit wide to add it as an
additional column, but otherwise I don't see a problem?


> BTW: before trying to even start to compile that AIO v2.2* and
> responding to the previous review, what are You looking interested to
> hear the most about it so that it adds some value?

Due to the rather limited "users" of AIO in the patchset, I think most
benchmarks aren't expected to show any meaningful gains. However, they
shouldn't show any significant regressions either (when not using direct
IO). I think trying to find regressions would be a rather valuable thing.


I'm tempted to collect a few of the reasonbly-ready read stream conversions
into the patchset, to make the potential gains more visible. But I am not sure
it's a good investment of time right now.


One small regression I do know about, namely scans of large relations that are
bigger than shared buffers but do fit in the kernel page cache. The increase
of BAS_BULKREAD does cause a small slowdown - but without it we never can do
sufficient asynchronous IO.   I think the slowdown is small enough to just
accept that, but it's worth qualifying that on a few machines.


> Any workload specific measurements? just general feedback, functionality
> gaps?

To see the benefits it'd be interesting to compare:

1) sequential scan performance with data not in shared buffers, using buffered IO
2) same, but using direct IO when testing the patch
3) checkpoint performance


In my experiments 1) gains a decent amount of performance in many cases, but
nothing overwhelming - sequential scans are easy for the kernel to read ahead.

I do see very significant gains for 2) - On a system with 10 striped NVMe SSDs
that each can do ~3.5 GB/s I measured very parallel sequential scans (I had
to use ALTER TABLE to get sufficient numbers of workers):

master:			~18 GB/s
patch, buffered:        ~20 GB/s
patch, direct, worker:	~28 GB/s
patch, direct, uring:	~35 GB/s


This was with io_workers=32, io_max_concurrency=128,
effective_io_concurrency=1000 (doesn't need to be that high, but it's what I
still have the numbers for).


This was without data checksums enabled as otherwise the checksum code becomes
a *huge* bottleneck.


I also see significant gains with 3). Bigger when using direct IO.  One
complicating factor measuring 3) is that the first write to a block will often
be slower than subsequent writes because the filesystem will need to update
some journaled metadata, presenting a bottleneck.

Checkpoint performance is also severely limited by data checksum computation
if enabled - independent of this patchset.


One annoying thing when testing DIO is that right now VACUUM will be rather
slow if the data isn't already in s_b, as it isn't yet read-stream-ified.



> Integrity/data testing with stuff like dm-dust, dm-flakey, dm-delay
> to try the error handling routines?

Hm. I don't think that's going to work very well even on master. If the
filesystem fails there's not much that PG can do...


> Some kind of AIO <-> standby/recovery interactions?

I wouldn't expect anything there. I think Thomas somewhere has a patch that
read-stream-ifies recovery prefetching, once that's done it would be more
interesting.


> * - btw, Date: 2025-01-01 04:03:33 - I saw what you did there! so
> let's officially recognize the 2025 as the year of AIO in PG, as it
> was 1st message :D

Hah, that was actually the opposite of what I intended :). I'd hoped to post
earlier, but jetlag had caught up with me...

Greetings,

Andres Freund

  35. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-08T22:56:04Z

    Hi,

On 2025-01-07 22:09:56 +0200, Heikki Linnakangas wrote:
> On 07/01/2025 18:11, Andres Freund wrote:
> > > I didn't quite understand the point of the prepare callbacks. For example,
> > > when AsyncReadBuffers() calls smgrstartreadv(), the
> > > shared_buffer_readv_prepare() callback will be called. Why doesn't
> > > AsyncReadBuffers() do the "prepare" work itself directly; why does it need
> > > to be in a callback?
> >
> > One big part of it is "ownership" - while the IO isn't completely "assembled",
> > we can release all buffer pins etc in case of an error. But if the error
> > happens just after the IO was staged, we can't - the buffer is still
> > referenced by the IO. For that the AIO subystem needs to take its own pins
> > etc.  Initially the prepare callback didn't exist, the code in
> > AsyncReadBuffers() was a lot more complicated before it.
> >
> >
> > > I assume it's somehow related to error handling, but I didn't quite get
> > > it. Perhaps an "abort" callback that'd be called on error, instead of a
> > > "prepare" callback, would be better?
> >
> > I don't think an error callback would be helpful - the whole thing is that we
> > basically need claim ownership of all IO related resources IFF the IO is
> > staged. Not before (because then the IO not getting staged would mean we have
> > a resource leak), not after (because we might error out and thus not keep
> > e.g. buffers pinned).
>
> Hmm. The comments say that when you call smgrstartreadv(), the IO handle may
> no longer be modified, as the IO may be executed immediately. What if we
> changed that so that it never submits the IO, only adds the necessary
> callbacks to it?

> In that world, when smgrstartreadv() returns, the necessary details and
> completion callbacks have been set in the IO handle, but the caller can
> still do more preparation before the IO is submitted. The caller must ensure
> that it gets submitted, however, so no erroring out in that state.
>
> Currently the call stack looks like this:
>
> AsyncReadBuffers()
> -> smgrstartreadv()
>   -> mdstartreadv()
>     -> FileStartReadV()
>       -> pgaio_io_prep_readv()
>         -> shared_buffer_readv_prepare() (callback)
>         <- (return)
>       <- (return)
>     <- (return)
>   <- (return)
> <- (return)
>
> I'm thinking that the prepare work is done "on the way up" instead:
>
> AsyncReadBuffers()
> -> smgrstartreadv()
>   -> mdstartreadv()
>     -> FileStartReadV()
>       -> pgaio_io_prep_readv()
>       <- (return)
>     <- (return)
>   <- (return)
> -> shared_buffer_readv_prepare()
> <- (return)
>
> Attached is a patch to demonstrate concretely what I mean.

I think this would be somewhat limiting. Right now it's indeed just bufmgr.c
that needs to do a preparation (or "moving of ownership") step - but I don't
think it's necessarily going to stay that way.

Consider e.g. a hypothetical threaded future in which we have refcounted file
descriptors. While AIO is ongoing, the AIO subsystem would need to ensure that
the FD refcount is increased, otherwise you'd obviously run into trouble if
the issuing backend errored out and released its own reference as part of
resowner release.

I don't think the approach you suggest above would scale well for such a
situation - shared_buffer_readv_prepare() would again need to call to
smgr->md->fd.  Whereas with the current approach md.c (or fd.c?) could just
define its own prepare callback that increased the refcount at the right
moment.


There's a few other scenarios I can think of:

- If somebody were - no idea what made me think of that - to write an smgr
  implementation where storage is accessed over the network, one might need to
  keep network buffers and sockets alive for the duration of the IO.

- It'd be rather useful to have support for asynchronously extending a
  relation, that often requires filesystem journal IO and thus is slow. If
  you're bulk loading, or the extension lock is contented, it'd be great if we
  could start the next relation extension *before* it's needed and the
  extension has to happen synchronously.  To avoid deadlocks, such an
  asynchronous extension would need to be able to release the lock in any
  other backend, just like it's needed for the content locks when
  asynchronously writing.  Which in turn would require transferring ownership
  of the relevant buffers *and* the extension lock.  You could mash this
  together, but it seems like a separate callback woul make it more
  composable.


Does that make any sense to you?


> This adds a new pgaio_io_stage() step to the issuer, and the issuer needs to
> call the prepare functions explicitly, instead of having them as callbacks.
> Nominally that's more steps, but IMHO it's better to be explicit. The same
> actions were happening previously too, it was just hidden in the callback. I
> updated the README to show that too.
>
> I'm not wedded to this, but it feels a little better to me.

Right now the current approach seems to make more sense to me, but I'll think
about it more. I might also have missed something with my theorizing above.


Greetings,

Andres Freund

  36. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-09T00:26:36Z

    Hi,

On 2025-01-07 14:59:58 -0500, Robert Haas wrote:
> On Tue, Jan 7, 2025 at 11:11 AM Andres Freund <andres@anarazel.de> wrote:
> > The difference between a handle and a reference is useful right now, to have
> > some separation between the functions that can be called by anyone (taking a
> > PgAioHandleRef) and only by the issuer (PgAioHandle). That might better be
> > solved by having a PgAioHandleIssuerRef ref or something.
>
> To me, those names don't convey that.

I'm certainly not wedded to these names - I went back and forth between
different names a fair bit, because I wasn't quite happy. I am however certain
that the current names are better than what it used to be (PgAioInProgress and
because that's long, a bunch of PgAioIP* names) :)


To make sure were talking about the same things, I am thinking of the
following "entities" needing names:


1) Shared memory representation of an IO, for the AIO subsystem internally

   Currently: PgAioHandle

   Because shared memory is limited, we need to reuse this entity. This reuse
   needs to be possible "immediately" after completion, to avoid a bunch of
   nasty scenarios.

   To distinguish a reused PgAioHandle from its "prior" incarnation, each
   PgAioHandle has a 64bit "generation counter.

   In addition to being referenceable via pointer, it's also possible to
   assign a 32bit integer to each PgAioHandle, as there is a fixed number of
   them.


2) A way for the issuer of an IO to reference 1), to attach information to the
   IO

   Currently: PgAioHandle*

   As long as the issuer hasn't yet staged the IO, it can't be
   reused. Therefore it's OK to just point to the PgAioHandle.

   One disadvantage of just using a pointer to PgAioHandle* is that it's
   harder to distinguish subystem-internal functions that accept PgAioHandle*
   from "public" functions that accept the "issuer reference".


3) A way for any backend to wait for a specific IO to complete

   Currently: PgAioHandleRef

   This references 1) using a 32 bit ID and the 64bit generation.

   This is used to allow any backend to wait for a specific IO to
   complete. E.g. by including it in the BufferDesc so that WaitIO can wait
   for it.

   Because it includes the generation it's trivial to detect whether the
   PgAioHandle was reused.



> I would perhaps call the thing that supports issuer-only operations a
> "PgAio" and the thing other people can use a "PgAioHandle". Or
> "PgAioRequest" and "PgAioHandle" or something like that. With
> PgAioHandleRef, IMHO you've got two words that both imply a layer of
> indirection -- "handle" and "ref" -- which doesn't seem quite as nice,
> because then the other thing -- "PgAioHandle" still sort of implies one
> layer of indirection and the whole thing seems a bit less clear.

It's indirections all the way down. The PG representation of "one IO" in the
end is just an indirection for a kernel operation :)


I would like to split 1) and 2) above.

1) PgAio{Handle,Request,} (a large struct)  - used internally by AIO subsystem,
   "pointed to" by the following
2) PgAioIssuerRef (an ID or pointer) - used by the issuer to incrementally
   define the IO
3) PgAioWaitRef - (an ID and generation) - used to wait for a specific IO to
   complete, not affected by reuse of PgAioHandle





> > > REAPED feels like a bad name. It sounds like a later stage than COMPLETED,
> > > but it's actually vice versa.
> >
> > What would you call having gotten "completion notifications" from the kernel,
> > but not having processed them?
>
> The Linux kernel calls those zombie processes, so we could call it a ZOMBIE
> state, but that seems like it might be a bit of inside baseball.

ZOMBIE feels even later than REAPED to me :)


> I do agree with Heikki that REAPED sounds later than COMPLETED, because you
> reap zombie processes by collecting their exit status. Maybe you could have
> AHS_COMPLETE or AHS_IO_COMPLETE for the state where the I/O is done but
> there's still completion-related work to be done, and then the other state
> could be AHS_DONE or AHS_FINISHED or AHS_FINAL or AHS_REAPED or something.

How about

AHS_COMPLETE_KERNEL or AHS_COMPLETE_RAW - raw syscall completed
AHS_COMPLETE_SHARED_CB - shared callback completed
AHS_COMPLETE_LOCAL_CB - local callback completed

?

Greetings,

Andres Freund

  37. Re: AIO v2.0

    Ants Aasma <ants.aasma@cybertec.at> — 2025-01-09T08:59:22Z

    On Wed, 8 Jan 2025 at 22:58, Andres Freund <andres@anarazel.de> wrote:
> master:                 ~18 GB/s
> patch, buffered:        ~20 GB/s
> patch, direct, worker:  ~28 GB/s
> patch, direct, uring:   ~35 GB/s
>
>
> This was with io_workers=32, io_max_concurrency=128,
> effective_io_concurrency=1000 (doesn't need to be that high, but it's what I
> still have the numbers for).
>
>
> This was without data checksums enabled as otherwise the checksum code becomes
> a *huge* bottleneck.

I'm curious about this because the checksum code should be fast enough
to easily handle that throughput. I remember checksum overhead being
negligible even when pulling in pages from page cache. Is it just that
the calculation is slow, or is it the fact that checksumming needs to
bring the page into the CPU cache. Did you notice any hints which
might be the case? I don't really have a machine at hand that can do
anywhere close to this amount of I/O.

I'm asking because if it's the calculation that is slow then it seems
like it's time to compile different ISA extension variants of the
checksum code and select the best one at runtime.

-- 
Ants Aasma

  38. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2025-01-09T16:25:39Z

    Hi,

On 2025-01-09 10:59:22 +0200, Ants Aasma wrote:
> On Wed, 8 Jan 2025 at 22:58, Andres Freund <andres@anarazel.de> wrote:
> > master:                 ~18 GB/s
> > patch, buffered:        ~20 GB/s
> > patch, direct, worker:  ~28 GB/s
> > patch, direct, uring:   ~35 GB/s
> >
> >
> > This was with io_workers=32, io_max_concurrency=128,
> > effective_io_concurrency=1000 (doesn't need to be that high, but it's what I
> > still have the numbers for).
> >
> >
> > This was without data checksums enabled as otherwise the checksum code becomes
> > a *huge* bottleneck.
> 
> I'm curious about this because the checksum code should be fast enough
> to easily handle that throughput.

It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
workstation. But we don't have a good ready-made way of testing that without
also doing IO, so it's kinda hard to say.


> I remember checksum overhead being negligible even when pulling in pages
> from page cache.

It's indeed much less of an issue when pulling pages from the page cache, as
the copy from the page cache is fairly slow. With direct-IO, where the copy
from the page cache isn't the main driver of CPU use anymore, it becomes much
clearer.

Even with buffered IO it became a bigger issue with 17, due to
io_combine_limit. It turns out that lots of tiny syscalls are slow, so the
peak throughput that could reach the checksumming code was lower.


I created a 21554MB relation and measured the time to do a pg_prewarm() of
that relation after evicting all of shared buffers (the first time buffers are
touched has a bit different perf characteristics). I am using direct IO and
io_uring here, as buffered IO would include the page cache copy cost and
worker mode could parallelize the checksum computation on reads.  The checksum
cost is a bigger issue for writes than reads, but it's harder to quickly
generate enough dirty data for a repeatable benchmark.

This system can do about 12.5GB/s of read IO.


Just to show the effect of the read size on page cache copy performance:

config                                                  checksums	time in ms
buffered io_engine=sync io_combine_limit=1              0               6712.153
buffered io_engine=sync io_combine_limit=2              0               5919.215
buffered io_engine=sync io_combine_limit=4              0               5738.496
buffered io_engine=sync io_combine_limit=8              0               5396.415
buffered io_engine=sync io_combine_limit=16             0               5312.803
buffered io_engine=sync io_combine_limit=32             0               5275.389


To see the effect of page cache copy overhead:

config                                                  checksums	time in ms
buffered io_engine=io_uring                             0               3901.625
direct io_engine=io_uring                               0               2075.330


Now to show the effect of checksums (enabled/disabled with pg_checksums):

config                                                  checksums	time in ms
buffered io_engine=io_uring                             0               3883.127
buffered io_engine=io_uring                             1               5880.892
direct io_engine=io_uring                               0               2067.142
direct io_engine=io_uring                               1               3835.968

So with direct + uring w/o checksums, we can reach 10427 MB/s (close-ish to
disk speed), but with checksums we only reach 5620 MB/s.


> Is it just that the calculation is slow, or is it the fact that checksumming
> needs to bring the page into the CPU cache. Did you notice any hints which
> might be the case?

I don't think the issue is that checksumming pulls the data into CPU caches

1) This is visible with SELECT that actually uses the data

2) I added prefetching to avoid any meaningful amount of cache misses and it
   doesn't change the overall timing much

3) It's visible with buffered IO, which has pulled the data into CPU caches
   already


> I don't really have a machine at hand that can do anywhere close to this
> amount of I/O.

It's visible even when pulling from the page cache, if to a somewhat lesser
degree.

I wonder if it's worth adding a test function that computes checksums of all
shared buffers in memory already. That'd allow exercising the checksum code in
a realistic context (i.e. buffer locking etc preventing some out-of-order
effects, using 8kB chunks etc) without also needing to involve the IO path.


> I'm asking because if it's the calculation that is slow then it seems
> like it's time to compile different ISA extension variants of the
> checksum code and select the best one at runtime.

You think it's ISA specific? I don't see a significant effect of compiling
with -march=native or not - and that should suffice to make the checksum code
built with sufficiently high ISA support, right?


FWIW CPU profiles show all the time being spent in the "main checksum
calculation" loop:

 Percent |	Source code & Disassembly of postgres for cycles:P (5866 samples, percent: local period)
--------------------------------------------------------------------------------------------------------
         :
         :
         :
         : 3      Disassembly of section .text:
         :
         : 5      00000000009e3670 <pg_checksum_page>:
         : 6      * calculation isn't affected by the old checksum stored on the page.
         : 7      * Restore it after, because actually updating the checksum is NOT part of
         : 8      * the API of this function.
         : 9      */
         : 10     save_checksum = cpage->phdr.pd_checksum;
         : 11     cpage->phdr.pd_checksum = 0;
    0.00 :   9e3670: xor    %eax,%eax
         : 13     CHECKSUM_COMP(sums[j], page->data[i][j]);
    0.00 :   9e3672: mov    $0x1000193,%r8d
         : 15     cpage->phdr.pd_checksum = 0;
    0.00 :   9e3678: vmovdqa -0x693fa0(%rip),%ymm3        # 34f6e0 <.LC0>
    0.05 :   9e3680: vmovdqa -0x6935c8(%rip),%ymm4        # 3500c0 <.LC1>
    0.00 :   9e3688: vmovdqa -0x693c10(%rip),%ymm0        # 34fa80 <.LC2>
    0.00 :   9e3690: vmovdqa -0x693598(%rip),%ymm1        # 350100 <.LC3>
         : 20     {
    0.00 :   9e3698: mov    %esi,%ecx
    0.02 :   9e369a: lea    0x2000(%rdi),%rdx
         : 23     save_checksum = cpage->phdr.pd_checksum;
    0.00 :   9e36a1: movzwl 0x8(%rdi),%esi
         : 25     CHECKSUM_COMP(sums[j], page->data[i][j]);
    0.00 :   9e36a5: vpbroadcastd %r8d,%ymm5
         : 27     cpage->phdr.pd_checksum = 0;
    0.00 :   9e36ab: mov    %ax,0x8(%rdi)
         : 29     for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
    0.14 :   9e36af: mov    %rdi,%rax
    0.00 :   9e36b2: nopw   0x0(%rax,%rax,1)
         : 32     CHECKSUM_COMP(sums[j], page->data[i][j]);
   15.36 :   9e36b8: vpxord (%rax),%ymm1,%ymm1
    4.79 :   9e36be: vmovdqu 0x80(%rax),%ymm2
         : 35     for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
    0.07 :   9e36c6: add    $0x100,%rax
         : 37     CHECKSUM_COMP(sums[j], page->data[i][j]);
    2.45 :   9e36cc: vpxord -0xe0(%rax),%ymm0,%ymm0
    2.85 :   9e36d3: vpmulld %ymm5,%ymm1,%ymm6
    0.02 :   9e36d8: vpsrld $0x11,%ymm1,%ymm1
    3.17 :   9e36dd: vpternlogd $0x96,%ymm6,%ymm1,%ymm2
    2.01 :   9e36e4: vpmulld %ymm5,%ymm0,%ymm6
   13.16 :   9e36e9: vpmulld %ymm5,%ymm2,%ymm1
    0.03 :   9e36ee: vpsrld $0x11,%ymm2,%ymm2
    0.02 :   9e36f3: vpsrld $0x11,%ymm0,%ymm0
    2.57 :   9e36f8: vpxord %ymm2,%ymm1,%ymm1
    0.89 :   9e36fe: vmovdqu -0x60(%rax),%ymm2
    0.12 :   9e3703: vpternlogd $0x96,%ymm6,%ymm0,%ymm2
    4.48 :   9e370a: vpmulld %ymm5,%ymm2,%ymm0
    0.49 :   9e370f: vpsrld $0x11,%ymm2,%ymm2
    0.99 :   9e3714: vpxord %ymm2,%ymm0,%ymm0
   11.88 :   9e371a: vpxord -0xc0(%rax),%ymm4,%ymm2
    2.80 :   9e3721: vpmulld %ymm5,%ymm2,%ymm6
    0.68 :   9e3726: vpsrld $0x11,%ymm2,%ymm4
    4.94 :   9e372b: vmovdqu -0x40(%rax),%ymm2
    1.45 :   9e3730: vpternlogd $0x96,%ymm6,%ymm4,%ymm2
    8.63 :   9e3737: vpmulld %ymm5,%ymm2,%ymm4
    0.17 :   9e373c: vpsrld $0x11,%ymm2,%ymm2
    1.81 :   9e3741: vpxord %ymm2,%ymm4,%ymm4
    0.10 :   9e3747: vpxord -0xa0(%rax),%ymm3,%ymm2
    0.70 :   9e374e: vpmulld %ymm5,%ymm2,%ymm6
    1.65 :   9e3753: vpsrld $0x11,%ymm2,%ymm3
    0.03 :   9e3758: vmovdqu -0x20(%rax),%ymm2
    0.85 :   9e375d: vpternlogd $0x96,%ymm6,%ymm3,%ymm2
    3.73 :   9e3764: vpmulld %ymm5,%ymm2,%ymm3
    0.07 :   9e3769: vpsrld $0x11,%ymm2,%ymm2
    1.48 :   9e376e: vpxord %ymm2,%ymm3,%ymm3
         : 68     for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
    0.02 :   9e3774: cmp    %rax,%rdx
    2.32 :   9e3777: jne    9e36b8 <pg_checksum_page+0x48>
         : 71     CHECKSUM_COMP(sums[j], 0);
    0.17 :   9e377d: vpmulld %ymm5,%ymm0,%ymm7
    0.07 :   9e3782: vpmulld %ymm5,%ymm1,%ymm6
         : 74     checksum = pg_checksum_block(cpage);
         : 75     cpage->phdr.pd_checksum = save_checksum;
    0.00 :   9e3787: mov    %si,0x8(%rdi)
         : 77     CHECKSUM_COMP(sums[j], 0);
    0.02 :   9e378b: vpsrld $0x11,%ymm0,%ymm0
    0.02 :   9e3790: vpsrld $0x11,%ymm1,%ymm1
    0.02 :   9e3795: vpsrld $0x11,%ymm4,%ymm2
    0.00 :   9e379a: vpxord %ymm0,%ymm7,%ymm7
    0.10 :   9e37a0: vpmulld %ymm5,%ymm4,%ymm0
    0.00 :   9e37a5: vpxord %ymm1,%ymm6,%ymm6
    0.17 :   9e37ab: vpmulld %ymm5,%ymm3,%ymm1
    0.19 :   9e37b0: vpmulld %ymm5,%ymm6,%ymm4
    0.00 :   9e37b5: vpsrld $0x11,%ymm6,%ymm6
    0.02 :   9e37ba: vpxord %ymm2,%ymm0,%ymm0
    0.00 :   9e37c0: vpsrld $0x11,%ymm3,%ymm2
    0.22 :   9e37c5: vpmulld %ymm5,%ymm7,%ymm3
    0.02 :   9e37ca: vpsrld $0x11,%ymm7,%ymm7
    0.00 :   9e37cf: vpxord %ymm2,%ymm1,%ymm1
    0.03 :   9e37d5: vpsrld $0x11,%ymm0,%ymm2
    0.15 :   9e37da: vpmulld %ymm5,%ymm0,%ymm0
         : 94     result ^= sums[i];
    0.00 :   9e37df: vpternlogd $0x96,%ymm3,%ymm7,%ymm2
         : 96     CHECKSUM_COMP(sums[j], 0);
    0.05 :   9e37e6: vpsrld $0x11,%ymm1,%ymm3
    0.19 :   9e37eb: vpmulld %ymm5,%ymm1,%ymm1
         : 99     result ^= sums[i];
    0.02 :   9e37f0: vpternlogd $0x96,%ymm4,%ymm6,%ymm0
    0.10 :   9e37f7: vpxord %ymm1,%ymm0,%ymm0
    0.07 :   9e37fd: vpternlogd $0x96,%ymm2,%ymm3,%ymm0
    0.15 :   9e3804: vextracti32x4 $0x1,%ymm0,%xmm1
    0.03 :   9e380b: vpxord %xmm0,%xmm1,%xmm0
    0.14 :   9e3811: vpsrldq $0x8,%xmm0,%xmm1
    0.12 :   9e3816: vpxord %xmm1,%xmm0,%xmm0
    0.09 :   9e381c: vpsrldq $0x4,%xmm0,%xmm1
    0.12 :   9e3821: vpxord %xmm1,%xmm0,%xmm0
    0.05 :   9e3827: vmovd  %xmm0,%eax
         :
         : 111    /* Mix in the block number to detect transposed pages */
         : 112    checksum ^= blkno;
    0.07 :   9e382b: xor    %ecx,%eax
         :
         : 115    /*
         : 116    * Reduce to a uint16 (to fit in the pd_checksum field) with an offset of
         : 117    * one. That avoids checksums of zero, which seems like a good idea.
         : 118    */
         : 119    return (uint16) ((checksum % 65535) + 1);
    0.00 :   9e382d: mov    $0x80008001,%ecx
    0.03 :   9e3832: mov    %eax,%edx
    0.27 :   9e3834: imul   %rcx,%rdx
    0.09 :   9e3838: shr    $0x2f,%rdx
    0.07 :   9e383c: lea    0x1(%rax,%rdx,1),%eax
    0.00 :   9e3840: vzeroupper
         : 126    }
    0.15 :   9e3843: ret


I did briefly experiment with changing N_SUMS. 16 is substantially worse, 64
seems to be about the same as 32.

Greetings,

Andres Freund

  39. Re: AIO v2.0

    Ants Aasma <ants.aasma@cybertec.at> — 2025-01-09T18:10:24Z

    On Thu, 9 Jan 2025 at 18:25, Andres Freund <andres@anarazel.de> wrote:
> > I'm curious about this because the checksum code should be fast enough
> > to easily handle that throughput.
>
> It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
> workstation. But we don't have a good ready-made way of testing that without
> also doing IO, so it's kinda hard to say.

Interesting, I wonder if it's related to Intel increasing vpmulld
latency to 10 already back in Haswell. The Zen 3 I'm testing on has
latency 3 and has twice the throughput.

Attached is a naive and crude benchmark that I used for testing here.
Compiled with:

gcc -O2 -funroll-loops -ftree-vectorize -march=native \
  -I$(pg_config --includedir-server) \
  bench-checksums.c -o bench-checksums-native

Just fills up an array of pages and checksums them, first argument is
number of checksums, second is array size. I used 1M checksums and 100
pages for in cache behavior and 100000 pages for in memory
performance.

869.85927ms @ 9.418 GB/s - generic from memory
772.12252ms @ 10.610 GB/s - generic in cache
442.61869ms @ 18.508 GB/s - native from memory
137.07573ms @ 59.763 GB/s - native in cache

> > Is it just that the calculation is slow, or is it the fact that checksumming
> > needs to bring the page into the CPU cache. Did you notice any hints which
> > might be the case?
>
> I don't think the issue is that checksumming pulls the data into CPU caches
>
> 1) This is visible with SELECT that actually uses the data
>
> 2) I added prefetching to avoid any meaningful amount of cache misses and it
>    doesn't change the overall timing much
>
> 3) It's visible with buffered IO, which has pulled the data into CPU caches
>    already

I didn't yet check the code, when doing aio completions checksumming
be running on the same core as is going to be using the page?

It could also be that for some reason the checksumming is creating
extra bandwidth on memory bus or CPU internal rings, which due to the
already high amount of data already flying around causes contention.

> > I don't really have a machine at hand that can do anywhere close to this
> > amount of I/O.
>
> It's visible even when pulling from the page cache, if to a somewhat lesser
> degree.

Good point, I'll see if I can reproduce.

> I wonder if it's worth adding a test function that computes checksums of all
> shared buffers in memory already. That'd allow exercising the checksum code in
> a realistic context (i.e. buffer locking etc preventing some out-of-order
> effects, using 8kB chunks etc) without also needing to involve the IO path.

OoO shouldn't matter that much, over here even in the best case it's
still taking 500+ cycles per iteration.

> > I'm asking because if it's the calculation that is slow then it seems
> > like it's time to compile different ISA extension variants of the
> > checksum code and select the best one at runtime.
>
> You think it's ISA specific? I don't see a significant effect of compiling
> with -march=native or not - and that should suffice to make the checksum code
> built with sufficiently high ISA support, right?

Right, the disassembly below looked very good.

> FWIW CPU profiles show all the time being spent in the "main checksum
> calculation" loop:

.. disassembly omitted for brevity

Not sure if it's applicable here or not due to microarch differences.
But in my case when bounded by memory bandwidth the main loop events
were clustered around a few instructions like it was in here, whereas
when running from cache all instructions were about equally
represented.

> I did briefly experiment with changing N_SUMS. 16 is substantially worse, 64
> seems to be about the same as 32.

This suggests that mulld latency is not the culprit.

Regards,
Ants

  40. Re: AIO v2.0

    Andres Freund <andres@anarazel.de> — 2025-01-09T20:53:20Z

    Hi,

On 2025-01-09 20:10:24 +0200, Ants Aasma wrote:
> On Thu, 9 Jan 2025 at 18:25, Andres Freund <andres@anarazel.de> wrote:
> > > I'm curious about this because the checksum code should be fast enough
> > > to easily handle that throughput.
> >
> > It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
> > workstation. But we don't have a good ready-made way of testing that without
> > also doing IO, so it's kinda hard to say.
>
> Interesting, I wonder if it's related to Intel increasing vpmulld
> latency to 10 already back in Haswell. The Zen 3 I'm testing on has
> latency 3 and has twice the throughput.

> Attached is a naive and crude benchmark that I used for testing here.
> Compiled with:
>
> gcc -O2 -funroll-loops -ftree-vectorize -march=native \
>   -I$(pg_config --includedir-server) \
>   bench-checksums.c -o bench-checksums-native
>
> Just fills up an array of pages and checksums them, first argument is
> number of checksums, second is array size. I used 1M checksums and 100
> pages for in cache behavior and 100000 pages for in memory
> performance.
>
> 869.85927ms @ 9.418 GB/s - generic from memory
> 772.12252ms @ 10.610 GB/s - generic in cache
> 442.61869ms @ 18.508 GB/s - native from memory
> 137.07573ms @ 59.763 GB/s - native in cache

printf '%16s\t%16s\t%s\n' march mem result; for mem in 100 100000 1000000; do for march in x86-64 x86-64-v2 x86-64-v3 x86-64-v4 native; do printf "%16s\t%16s\t" $march $mem; gcc -g -g3 -O2 -funroll-loops -ftree-vectorize -march=$march -I ~/src/postgresql/src/include/ -I src/include/ /tmp/bench-checksums.c -o bench-checksums-native && numactl --physcpubind 1 --membind 0 ./bench-checksums-native 1000000 $mem;done; done

Workstation w/ 2x Xeon Gold 6442Y:

           march	             mem	result
          x86-64	             100	731.87779ms @ 11.193 GB/s
       x86-64-v2	             100	327.18580ms @ 25.038 GB/s
       x86-64-v3	             100	264.03547ms @ 31.026 GB/s
       x86-64-v4	             100	282.08065ms @ 29.041 GB/s
          native	             100	246.13766ms @ 33.282 GB/s
          x86-64	          100000	842.66827ms @ 9.722 GB/s
       x86-64-v2	          100000	604.52959ms @ 13.551 GB/s
       x86-64-v3	          100000	477.16239ms @ 17.168 GB/s
       x86-64-v4	          100000	476.07039ms @ 17.208 GB/s
          native	          100000	456.08080ms @ 17.962 GB/s
          x86-64	         1000000	845.51132ms @ 9.689 GB/s
       x86-64-v2	         1000000	612.07973ms @ 13.384 GB/s
       x86-64-v3	         1000000	485.23738ms @ 16.882 GB/s
       x86-64-v4	         1000000	483.86411ms @ 16.930 GB/s
          native	         1000000	462.88461ms @ 17.698 GB/s



Zen 4 laptop (AMD Ryzen 7 PRO 7840U):
           march	             mem	result
          x86-64	             100	417.19762ms @ 19.636 GB/s
       x86-64-v2	             100	130.67596ms @ 62.689 GB/s
       x86-64-v3	             100	97.07758ms @ 84.386 GB/s
       x86-64-v4	             100	95.67704ms @ 85.621 GB/s
          native	             100	95.15734ms @ 86.089 GB/s
          x86-64	          100000	431.38370ms @ 18.990 GB/s
       x86-64-v2	          100000	215.74856ms @ 37.970 GB/s
       x86-64-v3	          100000	199.74492ms @ 41.012 GB/s
       x86-64-v4	          100000	186.98300ms @ 43.811 GB/s
          native	          100000	187.68125ms @ 43.648 GB/s
          x86-64	         1000000	433.87893ms @ 18.881 GB/s
       x86-64-v2	         1000000	217.46561ms @ 37.670 GB/s
       x86-64-v3	         1000000	200.40667ms @ 40.877 GB/s
       x86-64-v4	         1000000	187.51978ms @ 43.686 GB/s
          native	         1000000	190.29273ms @ 43.049 GB/s


Workstation w/ 2x Xeon Gold 5215:
           march	             mem	result
          x86-64	             100	780.38881ms @ 10.497 GB/s
       x86-64-v2	             100	389.62005ms @ 21.026 GB/s
       x86-64-v3	             100	323.97294ms @ 25.286 GB/s
       x86-64-v4	             100	274.19493ms @ 29.877 GB/s
          native	             100	283.48674ms @ 28.897 GB/s
          x86-64	          100000	1112.63898ms @ 7.363 GB/s
       x86-64-v2	          100000	831.45641ms @ 9.853 GB/s
       x86-64-v3	          100000	696.20789ms @ 11.767 GB/s
       x86-64-v4	          100000	685.61636ms @ 11.948 GB/s
          native	          100000	689.78023ms @ 11.876 GB/s
          x86-64	         1000000	1128.65580ms @ 7.258 GB/s
       x86-64-v2	         1000000	843.92594ms @ 9.707 GB/s
       x86-64-v3	         1000000	718.78848ms @ 11.397 GB/s
       x86-64-v4	         1000000	687.68258ms @ 11.912 GB/s
          native	         1000000	705.34731ms @ 11.614 GB/s


That's quite the drastic difference between amd and intel. Of course it's also
comparing a multi-core server uarch (lower per-core bandwidth, much higher
aggregate bandwidth) with a client uarch.


The difference between the baseline CPU target and a more modern profile is
also rather impressive.  Looks like some cpu-capability based dispatch would
likely be worth it, even if it didn't matter in my case due to -march=native.


I just realized that

a) The meson build doesn't use the relevant flags for bufpage.c - it didn't
   matter in my numbers though because I was building with -O3 and
   march=native.

   This clearly ought to be fixed.

b) Neither build uses the optimized flags for pg_checksum and pg_upgrade, both
   of which include checksum_imp.h directly.

   This probably should be fixed too - perhaps by building the relevant code
   once as part of fe_utils or such?


It probably matters less than it used to - these days -O2 turns on
-ftree-loop-vectorize -ftree-slp-vectorize. But loop unrolling isn't
enabled.

I do see a perf difference at -O2 between using/not using
-funroll-loops. Interestingly not at -O3, despite -funroll-loops not actually
being enabled by -O3. I think the relevant option that *is* turned on by O3 is
-fpeel-loops.

Here's a comparison of different flags run the 6442Y

printf '%16s\t%32s\t%16s\t%s\n' march flags mem result; for mem in 100 100000; do for march in x86-64 x86-64-v2 x86-64-v3 x86-64-v4 native; do for flags in "-O2" "-O2 -funroll-loops" "-O3" "-O3 -funroll-loops"; do printf "%16s\t%32s\t%16s\t" "$march" "$flags" "$mem"; gcc $flags -march=$march -I ~/src/postgresql/src/include/ -I src/include/ /tmp/bench-checksums.c -o bench-checksums-native && numactl --physcpubind 3 --membind 0 ./bench-checksums-native 3000000 $mem;done; done;done
           march	                           flags	             mem	result
          x86-64	                             -O2	             100	2280.86253ms @ 10.775 GB/s
          x86-64	              -O2 -funroll-loops	             100	2195.66942ms @ 11.193 GB/s
          x86-64	                             -O3	             100	2422.57588ms @ 10.145 GB/s
          x86-64	              -O3 -funroll-loops	             100	2243.75826ms @ 10.953 GB/s
       x86-64-v2	                             -O2	             100	1243.68063ms @ 19.761 GB/s
       x86-64-v2	              -O2 -funroll-loops	             100	979.67783ms @ 25.086 GB/s
       x86-64-v2	                             -O3	             100	988.80296ms @ 24.854 GB/s
       x86-64-v2	              -O3 -funroll-loops	             100	991.31632ms @ 24.791 GB/s
       x86-64-v3	                             -O2	             100	1146.90165ms @ 21.428 GB/s
       x86-64-v3	              -O2 -funroll-loops	             100	785.81395ms @ 31.275 GB/s
       x86-64-v3	                             -O3	             100	800.53627ms @ 30.699 GB/s
       x86-64-v3	              -O3 -funroll-loops	             100	790.21230ms @ 31.101 GB/s
       x86-64-v4	                             -O2	             100	883.82916ms @ 27.806 GB/s
       x86-64-v4	              -O2 -funroll-loops	             100	831.55372ms @ 29.554 GB/s
       x86-64-v4	                             -O3	             100	843.23141ms @ 29.145 GB/s
       x86-64-v4	              -O3 -funroll-loops	             100	821.19969ms @ 29.927 GB/s
          native	                             -O2	             100	1197.41357ms @ 20.524 GB/s
          native	              -O2 -funroll-loops	             100	718.05253ms @ 34.226 GB/s
          native	                             -O3	             100	747.94090ms @ 32.858 GB/s
          native	              -O3 -funroll-loops	             100	751.52379ms @ 32.702 GB/s
          x86-64	                             -O2	          100000	2911.47087ms @ 8.441 GB/s
          x86-64	              -O2 -funroll-loops	          100000	2525.45504ms @ 9.731 GB/s
          x86-64	                             -O3	          100000	2497.42016ms @ 9.841 GB/s
          x86-64	              -O3 -funroll-loops	          100000	2346.33551ms @ 10.474 GB/s
       x86-64-v2	                             -O2	          100000	2124.10102ms @ 11.570 GB/s
       x86-64-v2	              -O2 -funroll-loops	          100000	1819.09659ms @ 13.510 GB/s
       x86-64-v2	                             -O3	          100000	1613.45823ms @ 15.232 GB/s
       x86-64-v2	              -O3 -funroll-loops	          100000	1607.09245ms @ 15.292 GB/s
       x86-64-v3	                             -O2	          100000	1972.89390ms @ 12.457 GB/s
       x86-64-v3	              -O2 -funroll-loops	          100000	1432.58229ms @ 17.155 GB/s
       x86-64-v3	                             -O3	          100000	1533.18003ms @ 16.029 GB/s
       x86-64-v3	              -O3 -funroll-loops	          100000	1539.39779ms @ 15.965 GB/s
       x86-64-v4	                             -O2	          100000	1591.96881ms @ 15.437 GB/s
       x86-64-v4	              -O2 -funroll-loops	          100000	1434.91828ms @ 17.127 GB/s
       x86-64-v4	                             -O3	          100000	1454.30133ms @ 16.899 GB/s
       x86-64-v4	              -O3 -funroll-loops	          100000	1429.13733ms @ 17.196 GB/s
          native	                             -O2	          100000	1980.53734ms @ 12.409 GB/s
          native	              -O2 -funroll-loops	          100000	1373.95337ms @ 17.887 GB/s
          native	                             -O3	          100000	1517.90164ms @ 16.191 GB/s
          native	              -O3 -funroll-loops	          100000	1508.37021ms @ 16.293 GB/s



> > > Is it just that the calculation is slow, or is it the fact that checksumming
> > > needs to bring the page into the CPU cache. Did you notice any hints which
> > > might be the case?
> >
> > I don't think the issue is that checksumming pulls the data into CPU caches
> >
> > 1) This is visible with SELECT that actually uses the data
> >
> > 2) I added prefetching to avoid any meaningful amount of cache misses and it
> >    doesn't change the overall timing much
> >
> > 3) It's visible with buffered IO, which has pulled the data into CPU caches
> >    already
>
> I didn't yet check the code, when doing aio completions checksumming
> be running on the same core as is going to be using the page?

With io_uring normally yes, the exception being that another backend that
needs the same page could end up running the completion.

With worker mode normally no.

Greetings,

Andres Freund

  41. Re: AIO v2.0

    Ants Aasma <ants.aasma@cybertec.at> — 2025-01-10T10:33:39Z

    On Thu, 9 Jan 2025 at 22:53, Andres Freund <andres@anarazel.de> wrote:

<Edited to highlight interesting numbers>

> Workstation w/ 2x Xeon Gold 6442Y:
>
>            march                     mem        result
>           native                     100        246.13766ms @ 33.282 GB/s
>           native                  100000        456.08080ms @ 17.962 GB/s
>
> Zen 4 laptop (AMD Ryzen 7 PRO 7840U):
>            march                     mem        result
>           native                     100        95.15734ms @ 86.089 GB/s
>           native                  100000        187.68125ms @ 43.648 GB/s
>
> Workstation w/ 2x Xeon Gold 5215:
>            march                     mem        result
>           native                     100        283.48674ms @ 28.897 GB/s
>           native                  100000        689.78023ms @ 11.876 GB/s
>
> That's quite the drastic difference between amd and intel. Of course it's also
> comparing a multi-core server uarch (lower per-core bandwidth, much higher
> aggregate bandwidth) with a client uarch.

In hindsight building the hash around mulld primitive was a bad decision
because Intel for whatever reason decided to kill the performance of it:

vpmulld          latency throughput
                         (values/cycle)
Sandy Bridge     5       4
Alder Lake      10       8
Zen 4            3       16
Zen 5            3       32

Most top performing hashes these days seem to be built around AES
instructions.

But I was curious why there is such a difference in streaming results.
Turns out that for whatever reason one core gets access to much less
bandwidth on Intel than on AMD. [1]

This made me take another look at your previous prewarm numbers. It looks
like performance is suspiciously proportional to the number of copies of
data the CPU has to make:

config                        checksums   time in ms   number of copies
buffered io_engine=io_uring   0           3883.127     2
buffered io_engine=io_uring   1           5880.892     3
direct io_engine=io_uring     0           2067.142     1
direct io_engine=io_uring     1           3835.968     2

To me that feels like there is a bandwidth bottleneck in this workload and
checksumming the page when the contents is not looked at just adds to
consumed bandwidth, bringing down the performance correspondingly.

This doesn't explain why you observed slowdown in the select case, but I
think that might be due to the per-core bandwidth limitation. Both cases
might pull in the same amount of data into the cache, but without checksums
it is spread out over a longer time allowing other work to happen
concurrently.

[1] https://chipsandcheese.com/p/a-peek-at-sapphire-rapids#%C2%A7bandwidth


> The difference between the baseline CPU target and a more modern profile is
> also rather impressive.  Looks like some cpu-capability based dispatch would
> likely be worth it, even if it didn't matter in my case due to -march=native.

Yes, along with using function attributes for the optimization flags to avoid
the build system hacks.

--
Ants

  42. Re: AIO v2.2

    Robert Haas <robertmhaas@gmail.com> — 2025-01-13T20:43:46Z

    On Wed, Jan 8, 2025 at 7:26 PM Andres Freund <andres@anarazel.de> wrote:
> 1) Shared memory representation of an IO, for the AIO subsystem internally
>
>    Currently: PgAioHandle
>
> 2) A way for the issuer of an IO to reference 1), to attach information to the
>    IO
>
>    Currently: PgAioHandle*
>
> 3) A way for any backend to wait for a specific IO to complete
>
>    Currently: PgAioHandleRef

With that additional information, I don't mind this naming too much,
but I still think PgAioHandle -> PgAio and PgAioHandleRef ->
PgAioHandle is worth considering. Compare BackgroundWorkerSlot and
BackgroundWorkerHandle, which suggests PgAioHandle -> PgAioSlot and
PgAioHandleRef -> PgAioHandle.

> ZOMBIE feels even later than REAPED to me :)

Makes logical sense, because you would assume that you die first and
then later become an undead creature, but the UNIX precedent is that
dying turns you into a zombie and someone then has to reap the exit
status for you to be just plain dead. :-)

> > I do agree with Heikki that REAPED sounds later than COMPLETED, because you
> > reap zombie processes by collecting their exit status. Maybe you could have
> > AHS_COMPLETE or AHS_IO_COMPLETE for the state where the I/O is done but
> > there's still completion-related work to be done, and then the other state
> > could be AHS_DONE or AHS_FINISHED or AHS_FINAL or AHS_REAPED or something.
>
> How about
>
> AHS_COMPLETE_KERNEL or AHS_COMPLETE_RAW - raw syscall completed
> AHS_COMPLETE_SHARED_CB - shared callback completed
> AHS_COMPLETE_LOCAL_CB - local callback completed
>
> ?

That's not bad. I like RAW better than KERNEL. I was hoping to use
different works like COMPLETE and DONE rather than, as you did it
here, COMPLETE and COMPLETE, but it's probably fine.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  43. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-01-13T21:46:13Z

    Hi,

On 2025-01-13 15:43:46 -0500, Robert Haas wrote:
> On Wed, Jan 8, 2025 at 7:26 PM Andres Freund <andres@anarazel.de> wrote:
> > 1) Shared memory representation of an IO, for the AIO subsystem internally
> >
> >    Currently: PgAioHandle
> >
> > 2) A way for the issuer of an IO to reference 1), to attach information to the
> >    IO
> >
> >    Currently: PgAioHandle*
> >
> > 3) A way for any backend to wait for a specific IO to complete
> >
> >    Currently: PgAioHandleRef
>
> With that additional information, I don't mind this naming too much,
> but I still think PgAioHandle -> PgAio and PgAioHandleRef ->
> PgAioHandle is worth considering. Compare BackgroundWorkerSlot and
> BackgroundWorkerHandle, which suggests PgAioHandle -> PgAioSlot and
> PgAioHandleRef -> PgAioHandle.

I don't love PgAioHandle -> PgAio as there are other things (e.g. per-backend
state) in the PgAio namespace...



> > > I do agree with Heikki that REAPED sounds later than COMPLETED, because you
> > > reap zombie processes by collecting their exit status. Maybe you could have
> > > AHS_COMPLETE or AHS_IO_COMPLETE for the state where the I/O is done but
> > > there's still completion-related work to be done, and then the other state
> > > could be AHS_DONE or AHS_FINISHED or AHS_FINAL or AHS_REAPED or something.
> >
> > How about
> >
> > AHS_COMPLETE_KERNEL or AHS_COMPLETE_RAW - raw syscall completed
> > AHS_COMPLETE_SHARED_CB - shared callback completed
> > AHS_COMPLETE_LOCAL_CB - local callback completed
> >
> > ?
>
> That's not bad. I like RAW better than KERNEL.

Cool.


> I was hoping to use different works like COMPLETE and DONE rather than, as
> you did it here, COMPLETE and COMPLETE, but it's probably fine.

Once the IO is really done, the handle is immediately recycled (and moved into
IDLE state, ready to be used again).

Greetings,

Andres Freund

  44. Re: AIO v2.2

    Robert Haas <robertmhaas@gmail.com> — 2025-01-14T13:40:46Z

    On Mon, Jan 13, 2025 at 4:46 PM Andres Freund <andres@anarazel.de> wrote:
> Once the IO is really done, the handle is immediately recycled (and moved into
> IDLE state, ready to be used again).

OK, fair enough.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  45. Re: AIO v2.3

    Andres Freund <andres@anarazel.de> — 2025-01-23T04:29:02Z

    Hi,

Attached is v2.3.

There are a lot of changes - primarily renaming things based on on-list and
off-list feedback. But also some other things


Functional:

- Added pg_aios view

- md.c registering sync requests, that was previously omitted
  - This triggered stats issues during shutdown, as it can lead to IO workers
    emitting stats in some corner cases. I've written a patch series to
    address this [1]. For now I've included them in this patchset, but I would
    like to push the reordering patches soon.


- Testing error handling for temp table IO made me realize that the previous
  pattern of just tracking the refcount held by the IO subsystem in the
  LocalRefCount array leads to spurious buffer leak warnings [2].  I attached
  a prototype patch to deal with this by bringing localbuf.c more in line with
  bufmgr.c, but it needs some cleanup.

  That's in v2.3-0020-WIP-localbuf-Track-pincount-in-BufferDesc-as-we.patch


- Wait for all IOs to finish during shutdown. This is primarily required to
  ensure there aren't IOs initiated by a prior "owner" of a ProcNumber when a
  new backend starts. But there are also some kernels that don't like when
  exiting while IO is in flight.


- Re-armed local completion callbacks, they're required for correctness of
  temporary table IO


- Added a bunch of central debug helpers that only lead to output if
  PGAIO_VERBOSE is defined. That did make code a good bit more readable.



Polishing:

- Lots of copy editing, a lot of it thanks to feedback by Noah and Heikki


- Renamed the previous concept of a "subject" of an IO (i.e. what the IO is
  executed on, an smgr relation, a WAL file, ...) to "target".  I'm not in
  love with that name, but I went through dozens of variations, and it does
  seem better than subject.

  Not sure anymore how I ended up with subject, it's grammatically off and not
  very descriptive to boot.


- Renamed "PgAioHandleRef" and related functions to
  PgAioWaitRef/pgaio_wref_*(), that seems a lot more descriptive.


- Renamed pgaio_io_get() to pgaio_io_acquire()


- Renamed the IO handle states (PREPARED to STAGED, IN_FLIGHT to SUBMITTED,
  REAPED to COMPLETED_IO).

  Particularly the various COMPLETED state names aren't necessarily final,
  I've been debating a bunch of variations with Thomas and Robert


- Renamed aio_ref.h to aio_types.h, moved a few more types into it.


- Renamed completion callbacks to not use "shared" anymore - ->prepare was not
  really shared and now local callbacks are back (in a restricted form).

  s/PgAioHandleSharedCallback/PgAioHandleCallback/
  s/pgaio_io_add_shared_cb/pgaio_io_register_callbacks/

  Not entirely sure *register_callbacks is the best, happy to adjust.


- Renamed the ->error IO handle callback to ->report

  Also renamed s/pgaio_result_log/pgaio_result_report/g


- Renamed the ->prepare IO handle callback to ->stage

  Also renamed s/pgaio_result_log/pgaio_result_report/g


- Partially addressed request to reorder aio/README.md


- Determine shared memory allocation size with PG_IOV_MAX not io_combine_limit

  io_combine_limit is USERSET, so it's not correct to use it for shmem
  allocations. I chose PG_IOV_MAX instead of MAX_IO_COMBINE_LIMIT because this
  is a more generic limit than bufmgr.c IO.


- Prefix PgAio* enums with PGAIO_, global variables with pgaio_*


- Split out callback related cod from aio_subject.c (now aio_target.c) into
  aio_callback.c. The target specific code is rather small, so this makes a
  lot more sense.


- Distributed functions into more appropriate .c files, documented the choice
  in aio.h, reorder them


- Disowned lwlock: More consistent naming, reduce diff size, resume interrupts

  Heikki asked to clear ->owner when disowning the lock - but we currently
  *never* clear it doesn't seem right to do so when disowning the lock.


- IO data that can be set on a handle (to e.g. transport an array of Buffers
  to the completion callbacks) is now done with
  pgaio_io_(get|set)_handle_data(). Mainly to distinguish it from data that's
  actually the target/source of a read/write.

  Heikki suggested to make this per-callback data, but I don't think there's
  currently a use case for that, and it'd add a fair bit of memory overhead. I
  added a comment documenting this.


- Lots of other cleanups, added comments and the like



Todo:

- Reorder README further


- Make per backend state not indexed by ProcNumber, as that requires reserving
  per-backend state for IO workers, which will never need them


- Clean up localbuf.c "preparation" patches


- Add more tests - I had hoped to get to this, but got sidetracked with a
  bunch of things I found while testing


- I started looking into having a distinct type for the public pgaio_io_*
  related functions that can be used just by the issuer of the IO. It does
  make things a bit easier to understand, but also complicates naming. Not
  sure if it's worth it yet.


- Need to define (and test) the behavior when an IO worker fails to reopen the
  file for an IO


- Heikki doesn't love pgaio_submit_staged(), suggested pgaio_kick_staged() or
  such. I don't love that name though.


- There's some duplicated code in aio_callback.c, it'd be nice to deduplicate
  the callback invaction of the different callbacks


- Local callbacks are triggered from within pgaio_io_reclaim(), that's not
  exactly pretty. But it's currently the most central place to deal with the
  case of IOs for which the shared completion callback was called in another
  backend.


- As Jakub suggested (below [3]), when io_method=io_uring is used, we can run
  of out file descriptors much more easily. At the very least we need a good
  error message, perhaps also some rlimit adjusting (probably as a second
  step, if so).


- Thomas is working on the read_stream.c <-> bufmgr.c integration piece


- Start to write docs adjustments



[1] https://postgr.es/m/kgng5nrvnlv335evmsuvpnh354rw7qyazl73kdysev2cr2v5zu%40m3cfzxicm5kp
[2] https://postgr.es/m/j6hny5ivrfqw356ugoy3ti5ccadamluekxod4k6amao5snew6c%40t5h3bwhrgfqx
[3] https://postgr.es/m/tp63m6tcbi7mmsjlqgxd55sghhwvjxp3mkgeljffkbaujezvdl%40fvmdr3c6uhat

  46. Re: AIO v2.3

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-02-06T10:50:04Z

    On Thu, Jan 23, 2025 at 5:29 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> Attached is v2.3.
>
> There are a lot of changes - primarily renaming things based on on-list and
> off-list feedback. But also some other things

[..snip]

Hi Andres, OK, so I've hastily launched AIO v2.3 (full, 29 patches)
patchset probe run before going for short vacations and here results
are attached*. TLDR; in terms of SELECTs the master vs aioworkers
looks very solid! I was kind of afraid that additional IPC to separate
processes would put workers at a disadvantage a little bit , but
that's amazingly not true. The intention of this effort just to see if
committing AIO with defaults as it stands is good enough to not cause
basic regressions for users and to me it looks like it is nearly
finished :)). So here to save time I have *not* tested aio23 with
io_uring, it's just about aioworkers (the future default).

Random notes and thoughts:

1. not a single crash was observed , but those were pretty short runs

2. my very limited in terms of time data analysis thoughts
- most of the time perf  with aioworkers is identical (+/- 3%) as of
the master, in most cases it is much BETTER
- up to like 2.01x boosts can be spotted even on low-end like this but
with fast I/O even without IO_URING (just workers)
- on seqscans "sata" with datasets bigger than VFS-cache ("big") and
without parallel workers, it looks like it's always better
- on parallel seqscans "sata" with datasets bigger than VFS-cache
("big") and high e_io_c with high client counts(sigh!), it looks like
it would user noticeable big regression but to me it's not regression
itself, probably we are issuing way too many posix_fadvise()
readaheads with diminishing returns. Just letting you know. Not sure
it is worth introducing some global (shared aioworkers e_io_c
limiter), I think not. I think it could also be some maintenance noise
on that I/O device, but I have no isolated SATA RAID10 with like 8x
HDDs in home to launch such a test to be absolutely sure.

3. with aioworkers in documentation it would worth pointing out that
`iotop` won't be good enough to show which PID is doing I/O anymore .
I've often get question like this: who is taking the most of I/O right
now because storage is fully saturated on multi-use system. Not sure
it would require new view or not (pg_aios output seems to be not more
like in-memory debug view that would be have to be sampled
aggressively, and pg_statio_all_tables shows well table, but not PID
-- same for pg_stat_io). IMHO if docs would be simple like
"In order to understand which processes (PIDs) are issuing lots of
IOs, please check pg_stat_activty for *IO/AioCompletion* waits events"
it should be good enough for a start.

Bench machine: it was intentionally much smaller hardware. Azure's
Lsv2 L8s_v2 (1st gen EPYC/1s4c8t, with kernel 6.10.11+bpo-cloud-amd64
and booted with mem=12GB that limited real usable RAM memory to just
like ~8GB to stress I/O). liburing 2.9. Normal standard compile
options were used without asserts (such as normal users would use).
Bench had those two I/O storage (with XFS) attached:
- "sata" stands for Azure's "Premium SSD LRS" mounted on /sata
(Size=255GB, Max IOPS=1100 (@ 4kB?), Max throughput=125MB/s)
- "nvme" stands for bulit-in NVME on that VM mounted on /nvme
(Size=1788GB, Max IOPS=8000 (@ 4kB?))

I'll try to see in the coming weeks if dedicating more time is
possible (long run tests, more write tests, maybe some basic I/O
failure injections tests).

-J.

* = 8640 test runs, always with restart and flushing VFS cache, took
probably 2-3 days? I've had to reduce tries to 1 and limit myself to
just reads just to get it running solid, before I left and not to miss
the plane :^)

  47. Re: AIO v2.3

    Thomas Munro <thomas.munro@gmail.com> — 2025-02-10T22:48:38Z

    On Thu, Jan 23, 2025 at 5:29 PM Andres Freund <andres@anarazel.de> wrote:
+/* caller will issue more io, don't submit */
+#define READ_BUFFERS_MORE_MORE_MORE (1 << 3)

> - Heikki doesn't love pgaio_submit_staged(), suggested pgaio_kick_staged() or
>  such. I don't love that name though.

Problem statement: You want to be able to batch I/O submission, ie
make a single call to ioring_enter() (and other mechanisms) to start
several I/Os, but the code that submits is inside StartReadBuffers()
and the code that knows how many I/Os it wants to start now is at a
higher level, read_stream.c and in future elsewhere.  So you invented
this flag to tell StartReadBuffers() not to call
pgaio_submit_staged(), because you promise to do it later, via this
staging list.  Additionally, there is a kind of programming rule here
that you *must* submit I/Os that you stage, you aren't allowed to (for
example) stage I/Os and then sleep, so it has to be a fairly tight
piece of code.

Would the API be better like this?:  When you want to create a batch
of I/Os submitted together, you wrap the work in pgaio_begin_batch()
and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
Then bufmgr wouldn't need this flag: when it (or anything else) calls
smgrstartreadv(), if there is not currently an explicit batch then it
would be submitted immediately, and otherwise it would only be staged.
This way, batch construction (or whatever word you prefer for batch)
is in a clearly and explicitly demarcated stretch of code in one
lexical scope (though its effect is dynamically scoped just like the
staging list itself because we don't want to pass explicit I/O
contexts through the layers), but code that doesn't call those and
reaches AsyncReadBuffer() or whatever gets an implicit batch of size
one and that's also OK.  Not sure what semantics nesting would have
but I doubt it matters much.

> Things that need to be fixed / are fixed in this:
> - max pinned buffers should be limited by io_combine_limit, not * 4
> - overflow distance
> - pins need to be limited in more places

I have patches for these and a few more things and will post in a
separate thread shortly because they can be understood without
reference to this AIO stuff and that'll hopefully be more digestible.

+    /*
+     * In some rare-ish cases one operation causes multiple reads (e.g. if a
+     * buffer was concurrently read by another backend). It'd be much better
+     * if we ensured that each ReadBuffersOperation covered only one IO - but
+     * that's not entirely trivial, due to having pinned victim buffers before
+     * starting IOs.
+     *
+     * TODO: Change the API of StartReadBuffers() to ensure we only ever need
+     * one IO.

Likewise.

+    /* IO finished, but result has not yet been processed */
+    PGAIO_HS_COMPLETED_IO,
+
+    /* IO completed, shared completion has been called */
+    PGAIO_HS_COMPLETED_SHARED,
+
+    /* IO completed, local completion has been called */
+    PGAIO_HS_COMPLETED_LOCAL,

(Repeating something I mentioned in off-list bikeshedding)  I wondered
if it might be clearer to use the terminology "terminated" for the
work that PostgreSQL has to do after an I/O completes, instead of
overloading/subdividing the term "completed".  We already "terminate"
an I/O when smgr I/O completes in pre-existing bufmgr terminology, and
this feels like a sort of generalisation of that notion.  In this AIO
world, some work is done by the backend that receives the completion
notification from the kernel, and some is done by the backend that
submitted the I/O in the first place, a division that doesn't exist
with simple synchronous system calls.  I wonder if it would be clearer
to use terms based on those two roles, rather than "shared" and
"local", leading to something like:

PGAIO_HS_COMPLETED,
PGAIO_HS_TERMINATED_BY_COMPLETER,
PGAIO_HS_TERMINATED_BY_SUBMITTER,

  48. Re: AIO v2.3

    Andres Freund <andres@anarazel.de> — 2025-02-10T22:54:22Z

    Hi,

On 2025-02-11 11:48:38 +1300, Thomas Munro wrote:
> On Thu, Jan 23, 2025 at 5:29 PM Andres Freund <andres@anarazel.de> wrote:
> > - Heikki doesn't love pgaio_submit_staged(), suggested pgaio_kick_staged() or
> >  such. I don't love that name though.
> 
> Problem statement: You want to be able to batch I/O submission, ie
> make a single call to ioring_enter() (and other mechanisms) to start
> several I/Os, but the code that submits is inside StartReadBuffers()
> and the code that knows how many I/Os it wants to start now is at a
> higher level, read_stream.c and in future elsewhere.  So you invented
> this flag to tell StartReadBuffers() not to call
> pgaio_submit_staged(), because you promise to do it later, via this
> staging list.  Additionally, there is a kind of programming rule here
> that you *must* submit I/Os that you stage, you aren't allowed to (for
> example) stage I/Os and then sleep, so it has to be a fairly tight
> piece of code.
> 
> Would the API be better like this?:  When you want to create a batch
> of I/Os submitted together, you wrap the work in pgaio_begin_batch()
> and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
> Then bufmgr wouldn't need this flag: when it (or anything else) calls
> smgrstartreadv(), if there is not currently an explicit batch then it
> would be submitted immediately, and otherwise it would only be staged.
> This way, batch construction (or whatever word you prefer for batch)
> is in a clearly and explicitly demarcated stretch of code in one
> lexical scope (though its effect is dynamically scoped just like the
> staging list itself because we don't want to pass explicit I/O
> contexts through the layers), but code that doesn't call those and
> reaches AsyncReadBuffer() or whatever gets an implicit batch of size
> one and that's also OK.  Not sure what semantics nesting would have
> but I doubt it matters much.

I'm a bit unexcited about the work to redesign this, but I also admit that you
have a point :)

Linux calls a similar concept "plugging" the queue. I think I like "batch"
better, but only marginally.


> > Things that need to be fixed / are fixed in this:
> > - max pinned buffers should be limited by io_combine_limit, not * 4
> > - overflow distance
> > - pins need to be limited in more places
> 
> I have patches for these and a few more things and will post in a
> separate thread shortly because they can be understood without
> reference to this AIO stuff and that'll hopefully be more digestible.

Yay!


> +    /* IO finished, but result has not yet been processed */
> +    PGAIO_HS_COMPLETED_IO,
> +
> +    /* IO completed, shared completion has been called */
> +    PGAIO_HS_COMPLETED_SHARED,
> +
> +    /* IO completed, local completion has been called */
> +    PGAIO_HS_COMPLETED_LOCAL,
> 
> (Repeating something I mentioned in off-list bikeshedding)  I wondered
> if it might be clearer to use the terminology "terminated" for the
> work that PostgreSQL has to do after an I/O completes, instead of
> overloading/subdividing the term "completed".  We already "terminate"
> an I/O when smgr I/O completes in pre-existing bufmgr terminology, and
> this feels like a sort of generalisation of that notion.

I have a mild laziness preference for complete over terminate, but not
more. If others agree with Thomas, I'm ok with changing it.


> In this AIO world, some work is done by the backend that receives the
> completion notification from the kernel, and some is done by the backend
> that submitted the I/O in the first place, a division that doesn't exist
> with simple synchronous system calls.  I wonder if it would be clearer to
> use terms based on those two roles, rather than "shared" and "local",
> leading to something like:
> 
> PGAIO_HS_COMPLETED,
> PGAIO_HS_TERMINATED_BY_COMPLETER,
> PGAIO_HS_TERMINATED_BY_SUBMITTER,

I don't love those, because the SHARED/LOCAL does imply more clearly what you
have access to. I.e. executing things in a shared completion callback for IO
on a temporary buffer doesn't make sense, you won't have access to the local
buffer table.

Greetings,

Andres Freund

  49. Re: AIO v2.3

    Andres Freund <andres@anarazel.de> — 2025-02-10T23:10:46Z

    Hi,

On 2025-02-06 11:50:04 +0100, Jakub Wartak wrote:
> Hi Andres, OK, so I've hastily launched AIO v2.3 (full, 29 patches)
> patchset probe run before going for short vacations and here results
> are attached*.

Thanks for doing that work!


> TLDR; in terms of SELECTs the master vs aioworkers looks very solid!

Phew! Weee! Yay.


> I was kind of afraid that additional IPC to separate processes would put
> workers at a disadvantage a little bit , but that's amazingly not true.

It's a measurable disadvantage, it's just more than counteracted by being able
to do IO asynchronously :).

It's possible to make it more visible, by setting io_combine_limit = 1. If you
have a small shared buffers with everything in the kernel cache, the dispatch
overhead starts to be noticeable above several GB/s. But that's ok, I think.


> The intention of this effort just to see if committing AIO with defaults as
> it stands is good enough to not cause basic regressions for users and to me
> it looks like it is nearly finished :)).

That's really good to hear.  I think we can improve things a lot in the
future, but we gotta start somewhere...


> 1. not a single crash was observed , but those were pretty short runs
> 
> 2. my very limited in terms of time data analysis thoughts
> - most of the time perf  with aioworkers is identical (+/- 3%) as of
> the master, in most cases it is much BETTER

I assume s/most/some/ for the second most?


> - on parallel seqscans "sata" with datasets bigger than VFS-cache
> ("big") and high e_io_c with high client counts(sigh!), it looks like
> it would user noticeable big regression but to me it's not regression
> itself, probably we are issuing way too many posix_fadvise()
> readaheads with diminishing returns. Just letting you know. Not sure
> it is worth introducing some global (shared aioworkers e_io_c
> limiter), I think not. I think it could also be some maintenance noise
> on that I/O device, but I have no isolated SATA RAID10 with like 8x
> HDDs in home to launch such a test to be absolutely sure.

I'm inclined to not introduce a global limit for now - it's pretty hard to
make that scale to fast IO devices, so you need a multi-level design, where
each backend can issue a few IOs without consulting the global limit and only
after that you need to get the right to issue even more IOs from the shared
"pool".

I think this is basically a configuration issue - configuring a high e_io_c
for a device that can't handle that and then load it up with a lot of clients,
well, that'll not work out great.


> 3. with aioworkers in documentation it would worth pointing out that
> `iotop` won't be good enough to show which PID is doing I/O anymore .
> I've often get question like this: who is taking the most of I/O right
> now because storage is fully saturated on multi-use system. Not sure
> it would require new view or not (pg_aios output seems to be not more
> like in-memory debug view that would be have to be sampled
> aggressively, and pg_statio_all_tables shows well table, but not PID
> -- same for pg_stat_io). IMHO if docs would be simple like
> "In order to understand which processes (PIDs) are issuing lots of
> IOs, please check pg_stat_activty for *IO/AioCompletion* waits events"
> it should be good enough for a start.

pg_stat_get_backend_io() should allow to answer that, albeit with the usual
weakness of our stats system, namely that the user has to diff two snapshots
themselves. It probably also has the weakness of not showing results for
queries before they've finished, although I think that's something we should
be able to improve without too much trouble (not in this release though, I
suspect).

I guess we could easily reference pg_stat_get_backend_io(), but a more
complete recipe isn't entirely trivial...


> Bench machine: it was intentionally much smaller hardware. Azure's
> Lsv2 L8s_v2 (1st gen EPYC/1s4c8t, with kernel 6.10.11+bpo-cloud-amd64
> and booted with mem=12GB that limited real usable RAM memory to just
> like ~8GB to stress I/O). liburing 2.9. Normal standard compile
> options were used without asserts (such as normal users would use).

Good - the asserts in the aio patches are a bit more noticeable than the ones
in master.


Thanks again for running these!


Greetings,

Andres Freund

  50. Re: AIO v2.3

    James Hunter <james.hunter.pg@gmail.com> — 2025-02-11T17:10:51Z

    On Mon, Feb 10, 2025 at 2:40 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>
...
> Problem statement: You want to be able to batch I/O submission, ie
> make a single call to ioring_enter() (and other mechanisms) to start
> several I/Os, but the code that submits is inside StartReadBuffers()
> and the code that knows how many I/Os it wants to start now is at a
> higher level, read_stream.c and in future elsewhere.  So you invented
> this flag to tell StartReadBuffers() not to call
> pgaio_submit_staged(), because you promise to do it later, via this
> staging list.  Additionally, there is a kind of programming rule here
> that you *must* submit I/Os that you stage, you aren't allowed to (for
> example) stage I/Os and then sleep, so it has to be a fairly tight
> piece of code.
>
> Would the API be better like this?:  When you want to create a batch
> of I/Os submitted together, you wrap the work in pgaio_begin_batch()
> and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
> Then bufmgr wouldn't need this flag: when it (or anything else) calls
> smgrstartreadv(), if there is not currently an explicit batch then it
> would be submitted immediately, and otherwise it would only be staged.
> This way, batch construction (or whatever word you prefer for batch)
> is in a clearly and explicitly demarcated stretch of code in one
> lexical scope (though its effect is dynamically scoped just like the
> staging list itself because we don't want to pass explicit I/O
> contexts through the layers), but code that doesn't call those and
> reaches AsyncReadBuffer() or whatever gets an implicit batch of size
> one and that's also OK.  Not sure what semantics nesting would have
> but I doubt it matters much.

I like this idea. If we want to submit a batch, then just submit a batch.

James

  51. Re: AIO v2.3

    Robert Haas <robertmhaas@gmail.com> — 2025-02-11T17:34:25Z

    On Tue, Feb 11, 2025 at 12:11 PM James Hunter <james.hunter.pg@gmail.com> wrote:
> I like this idea. If we want to submit a batch, then just submit a batch.

Sounds good to me, too.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  52. Re: AIO v2.3

    Andres Freund <andres@anarazel.de> — 2025-02-11T21:43:22Z

    Hi,

On 2025-02-11 11:48:38 +1300, Thomas Munro wrote:
> Would the API be better like this?:  When you want to create a batch
> of I/Os submitted together, you wrap the work in pgaio_begin_batch()
> and pgaio_submit_batch(), eg the loop in read_stream_lookahead().

One annoying detail is that an API like this would afaict need resowner
support or something along those lines (e.g. xact callbacks plus code in each
aux process' sigsetjmp() block). Otherwise I don't know how we would ensure
that the "batch-is-in-progress" flag/counter would get reset.

Alternatively we could make pgaio_batch_begin() basically start a critical
section, but that doesn't seem like a good idea, because too much that needs
to happen around buffered IO isn't compatible with critical sections.


Does anybody see a need for batches to be nested? I'm inclined to think that
that would be indicative of bugs and should therefore error/assert out.


One way we could avoid the need for a mechanism to reset-batch-in-progress
would be to make batch submission controlled by a flag on the IO. Something
like
    pgaio_io_set_flag(ioh, PGAIO_HF_BATCH_SUBMIT)

IFF PGAIO_HF_BATCH_SUBMIT is set, the IOs would need to be explicitly
submitted using something like the existing
    pgaio_submit_staged();
(although renaming it to something with batch in the name might be
appropriate)

That way there's no explicit "we are in a batch" state that needs to be reset
in case of errors.

Greetings,

Andres Freund

  53. Re: AIO v2.3

    Robert Haas <robertmhaas@gmail.com> — 2025-02-12T18:00:22Z

    On Tue, Feb 11, 2025 at 4:43 PM Andres Freund <andres@anarazel.de> wrote:
> Alternatively we could make pgaio_batch_begin() basically start a critical
> section, but that doesn't seem like a good idea, because too much that needs
> to happen around buffered IO isn't compatible with critical sections.

A critical section sounds like a bad plan.

> Does anybody see a need for batches to be nested? I'm inclined to think that
> that would be indicative of bugs and should therefore error/assert out.

I can imagine somebody wanting to do it, but I think we can just say
no. I mean, it's no different from WAL record construction. There's no
theoretical reason you couldn't want to concurrently construct
multiple WAL records and then submit them one after another, but if
you want to do that, you have to do your own bookkeeping. It seems
fine to apply the same principle here.

> One way we could avoid the need for a mechanism to reset-batch-in-progress
> would be to make batch submission controlled by a flag on the IO. Something
> like
>     pgaio_io_set_flag(ioh, PGAIO_HF_BATCH_SUBMIT)
>
> IFF PGAIO_HF_BATCH_SUBMIT is set, the IOs would need to be explicitly
> submitted using something like the existing
>     pgaio_submit_staged();
> (although renaming it to something with batch in the name might be
> appropriate)
>
> That way there's no explicit "we are in a batch" state that needs to be reset
> in case of errors.

I'll defer to Thomas or others on whether this is better or worse,
because I don't know. It means that the individual I/Os have to know
that they are in a batch, which isn't necessary with the begin/end
batch interface. But if we're expecting that to happen in a pretty
confined amount of code -- similar to WAL record construction -- then
that might not be a problem anyway.

I think if you don't do this, I'd do (sub)xact callbacks rather than a
resowner integration, unless you decide you want to support multiple
concurrent batches. You don't really need or want to tie it to a
resowner unless there are multiple objects each of which can have its
own resources.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  54. Re: AIO v2.3

    James Hunter <james.hunter.pg@gmail.com> — 2025-02-12T20:26:45Z

    On Tue, Feb 11, 2025 at 1:44 PM Andres Freund <andres@anarazel.de> wrote:
>
...

> Alternatively we could make pgaio_batch_begin() basically start a critical
> section, but that doesn't seem like a good idea, because too much that needs
> to happen around buffered IO isn't compatible with critical sections.
>
>
> Does anybody see a need for batches to be nested? I'm inclined to think that
> that would be indicative of bugs and should therefore error/assert out.

Fwiw, in a similar situation in the past, I just blocked, waiting for
the in-flight batch to complete, before sending the next batch. So I
had something like:

void begin_batch(...)
{
  if (batch_in_progress())
    complete_batch(...);

  /* ok start the batch now */
}

In my case, batches were nested because different access methods
(e.g., Index) can call/trigger other access methods (Heap), and both
access methods might want to issue batch reads. However, the "inner"
access method  might not be aware of the "outer" access method.

For simplicity, then, I just completed the outer batch. Note that this
is not optimal for performance (because a nested batch ends up
stalling the outer batch), but it does keep the code simple...

James

  55. Re: AIO v2.3

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-02-14T07:40:48Z

    On Tue, Feb 11, 2025 at 12:10 AM Andres Freund <andres@anarazel.de> wrote:

>> TLDR; in terms of SELECTs the master vs aioworkers looks very solid!

> Phew! Weee! Yay.

Another good news: I've completed a full 24h pgbench run on the same
machine and it did not fail or report anything suspicious. FYI,
patchset didn't not apply anymore (seems patches 1..6 are already
applied on master due to checkpoint shutdown sequence), but there was
a failed hunk in patch #12 yesterday too:
[..]
patching file src/backend/postmaster/postmaster.c
Hunk #10 succeeded at 2960 (offset 14 lines).
Hunk #11 FAILED at 3047.
[..]
1 out of 15 hunks FAILED -- saving rejects to file
src/backend/postmaster/postmaster.c.rej

anyway, so on master @ a5579a90af05814eb5dc2fd5f68ce803899d2504 (~ Jan
24) to have clean apply I've used the below asserted build:

meson setup build --reconfigure --prefix=/usr/pgsql18.aio --debug
-Dsegsize_blocks=13 -Dcassert=true
/usr/pgsql18.aio/bin/pgbench -i -s 500 --partitions=100 # ~8GB
/usr/pgsql18.aio/bin/pgbench -R 1500 -c 100 -j 4 -P 1 -T 86400

with some add-on functionalities:
effective_io_concurrency = '4'
shared_buffers = '2GB'
max_connections = '1000'
archive_command = 'cp %p /dev/null'
archive_mode = 'on'
summarize_wal = 'on'
wal_summary_keep_time = '1h'
wal_compression = 'on'
wal_log_hints = 'on'
max_wal_size = '1GB'
shared_preload_libraries = 'pg_stat_statements'
huge_pages = 'off'
wal_receiver_status_interval = '1s'

so the above got perfect run:
[..]
duration: 86400 s
number of transactions actually processed: 129615534
number of failed transactions: 0 (0.000%)
latency average = 5.332 ms
latency stddev = 24.107 ms
rate limit schedule lag: avg 0.748 (max 1992.517) ms
initial connection time = 124.472 ms
tps = 1500.179231 (without initial connection time)

> > I was kind of afraid that additional IPC to separate processes would put
> > workers at a disadvantage a little bit , but that's amazingly not true.
>
> It's a measurable disadvantage, it's just more than counteracted by being able
> to do IO asynchronously :).
>
> It's possible to make it more visible, by setting io_combine_limit = 1. If you
> have a small shared buffers with everything in the kernel cache, the dispatch
> overhead starts to be noticeable above several GB/s. But that's ok, I think.

Sure it is.

> > 2. my very limited in terms of time data analysis thoughts
> > - most of the time perf  with aioworkers is identical (+/- 3%) as of
> > the master, in most cases it is much BETTER
>
> I assume s/most/some/ for the second most?

Right, pardon for my excited moment ;)

> > - on parallel seqscans "sata" with datasets bigger than VFS-cache
> > ("big") and high e_io_c with high client counts(sigh!), it looks like
> > it would user noticeable big regression but to me it's not regression
> > itself, probably we are issuing way too many posix_fadvise()
> > readaheads with diminishing returns. Just letting you know. Not sure
> > it is worth introducing some global (shared aioworkers e_io_c
> > limiter), I think not. I think it could also be some maintenance noise
> > on that I/O device, but I have no isolated SATA RAID10 with like 8x
> > HDDs in home to launch such a test to be absolutely sure.
>
> I think this is basically a configuration issue - configuring a high e_io_c
> for a device that can't handle that and then load it up with a lot of clients,
> well, that'll not work out great.

Sure, btw i'm going to also an idea about autotuning that e_io_c in
that related thread where everybody is complaining about it

> > 3. with aioworkers in documentation it would worth pointing out that
> > `iotop` won't be good enough to show which PID is doing I/O anymore .
> > I've often get question like this: who is taking the most of I/O right
> > now because storage is fully saturated on multi-use system. Not sure
> > it would require new view or not (pg_aios output seems to be not more
> > like in-memory debug view that would be have to be sampled
> > aggressively, and pg_statio_all_tables shows well table, but not PID
> > -- same for pg_stat_io). IMHO if docs would be simple like
> > "In order to understand which processes (PIDs) are issuing lots of
> > IOs, please check pg_stat_activty for *IO/AioCompletion* waits events"
> > it should be good enough for a start.
>
> pg_stat_get_backend_io() should allow to answer that, albeit with the usual
> weakness of our stats system, namely that the user has to diff two snapshots
> themselves. It probably also has the weakness of not showing results for
> queries before they've finished, although I think that's something we should
> be able to improve without too much trouble (not in this release though, I
> suspect).
>
> I guess we could easily reference pg_stat_get_backend_io(), but a more
> complete recipe isn't entirely trivial...

I was trying to come out with something that could be added to docs,
but the below thing is too ugly and as you stated the primary weakness
is that query needs to finish before it is reflected:

WITH
    b AS (SELECT 0 AS step, pid, round(sum(write_bytes)/1024/1024) AS
wMB, NULL::void, NULL::void FROM pg_stat_activity,
pg_stat_get_backend_io(pid) GROUP BY pid),
    flush AS (SELECT 0 as step, 0, 0, pg_sleep(1), pg_stat_clear_snapshot()),
    e AS (SELECT 1 AS step, pid, round(sum(write_bytes)/1024/1024) AS
wMB, NULL::void, NULL::void FROM pg_stat_activity,
pg_stat_get_backend_io(pid) GROUP BY pid),
    picture AS MATERIALIZED (
        SELECT * FROM b
        UNION ALL
        SELECt * FROM flush
        UNION ALL
        SELECT * FROM e
    )
SELECT * FROM (
    SELECT pid, wMB - LAG(wMB, 1) OVER (PARTITION BY pid ORDER BY
step) AS "wMB/s" FROM picture
) WHERE "wMB/s" > 0;

\watch 1

-J.

  56. Re: AIO v2.3

    Andres Freund <andres@anarazel.de> — 2025-02-17T21:01:42Z

    Hi,

On 2025-02-12 13:00:22 -0500, Robert Haas wrote:
> On Tue, Feb 11, 2025 at 4:43 PM Andres Freund <andres@anarazel.de> wrote:
> > One way we could avoid the need for a mechanism to reset-batch-in-progress
> > would be to make batch submission controlled by a flag on the IO. Something
> > like
> >     pgaio_io_set_flag(ioh, PGAIO_HF_BATCH_SUBMIT)
> >
> > IFF PGAIO_HF_BATCH_SUBMIT is set, the IOs would need to be explicitly
> > submitted using something like the existing
> >     pgaio_submit_staged();
> > (although renaming it to something with batch in the name might be
> > appropriate)
> >
> > That way there's no explicit "we are in a batch" state that needs to be reset
> > in case of errors.
> 
> I'll defer to Thomas or others on whether this is better or worse,
> because I don't know. It means that the individual I/Os have to know
> that they are in a batch, which isn't necessary with the begin/end
> batch interface. But if we're expecting that to happen in a pretty
> confined amount of code -- similar to WAL record construction -- then
> that might not be a problem anyway.
> 
> I think if you don't do this, I'd do (sub)xact callbacks rather than a
> resowner integration, unless you decide you want to support multiple
> concurrent batches. You don't really need or want to tie it to a
> resowner unless there are multiple objects each of which can have its
> own resources.

I have working code for that. There unfortunately is an annoying problem:

It afaict is not really possible to trigger a WARNING/ERRROR/Assert in xact
callbacks at the end of commands in an explicit transaction.  Consider
something like:

  BEGIN;
  SELECT start_but_not_end_aio_batch();

we would like to flag that the SELECT query started an AIO batch but didn't
end it. But at the momment there, afaict, is no proper way to do that, because
xact.c won't actually do much at the end of a command in a transaction block:

			/*
			 * This is the case when we have finished executing a command
			 * someplace within a transaction block.  We increment the command
			 * counter and return.
			 */
		case TBLOCK_INPROGRESS:
		case TBLOCK_IMPLICIT_INPROGRESS:
		case TBLOCK_SUBINPROGRESS:
			CommandCounterIncrement();
			break;

If one instead integrates with resowners, that kind of thing works, because
exec_simple_query() calls PortalDrop(), which in turn calls
ResourceOwnerRelease().


And we don't reliably warn at a later time. While xact.c integration triggers
a warning for:

  BEGIN;
  SELECT start_but_not_end_aio_batch();
  COMMIT;

as we'd still have the batch open at COMMIT, it wouldn't trigger for

  BEGIN;
  SELECT start_but_not_end_aio_batch();
  ROLLBACK;

as AbortTransaction() might be called in an error and therefore can't assume
that it's a problem for a batch to still be open.


I guess I could just put something alongside that CommandCounterIncrement()
call, but that doesn't seem right.  I guess putting it alongside the
ResourceOwnerRelease() in PortalDrop() is a bit less bad? But still doesn't
seem great.


Just using resowners doesn't seem right either, it's not really free to
register something with resowners, and for read intensive IO we can start a
*lot* of batches, so doing unnecessary work isn't great.

Greetings,

Andres Freund

  57. Re: AIO v2.3

    Robert Haas <robertmhaas@gmail.com> — 2025-02-18T19:19:25Z

    On Mon, Feb 17, 2025 at 4:01 PM Andres Freund <andres@anarazel.de> wrote:
> If one instead integrates with resowners, that kind of thing works, because
> exec_simple_query() calls PortalDrop(), which in turn calls
> ResourceOwnerRelease().

Hmm, so maybe that's a reason to do it via resowner.c, then. The fact
that it's a singleton object is a bit annoying, but you could make it
not a singleton, and then either pass the relevant one to the
interface functions, or store the current one in a global variable
similar to CurrentMemoryContext or similar.

> I guess I could just put something alongside that CommandCounterIncrement()
> call, but that doesn't seem right.  I guess putting it alongside the
> ResourceOwnerRelease() in PortalDrop() is a bit less bad? But still doesn't
> seem great.

The thing that's weird about that is that it isn't really logically
linked to the portal. It feels like it more properly belongs in
StartTransactionCommand() / CommitTransactionCommand().

> Just using resowners doesn't seem right either, it's not really free to
> register something with resowners, and for read intensive IO we can start a
> *lot* of batches, so doing unnecessary work isn't great.

You don't necessarily have to register a new object for every batch,
do you? You could just register one and keep reusing it for the
lifetime of the query.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  58. Re: AIO v2.4

    Andres Freund <andres@anarazel.de> — 2025-02-19T19:10:44Z

    Hi,

Attached is v2.4 of the AIO patchset.

Changes:

- Introduce "batchmode", while not in batchmode, IOs get submitted immediately.

  Thomas didn't like how this worked previously, and while this was a
  surprisingly large amount of work, I agree that it looks better now.

  I vaccilated a bunch on the naming. For now it's

  extern void pgaio_enter_batchmode(void);
  extern void pgaio_exit_batchmode(void);

  I did adjust the README and wrote a reasonably long comment above enter:
  https://github.com/anarazel/postgres/blob/a324870186ddff9a31b10472b790eb4e744c40b3/src/backend/storage/aio/aio.c#L931-L960


- Batchmode needs to be exited in case of errors, for that

  - a new pgaio_after_error() call has been added to all the relevant places

  - xact.c calls to aio have been (re-)added to check that there are no
    in-progress batches / unsubmitted IOs at the end of a transaction.

    Before that I had just removed at-eoxact "callbacks" :)

    This checking has holes though:
    https://postgr.es/m/upkkyhyuv6ultnejrutqcu657atw22kluh4lt2oidzxxtjqux3%40a4hdzamh4wzo

    Because this only means that we will not detect all buggy code, rather
    than misbehaving for correct code, I think this may be ok for now.


- Renamed aio_init.h to aio_subsys.h

  The newly added pgaio_after_error() calls would have required including
  aio.h in a good bit more places that won't themselves issue AIO. That seemed
  wrong.  There already was a aio_init.h to avoid needing to include aio.h in
  places like ipci.c, but it seemed wrong to put pgaio_after_error() in
  aio_init.h.  So I renamed it to aio_subsys.h - not sure that's the best
  name, but I can live with it.


- Now that Thomas submitted the necessary read_stream.c improvements, the
  prior big TODO about one StartReadBuffers() call needing to start many IOs
  has been addressed.

  Thomas' thread: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com

  For now I've also included Thomas patches in my queue, but they should get
  pushed independently.  Review comments specific to those patches probably
  are better put on the other thread.

  Thomas' patches also fix several issues that were addressed in my WIP
  adjustments to read_stream.c.  There are a few left, but it does look
  better.

  The included commits are 0003-0008.


- I rewrote the tests into a tap test. That was exceedingly painful. Partially
  due to tap infrastructure bugs on windows that would sometimes cause
  inscrutable failures, see
  https://www.postgresql.org/message-id/wmovm6xcbwh7twdtymxuboaoarbvwj2haasd3sikzlb3dkgz76%40n45rzycluzft

  I just pushed that fix earlier today.


- Added docs for new GUCs, moved them to a more appropriate section

  See also https://postgr.es/m/x3tlw2jk5gm3r3mv47hwrshffyw7halpczkfbk3peksxds7bvc%40lguk43z3bsyq


- If IO workers fail to reopen the file for an IO, the IO is now marked as
  failed. Previously we'd just hang.

  To test this I added an injection point that triggers the failure. I don't
  know how else this could be tested.


- Added liburing dependency build documentation


- Added check hook to ensure io_max_concurrency = isn't set to 0 (-1 is for
  auto-config)


- Fixed that with io_method == sync we'd issue fadvise calls when not
  appropriate, that was a consequence of my hacky read_stream.c changes.


- Renamed some the aio<->bufmgr.c interface functions. Don't think they're
  quite perfect, but they're in later patches, so I don't want to focus too
  much on them rn.


- Comment improvements etc.


- Got rid of an unused wait event and renamed other wait events to make more
  sense.


- Previously the injection points were added as part of the test patch, I now
  moved them into the commits adding the code being tested. Was too annoying
  to edit otherwise.


Todo:

- there's a decent amount of FIXMEs in later commits related to ereport(LOG)s
  needing relpath() while in a critical section.  I did propose a solution to
  that yesterday:

  https://postgr.es/m/h3a7ftrxypgxbw6ukcrrkspjon5dlninedwb5udkrase3rgqvn%403cokde6btlrl

- A few more corner case tests for the interaction of multiple backends trying
  to do IO on overlapping buffers would be good.

- Our temp table test coverage is atrociously bad


Questions:

- The test module requires StartBufferIO() to be visible outside of bufmgr.c -
  I think that's ok, would be good to know if others agree.


I'm planning to push the first two commits soon, I think they're ok on their
own, even if nothing else were to go in.


Greetings,

Andres Freund

  59. Re: AIO v2.2

    Andres Freund <andres@anarazel.de> — 2025-02-21T19:31:33Z

    Hi,

I was just going through comments about LWLockDisown() and was reminded of
this:

On 2025-01-07 18:08:51 +0200, Heikki Linnakangas wrote:
> On LWLockDisown():
> > + * NB: This will leave lock->owner pointing to the current backend (if
> > + * LOCK_DEBUG is set). We could add a separate flag indicating that, but it
> > + * doesn't really seem worth it.
> 
> Hmm. I won't insist, but I feel it probably would be worth it. This is only
> in LOCK_DEBUG mode so there's no performance penalty in non-debug builds,
> and when you do compile with LOCK_DEBUG you probably appreciate any extra
> information.

I don't think that makes sense, as we, independent of this change, never clear
lock->owner. Not even when releasing a lock!  The background to that, I think,
is that there were some cases where we forgot to wake up all backends due to
race conditions, and that for that it's really useful to know the last owner.

That could perhaps be evolved or documented better, but it's pretty much
independent of the patch at hand.

Greetings,

Andres Freund

  60. Re: AIO v2.4

    Andres Freund <andres@anarazel.de> — 2025-02-24T10:50:56Z

    Hi,

On 2025-02-19 14:10:44 -0500, Andres Freund wrote:
> I'm planning to push the first two commits soon, I think they're ok on their
> own, even if nothing else were to go in.

I did that for the lwlock patch.

But I think I might not do the same for the "Ensure a resowner exists for all
paths that may perform AIO" patch. The paths for which we are missing
resowners are concerned WAL writes - but it'll be a while before we get
AIO WAL writes.

It'd be fairly harmless to do this change before, but I found the justifying
code comments hard to rephrase. E.g.:

--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
     BaseInit();
 
     bootstrap_signals();
+
+    /* need a resowner for IO during BootStrapXLOG() */
+    CreateAuxProcessResourceOwner();
+
     BootStrapXLOG(bootstrap_data_checksum_version);
 
+    ReleaseAuxProcessResources(true);
+    CurrentResourceOwner = NULL;
+
     /*
      * To ensure that src/common/link-canary.c is linked into the backend, we
      * must call it from somewhere.  Here is as good as anywhere.

Given that there's no use of resowners inside BootStrapXLOG() today and not
for the next months it seems confusing?


Greetings,

Andres Freund

  61. AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-04T19:00:14Z

    Hi,

Attached is v2.5 of the AIO patchset.

Relative to 2.4 I:

- Committed some earlier commits. I ended up *not* committing the patch to
  create resowners in more backends (e.g. walsender), as that's not really a
  dependency for now.

  One of the more important things to get committed was in a separate thread:
  https://postgr.es/m/b6vveqz6r3wno66rho5lqi6z5kyhfgtvi3jcodyq5rlpp3cu44%40c6dsgf3z7yhs

  Now relpath() can be used for logging while in a critical section. That
  alone allowed to remove most of the remaining FIXMEs.


- Split md.c read/write patches, the write side is more complicated and isn't
  needed before write support arrives (much later in the queue and very likely
  not for 18).

  The complicated bit about write support is needing to
  register_dirty_segment() after completion of the write. If
  RegisterSyncRequest() fails, the IO completer needs to open the file and
  sync itself, unfortunately PathNameOpenFile() allocates memory, which isn't
  ok while in a critical section (even though it'd not be detected, as it's
  using malloc()).


- Reordered patches so that Thomas' read_stream work is after the basic AIO
  infrastructure patches, there's no dependency to the earlier patches

  I think Thomas might have a newer version of some of these, but since
  they're not intended to be committed as part of this, I didn't spend the
  time to rebase to the last version.


- Added a small bit of data that can be provided to callbacks, that makes it a
  lot cleaner to transport information like ZERO_ON_ERROR.

  I also did s/shared_callbacks/callbacks/, as the prior name was outdated.


- Substantially expanded tests, most importantly generic temp file tests and
  AIO specific cross-backend tests

  As part of the expanded tests I also needed to export TerminateBufferIO(),
  like, as previously mentioned, already done in an earlier version for
  StartBufferIO().  Nobody commented on that, so I think that's ok.

  I also renamed the tests away from the very inventively named tbl_a, tbl_b...


- Moved the commit to create resownern in more places to much later in the
  queue, it's not actually needed for bufmgr.c IO, and nothing needing it will
  land in 18


- Added a proper commit message fo the main commit. I'd appreciate folks
  reading through it. I'm sure I forgot a lot of folks and a lot of things.


- Did a fair bit of of comment polishing


- Addressed an XXX in the "aio infrastructure" commit suggesting that we might
  want to error out if a backend is waiting on is own unsubmitted IO. Noah
  argued for erroring out. I now made it so.


- Temporarily added a commit to increase open-file limit on openbsd. I saw
  related errors without this patch too, but it fails more often with. I
  already sent a separate email about this.


At this point I am not aware of anything significant left to do in the main
AIO commit, safe some of the questions below.  There's a lot more potential
optimizations etc, but this is already a very complicated piece of work, so I
think they just has to wait for later.

There are a few things to clean up in the bufmgr.c commits, I don't yet quite
like the function naming and there could be a bit less duplication. But I
don't think that needs to be resolved before the main commit.


Questions:

- My current thinking is that we'd set io_method = worker initially - so we
  actually get some coverage - and then decide whether to switch to
  io_method=sync by default for 18 sometime around beta1/2. Does that sound
  reasonable?


- We could reduce memory usage a tiny bit if we made the mapping between
  pgproc and per-backend-aio-state more complicated, i.e. not just indexed by
  ProcNumber. Right now IO workers have the per-backend AIO state, but don't
  actually need it.  I'm mildly inclined to think that the complexity isn't
  worth it, but on the fence.


- Three of the commits in the series really are just precursor commits to
  their subsequent commits, which I found helpful for development and review,
  namely:

  - aio: Basic subsystem initialization
  - aio: Skeleton IO worker infrastructure
  - aio: Add liburing dependency

  Not sure if it's worth keeping these separate or whether they should just be
  merged with their "real commit".


- Thomas suggested renaming
  COMPLETED_IO->COMPLETED,
  COMPLETED_SHARED->TERMINATED_BY_COMPLETER,
  COMPLETED_SHARED->TERMINATED_BY_SUBMITTER
  in
  https://www.postgresql.org/message-id/CA%2BhUKGLxH1tsUgzZfng4BU6GqnS6bKF2ThvxH1_w5c7-sLRKQw%40mail.gmail.com

  While the other things in the email were commented upon by others and
  addressed in v2.4, the naming aspect wasn't further remarked upon by others.
  I'm not personally in love with the suggested names, but I could live with
  them.


- Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
  but all the ereport()s add a noticeable amount of overhead at high IO
  throughput (at multiple gigabytes/second), so that's probably not right
  forever.  I'd leave this on initially and then change it to default to off
  later.  I think that's ok?


- To allow io_workers to be PGC_SIGHUP, and to eventually allow to
  automatically in/decrease active workers, the max number of workers (32) is
  always allocated. That means we use more semaphores than before. I think
  that's ok, it's not 1995 anymore.  Alternatively we can add a
  "io_workers_max" GUC and probe for it in initdb.


- pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
  sure that's great?

  They could be an enum array or such too? That'd perhaps be a bit more
  extensible? OTOH, we don't currently use enums in the catalogs and arrays
  are somewhat annoying to conjure up from C.


Todo:

- A few more passes over the main commit, I'm sure there's a few more inartful
  comments, odd formatting and such.

  - Check if there's a decent way to deduplicate pgaio_io_call_complete_shared() and
    pgaio_io_call_complete_local()


- Figure out how to deduplicate support for LockBufferForCleanup() in
  TerminateBufferIO().


- Documentation for pg_stat_aios.


- Check if documentation for track_io_timing needs to be adjusted, after the
  bufmgr.c changes we only track waiting for an IO.


- Some of the test_aio code is specific to non-temp tables, it probably is
  worth generalizing to deal with temp tables and invoke them for both.

Greetings,

Andres

  62. Re: AIO v2.5

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-03-06T11:36:43Z

    On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:

> Attached is v2.5 of the AIO patchset.
[..]
Hi, Thanks for working on this!

> Questions:
>
> - My current thinking is that we'd set io_method = worker initially - so we
>   actually get some coverage - and then decide whether to switch to
>   io_method=sync by default for 18 sometime around beta1/2. Does that sound
>   reasonable?

IMHO, yes, good idea. Anyway final outcomes partially will depend on
how many other stream-consumers be committed, right?

> - Three of the commits in the series really are just precursor commits to
>   their subsequent commits, which I found helpful for development and review,
>   namely:
>
>   - aio: Basic subsystem initialization
>   - aio: Skeleton IO worker infrastructure
>   - aio: Add liburing dependency
>
>   Not sure if it's worth keeping these separate or whether they should just be
>   merged with their "real commit".

For me it was easier to read those when they are separate.

> - Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
>   but all the ereport()s add a noticeable amount of overhead at high IO
>   throughput (at multiple gigabytes/second), so that's probably not right
>   forever.  I'd leave this on initially and then change it to default to off
>   later.  I think that's ok?

+1, hopefully nothing is recording/logging/running with
log_min_messages>=debug3 because only then it starts to be visible.

> - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
>   automatically in/decrease active workers, the max number of workers (32) is
>   always allocated. That means we use more semaphores than before. I think
>   that's ok, it's not 1995 anymore.  Alternatively we can add a
>   "io_workers_max" GUC and probe for it in initdb.

Wouldn't that matter only on *BSDs?

BTW I somehow cannot imagine someone saturating >= 32 workers (if one
does, better to switch to uring anyway?), but I have a related
question about closing fd by those workers.

> - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
>   sure that's great?
>
>   They could be an enum array or such too? That'd perhaps be a bit more
>   extensible? OTOH, we don't currently use enums in the catalogs and arrays
>   are somewhat annoying to conjure up from C.

s/pg_stat_aios/pg_aios/ ? :^) It looks good to me as it is. Anyway it
is a debugging view - perhaps mark it as such in the docs - so there
is no stable API for that and shouldn't be queried by any software
anyway.

> - Documentation for pg_stat_aios.

pg_aios! :)

So, I've taken aio-2 branch from Your's github repo for a small ride
on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
question: perhaps IO workers should auto-close fd on errors or should
we use SIGUSR2 for it? The scenario is like this:

#dm-dust is not that available even on modern distros(not always
compiled), but flakey seemed to work on 4.18.x:
losetup /dev/loop0 /dd.img
mkfs.ext4 -j /dev/loop0
mkdir /flakey
mount /dev/loop0 /flakey # for now it will work
mkdir /flakey/tblspace
chown postgres /flakey/tblspace
chmod 0700 /flakey/tblspace
CREATE TABLESPACE test1 LOCATION '/flakey/tblspace'
CREATE TABLE on t1fail on that test1 tablespace + INSERT SOME DATA
pg_ctl stop
umount /flakey
echo "0 `blockdev --getsz /dev/loop0` flakey /dev/loop0 0 1 1" |
dmsetup create flakey # after 1s start throwing IO errors
mount /dev/mapper/flakey /flakey
#might even say: mount: /flakey: can't read superblock on /dev/mapper/flakey.
mount /dev/mapper/flakey /flakey
pg_ctl start

and then this will happen:

postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR:  could not read blocks 0..1 in file
"pg_tblspc/24579/PG_18_202503031/5/24586_fsm": Input/output error
postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR:  could not read blocks 0..1 in file
"pg_tblspc/24579/PG_18_202503031/5/24586_fsm": Input/output error
postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR:  could not read blocks 0..1 in file
"pg_tblspc/24579/PG_18_202503031/5/24586_fsm": Input/output error

postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR:  could not open file
"pg_tblspc/24579/PG_18_202503031/5/24586_vm": Read-only file system

so usual stuff with kernel remounting it RO, but here's the dragon
with io_method=worker:

# mount -o remount,rw /flakey/
mount: /flakey: cannot remount /dev/mapper/flakey read-write, is
write-protected.
# umount /flakey # to fsck or just mount rw again
umount: /flakey: target is busy.
# lsof /flakey/
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
postgres 103483 postgres   14u   REG  253,2 36249600   17
/flakey/tblspace/PG_18_202503031/5/24586
postgres 103484 postgres    6u   REG  253,2 36249600   17
/flakey/tblspace/PG_18_202503031/5/24586
postgres 103485 postgres    6u   REG  253,2 36249600   17
/flakey/tblspace/PG_18_202503031/5/24586

Those 10348[345] are IO workers, they have still open fds and there's
no way to close those without restart -- well without close()
injection probably via gdb.   pg_terminate_backend() on those won't
work. The only thing that works seems to be sending SIGUSR2, but is
that safe [there could be some errors after pwrite() ] ? With
io_worker=sync just quitting the backend of course works. Not sure
what your thoughts are because any other bgworker could be having open
fds there. It's a very minor thing. Otherwise that outage of separate
tablespace (rarely used) would potentially cause inability to fsck
there and lower the availability of the DB (due to potential restart
required). I'm thinking especially of scenarios where lots of schemas
are used with lots of tablespaces OR where temp_tablespace is employed
for some dedicated (fast/furious/faulty) device. So I'm hoping SIGUSR2
is enough right (4231f4059e5e54d78c56b904f30a5873da88e163 seems to be
doing it anyway) ?

BTW: While at this, I've tried amcheck/pg_surgery for 1 min and they
both seem to work.

-J.

  63. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-06T13:13:38Z

    Hi,

On 2025-03-06 12:36:43 +0100, Jakub Wartak wrote:
> On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> > Questions:
> >
> > - My current thinking is that we'd set io_method = worker initially - so we
> >   actually get some coverage - and then decide whether to switch to
> >   io_method=sync by default for 18 sometime around beta1/2. Does that sound
> >   reasonable?
>
> IMHO, yes, good idea. Anyway final outcomes partially will depend on
> how many other stream-consumers be committed, right?

I think it's more whether we find cases where it performs substantially worse
with the read stream users that exists.  The behaviour for non-read-stream IO
shouldn't change.


> > - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> >   automatically in/decrease active workers, the max number of workers (32) is
> >   always allocated. That means we use more semaphores than before. I think
> >   that's ok, it's not 1995 anymore.  Alternatively we can add a
> >   "io_workers_max" GUC and probe for it in initdb.
>
> Wouldn't that matter only on *BSDs?

Yea, NetBSD and OpenBSD only, I think.


> > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> >   sure that's great?
> >
> >   They could be an enum array or such too? That'd perhaps be a bit more
> >   extensible? OTOH, we don't currently use enums in the catalogs and arrays
> >   are somewhat annoying to conjure up from C.
>
> s/pg_stat_aios/pg_aios/ ? :^)

Ooops, yes.


> It looks good to me as it is.
> Anyway it
> is a debugging view - perhaps mark it as such in the docs - so there
> is no stable API for that and shouldn't be queried by any software
> anyway.

Cool


> > - Documentation for pg_stat_aios.
>
> pg_aios! :)
>
> So, I've taken aio-2 branch from Your's github repo for a small ride
> on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
> question: perhaps IO workers should auto-close fd on errors or should
> we use SIGUSR2 for it? The scenario is like this:

When you say "auto-close", you mean that one IO error should trigger *all*
workers to close their FDs?


> so usual stuff with kernel remounting it RO, but here's the dragon
> with io_method=worker:
>
> # mount -o remount,rw /flakey/
> mount: /flakey: cannot remount /dev/mapper/flakey read-write, is
> write-protected.
> # umount /flakey # to fsck or just mount rw again
> umount: /flakey: target is busy.
> # lsof /flakey/
> COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> postgres 103483 postgres   14u   REG  253,2 36249600   17
> /flakey/tblspace/PG_18_202503031/5/24586
> postgres 103484 postgres    6u   REG  253,2 36249600   17
> /flakey/tblspace/PG_18_202503031/5/24586
> postgres 103485 postgres    6u   REG  253,2 36249600   17
> /flakey/tblspace/PG_18_202503031/5/24586
>
> Those 10348[345] are IO workers, they have still open fds and there's
> no way to close those without restart -- well without close()
> injection probably via gdb.

The same is already true with bgwriter, checkpointer etc?


> pg_terminate_backend() on those won't work. The only thing that works seems
> to be sending SIGUSR2

Sending SIGINT works.


> , but is that safe [there could be some errors after pwrite() ]?

Could you expand on that?


> With
> io_worker=sync just quitting the backend of course works. Not sure
> what your thoughts are because any other bgworker could be having open
> fds there. It's a very minor thing. Otherwise that outage of separate
> tablespace (rarely used) would potentially cause inability to fsck
> there and lower the availability of the DB (due to potential restart
> required).

I think a crash-restart is the only valid thing to get out of a scenario like
that, independent of AIO:

- If there had been any writes we need to perform crash recovery anyway, to
  recreate those writes
- If there just were reads, it's good to restart as well, as otherwise there
  might be pages in the buffer pool that don't exist on disk anymore, due to
  the errors.

Greetings,

Andres Freund

  64. Re: AIO v2.5

    Robert Haas <robertmhaas@gmail.com> — 2025-03-06T15:33:33Z

    On Tue, Mar 4, 2025 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
> - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
>   sure that's great?

I don't like the name. Pluralization abbreviations is weird, and it's
even weirder when the abbreviation is not one that is universally
known. Maybe just drop the "s".

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  65. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-06T16:53:41Z

    Hi,

On 2025-03-06 10:33:33 -0500, Robert Haas wrote:
> On Tue, Mar 4, 2025 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
> > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> >   sure that's great?
> 
> I don't like the name.

I don't think it changes anything, but as Jakub pointed out, I thinko'd the
name in the email you're responding to, it's pg_aios, not pg_stat_aios.

It shows the currently in-flight IOs, not accumulated statistics about them,
hence no _stat_.

I don't like the name either, I IIRC asked for suggestions elsewhere in the
thread, not a lot was forthcoming, so I left it at pg_aios.


> Pluralization abbreviations is weird, and it's even weirder when the
> abbreviation is not one that is universally known. Maybe just drop the "s".

I went with plural because that's what we have in other views showing the
"current" state:
- pg_cursors
- pg_file_settings
- pg_prepared_statements
- pg_prepared_xacts
- pg_replication_slots
- pg_locks
- ...

But you're right that those aren't abbreviations.

Greetings,

Andres Freund

  66. Re: AIO v2.5

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-03-07T10:21:09Z

    On Thu, Mar 6, 2025 at 2:13 PM Andres Freund <andres@anarazel.de> wrote:

> On 2025-03-06 12:36:43 +0100, Jakub Wartak wrote:
> > On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> > > Questions:
> > >
> > > - My current thinking is that we'd set io_method = worker initially - so we
> > >   actually get some coverage - and then decide whether to switch to
> > >   io_method=sync by default for 18 sometime around beta1/2. Does that sound
> > >   reasonable?
> >
> > IMHO, yes, good idea. Anyway final outcomes partially will depend on
> > how many other stream-consumers be committed, right?
>
> I think it's more whether we find cases where it performs substantially worse
> with the read stream users that exist.  The behaviour for non-read-stream IO
> shouldn't change.

OK, so in order to to get full picture for v18beta this would mean
$thread + following ones?:
- Use read streams in autoprewarm
- BitmapHeapScan table AM violation removal (and use streaming read API)
- Index Prefetching (it seems it has stalled?)

or is there something more planned? (I'm asking what to apply on top
of AIO to minimize number of potential test runs which seem to take
lots of time, so to do it all in one go)

> > So, I've taken aio-2 branch from Your's github repo for a small ride
> > on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
> > question: perhaps IO workers should auto-close fd on errors or should
> > we use SIGUSR2 for it? The scenario is like this:
>
> When you say "auto-close", you mean that one IO error should trigger *all*
> workers to close their FDs?

Yeah I somehow was thinking about such a thing, but after You have
bolded that "*all*", my question sounds much more stupid than it was
yesterday. Sorry for asking stupid question :)

> The same is already true with bgwriter, checkpointer etc?

Yeah.. I was kind of looking for a way of getting "higher
availability" in the presence of partial IO (tablespace) errors.

> > pg_terminate_backend() on those won't work. The only thing that works seems
> > to be sending SIGUSR2
>
> Sending SIGINT works.

Ugh, ok, it looks like I've been overthinking that, cool.

> > , but is that safe [there could be some errors after pwrite() ]?
>
> Could you expand on that?

It is pure speculation on my side: well I'm always concerned about
leaving something out there without cleanup after errors and then
re-using it for something else much later, especially on edge-cases
like NFS or FUSE. In the backend we could maintain some state, but
io_workes are shared across backends. E.g. some pwrite() failing on
NFS, we are not closing that fd, and then reusing it for something
else much latter for different backend (although AFAIK close() does
not guarantee anything, but e.g. it could be that some inode/path or
something was simply marked dangling - the fresh pair of
close()/open() could could could return error, but here we would just
keep on pwriting() there?).

OK the only question remains: does it make sense to try something like
pgbench on NFS UDP mountopt=hard,nointr + intermittent iptables DROP
from time to time , or is it not worth trying?

> > With
> > io_worker=sync just quitting the backend of course works. Not sure
> > what your thoughts are because any other bgworker could be having open
> > fds there. It's a very minor thing. Otherwise that outage of separate
> > tablespace (rarely used) would potentially cause inability to fsck
> > there and lower the availability of the DB (due to potential restart
> > required).
>
> I think a crash-restart is the only valid thing to get out of a scenario like
> that, independent of AIO:
>
> - If there had been any writes we need to perform crash recovery anyway, to
>   recreate those writes
> - If there just were reads, it's good to restart as well, as otherwise there
>   might be pages in the buffer pool that don't exist on disk anymore, due to
>   the errors.

OK, cool, thanks!

-J.

  67. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-07T13:54:16Z

    Hi,

On 2025-03-07 11:21:09 +0100, Jakub Wartak wrote:
> On Thu, Mar 6, 2025 at 2:13 PM Andres Freund <andres@anarazel.de> wrote:
> 
> > On 2025-03-06 12:36:43 +0100, Jakub Wartak wrote:
> > > On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> > > > Questions:
> > > >
> > > > - My current thinking is that we'd set io_method = worker initially - so we
> > > >   actually get some coverage - and then decide whether to switch to
> > > >   io_method=sync by default for 18 sometime around beta1/2. Does that sound
> > > >   reasonable?
> > >
> > > IMHO, yes, good idea. Anyway final outcomes partially will depend on
> > > how many other stream-consumers be committed, right?
> >
> > I think it's more whether we find cases where it performs substantially worse
> > with the read stream users that exist.  The behaviour for non-read-stream IO
> > shouldn't change.
> 
> OK, so in order to to get full picture for v18beta this would mean
> $thread + following ones?:
> - Use read streams in autoprewarm
> - BitmapHeapScan table AM violation removal (and use streaming read API)

Yep.


> - Index Prefetching (it seems it has stalled?)

I don't think there's any chance it'll be in 18. There's a good bit more work
needed before it can go in...


> or is there something more planned? (I'm asking what to apply on top
> of AIO to minimize number of potential test runs which seem to take
> lots of time, so to do it all in one go)

I think there may be some more (e.g. btree index vacuuming), but I don't think
they'll have *that* big an impact.


> > > So, I've taken aio-2 branch from Your's github repo for a small ride
> > > on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
> > > question: perhaps IO workers should auto-close fd on errors or should
> > > we use SIGUSR2 for it? The scenario is like this:
> >
> > When you say "auto-close", you mean that one IO error should trigger *all*
> > workers to close their FDs?
> 
> Yeah I somehow was thinking about such a thing, but after You have
> bolded that "*all*", my question sounds much more stupid than it was
> yesterday. Sorry for asking stupid question :)

Don't worry about that :)


> > The same is already true with bgwriter, checkpointer etc?
> 
> Yeah.. I was kind of looking for a way of getting "higher
> availability" in the presence of partial IO (tablespace) errors.

I'm really doubtful that's that worthwhile to pursue. IME the system is pretty
much hosed once this starts to happening and it's often made *worse* by trying
to limp along.


> OK the only question remains: does it make sense to try something like
> pgbench on NFS UDP mountopt=hard,nointr + intermittent iptables DROP
> from time to time , or is it not worth trying?

I don't think it's particularly interesting. But then I'd *never* trust any
meaningful data to a PG running on NFS.


Greetings,

Andres Freund

  68. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-07T21:23:51Z

    Hi,

On 2025-03-06 11:53:41 -0500, Andres Freund wrote:
> On 2025-03-06 10:33:33 -0500, Robert Haas wrote:
> > On Tue, Mar 4, 2025 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
> > > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> > >   sure that's great?
> > 
> > I don't like the name.
> 
> I don't think it changes anything, but as Jakub pointed out, I thinko'd the
> name in the email you're responding to, it's pg_aios, not pg_stat_aios.
> 
> It shows the currently in-flight IOs, not accumulated statistics about them,
> hence no _stat_.
> 
> I don't like the name either, I IIRC asked for suggestions elsewhere in the
> thread, not a lot was forthcoming, so I left it at pg_aios.

What about pg_io_handles?

Greetings,

Andres Freund

  69. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-08T02:11:15Z

    Hi,

Tom, CCed you since you have worked most on elog.c


On 2025-03-07 16:23:51 -0500, Andres Freund wrote:
> What about pg_io_handles?

While looking at the view I felt motivated to tackle the one FIXME in the
implementation of the view. Namely that the "error_desc" column wasn't
populated (the view did show that there was an error, but not what the error
was).

Which lead me down a sad sad rabbit hole, largely independent of AIO.


A bit of background:

For AIO completion callbacks can signal errors (e.g. a page header failing
validation). That error can be logged in the callback and/or raised later,
e.g. by the query that issued the IO.

AIO callbacks happen in critical sections, which is required to be able to use
AIO for WAL (see README.md for more details).

Currently errors are logged/raised by ereport()s in functions that gets passed
in an elevel, pretty standard.

A few of the ereports() use errcode_for_file_access() to translate an errno to
an sqlerrcode.


Now on to the problem:

The result of an ereport() can't be put into a view, obviously. I didn't think
it'd be good if the each kind of error needed to be implemented twice, once
with ereport() and once to just return a string to put in the view.


I tried a few things:

1) Use errsave() to allow delayed reporting of the error

I encountered a few problems:

- errsave() doesn't allow the log level to be specified, which means it can't
  directly be used to LOG if no context is specified.

  This could be worked around by always specifying the context, with
  ErrorSaveContext.details_wanted = true and having generic code that changes
  the elevel to whatever is appropriate and then using ThrowErrorData() to log the
  message.

- ersave_start() sets assoc_context to CurrentMemoryContext and
  errsave_finish() allocates an ErrorData copy in CurrentMemoryContext

  This makes naive use of this approach when logging in a critical section
  impossible. If ErrorSaveContext is not passed in an ERROR will be raised,
  even if we just want to log.  If ErrorSaveContext is used, we allocate
  memory in the caller context, which isn't allowed in a critical section.

  The only way I saw to work around that was to switch to ErrorContext before
  calling errsave(). That's doable, the logging is called from one function
  (pgaio_result_report()). That kinda works, but as a consequence we more than
  double the memory usage in ErrorContext as errsave_finish() will palloc a
  new ErrorData and ThrowErrorData() copies that ErrorData and all its string
  back to ErrorContext.


2) Have the error callback format the error using a helper function instead of
   using ereport()

Problems:

- errcode_for_file_access() would need to be reimplemented / split into a
  function translating an errnode into an sqlerrcode without getting it from
  the error data stack

- emitting the log message in a critical section would require either doing
  the error formatting in ErrorContext or creating another context with
  reserved memory to do so.

- allowing to specify DETAIL, HINT etc basically requires a small elog.c
  interface reimplementation


3) Use pre_format_elog_string(), format_elog_string() similar to what guc.c
   does for check hooks, via GUC_check_errmsg(), GUC_check_errhint() ...

Problems:

- Requires to duplicate errcode_for_file_access() for similar reason as in 2)

- Not exactly pretty

- Somewhat gnarly, but doable, to make use of %m safe, the way it's done in
  guc.h afaict isn't safe:
  pre_format_elog_string() is called for each of
  GUC_check_{errmsg,errdetail,errhint}. As the global errno might get set
  during the format_elog_string(), it'll not be the right one during
  the next GUC_check_*.



4) Don't use ereport() directly, but instead put the errstart() in
   pgaio_result_report(), before calling the error description callback.

   When emitting a log message, call errfinish() after the callback. For the
   view, get the message out via CopyErrorData() and free the memory again
   using FlushErrorState

Problems:

- Seems extremely hacky


I implemented all, but don't really like any of them.


Unless somebody has a better idea or we agree that one of the above is
actually a acceptable approach, I'm inclined to simply remove the column
containing the description of the error. The window in which one could see an
IO with an error is rather short most of the time anyway and the error will
also be logged.

It's a bit annoying that adding the column later would require revising the
signature of the error reporting callback at that time, but I think that
degree of churn is acceptable.


The main reason I wanted to write this up is that it seems that we're just
lacking some infrastructure here.

Greetings,

Andres Freund

  70. Re: AIO v2.5

    Tom Lane <tgl@sss.pgh.pa.us> — 2025-03-09T17:51:47Z

    Andres Freund <andres@anarazel.de> writes:
> While looking at the view I felt motivated to tackle the one FIXME in the
> implementation of the view. Namely that the "error_desc" column wasn't
> populated (the view did show that there was an error, but not what the error
> was).

> Which lead me down a sad sad rabbit hole, largely independent of AIO.

> ...

> The main reason I wanted to write this up is that it seems that we're just
> lacking some infrastructure here.

Maybe.  The mention of elog.c in the same breath with critical
sections is already enough to scare me; we surely daren't invoke
gettext() in a critical section, for instance.  I feel the most
we could hope for here is to report a constant string that would
not get translated till later, outside the critical section.
That seems less about infrastructure and more about how the AIO
error handling/reporting code is laid out.  In the meantime,
if leaving the error out of this view is enough to make the problem
go away, let's do that.

			regards, tom lane

  71. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-10T18:23:12Z

    Hi,

Attached is v2.6 of the AIO patchset.

Relative to 2.5 I:

- Improved the split between subsystem initialization and main AIO commit, as
  well as the one between worker infrastructure and io_method=worker

  Seemed worth as the only one voicing an opinion about squashing those
  commits was opposed.


- Added a lot more comments to aio.h/aio_internal.h. I think just about
  anything that should conceivably have a comment has one.


- Reordered fields in PgAioHandle to waste less due to padding


- Narrowed a few *count fields, they were 64bit without ever being al to reach
  that


- Used aio_types.h more widely, instead of "manual" forward declarations. This
  required moving a few typedefs to aio_types.h


- Substantial commit message improvements.


- Removed the pg_aios.error_desc column, due to:
  https://postgr.es/m/qzxq6mqqozctlfcg2kg5744gmyubicvuehnp4a7up472thlvz2%40y5xqgd5wcwhw


- Reordered the commits slightly, to put the README just after the
  smgr.c/md.c/... support, as the readme references those in the examples


- Stopped creating backend-local io_uring instances, that is vestigial for
  now. We likely will want to reintroduce them at some point (e.g. for network
  IO), but we can do that at that time.


- There were a lot of duplicated codepaths in bufmgr.c support for AIO due to
  temp tables. I added a few commits refactoring the temp buffers state
  management to look a lot more like the shared buffer code.

  I'm not sure that that's the best path, but they all seemed substantial
  improvements on their own.


- putting io_method in PG_TEST_INITDB_EXTRA_OPTS previously broke a test,
  because Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the # options
  specified by ->extra. I now worked around that by appending the io method to
  a local PG_TEST_INITDB_EXTRA_OPTS, but brrr.


- The tracepoint for read completion omitted the fact that it was a temp
  table, if so.


- Fixed some duplicated function decls, due to a misresolved merge-conflict


Current state:

- 0001, 0002 - core AIO - IMO pretty much ready

- 0003, 0004 - IO worker - same

- 0005, 0006 - io_uring support - close, but we need to do something about
  set_max_fds(), which errors out spuriously in some cases

- 0007 - smgr/md/fd.c readv support - seems quite close, but might benefit from
  another pass through

- 0008 - README - I think it's good, but I'm probably not seeing the trees for
  the forest anymore

- 0009 - pg_aios view - naming not resolved, docs missing

- 0010 to 0014 - from another thread, just included here due to a dependency

- 0016 to 0020 - cleanups for temp buffers code - I just wrote these to clean
  up the code before making larger changes, needs review

- 0021 - keep BufferDesc refcount up2date for temp buffers - I think that's
  pretty much ready, but depends on earlier patches

- 0022 - bufmgr readv AIO suppot - some naming, some code duplication needs to
  be resolved, but otherwise quite close

- 0023 - use AIO in StartReadBuffers() - perhaps a bit of polishing needed

- 0024 - adjust read_stream.c for AIO - I think Thomas has a better patch for
  this in the works

- 0025 - tests for AIO - I think it's reasonable, unless somebody objects to
  exporting a few bufmgr.c functions to the test

- the rest: Not for 18

Greetings,

Andres Freund

  72. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-11T15:31:18Z

    On Mon, Mar 10, 2025 at 2:23 PM Andres Freund <andres@anarazel.de> wrote:
>
> - 0016 to 0020 - cleanups for temp buffers code - I just wrote these to clean
>   up the code before making larger changes, needs review

This is a review of 0016-0020

Commit messages for 0017-0020 are thin. I assume you will beef them up
a bit before committing. Really, though, those matter much less than
0016 which is an actual bug (or pre-bug) fix. I called out the ones
where I think you should really consider adding more detail to the
commit message.

0016:

      * the case, write it out before reusing it!
      */
-    if (buf_state & BM_DIRTY)
+    if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
     {
+        uint32        buf_state = pg_atomic_read_u32(&bufHdr->state);

I don't love that you fetch in the if statement and inside the if
statement. You wouldn't normally do this, so it sticks out. I get that
you want to avoid having the problem this commit fixes again, but
maybe it is worth just fetching the buf_state above the if statement
and adding a comment that it could have changed so you must do that.
Anyway, I think your future patches make the local buf_state variable
in this function obsolete, so perhaps it doesn't matter.

Not related to this patch, but while reading this code, I noticed that
this line of code is really weird:
        LocalBufHdrGetBlock(bufHdr) = GetLocalBufferStorage();
I actually don't understand what it is doing ... setting the result of
the macro to the result of GetLocalBufferStorage()? I haven't seen
anything like that before.

Otherwise, this patch LGTM.

0017:

+++ b/src/backend/storage/buffer/localbuf.c
@@ -56,6 +56,7 @@ static int    NLocalPinnedBuffers = 0;
 static Buffer GetLocalVictimBuffer(void);
+static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);

Technically this line is too long

+ * InvalidateLocalBuffer -- mark a local buffer invalid.
+ *
+ * If check_unreferenced is true, error out if the buffer is still
+ * used. Passing false is appropriate when redesignating the buffer instead
+ * dropping it.
+ *
+ * See also InvalidateBuffer().
+ */
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
+{

I was on the fence about the language "buffer is still used", since
this is about the ref count and not the usage count. If this is the
language used elsewhere perhaps it is fine.

I also was not sure what redesignate means here. If you mean to use
this function in the future in other contexts than eviction and
dropping buffers, fine. But otherwise, maybe just use a more obvious
word (like eviction).

0018:

Compiler now warns that buf_state is unused in GetLocalVictimBuffer().

@@ -4564,8 +4548,7 @@ FlushRelationBuffers(Relation rel)
                                         IOCONTEXT_NORMAL, IOOP_WRITE,
                                         io_start, 1, BLCKSZ);

-                buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
-                pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+                TerminateLocalBufferIO(bufHdr, true, 0);

FlushRelationBuffers() used to clear BM_JUST_DIRTIED, which it seems
like wouldn't have been applicable to local buffers before, but,
actually with async IO could perhaps happen in the future? Anyway,
TerminateLocalBuffers() doesn't clear that flag, so you should call
that out if it was intentional.

@@ -5652,8 +5635,11 @@ TerminateBufferIO(BufferDesc *buf, bool
clear_dirty, uint32 set_flag_bits,
+    buf_state &= ~BM_IO_IN_PROGRESS;
+    buf_state &= ~BM_IO_ERROR;
-    buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);

Is it worth mentioning in the commit message that you made a cosmetic
change to TerminateBufferIO()?

0019:
LGTM

0020:
This commit message is probably tooo thin. I think you need to at
least say something about this being used by AIO in the future. Out of
context of this patch set, it will be confusing.

+/*
+ * Like StartBufferIO, but for local buffers
+ */
+bool
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
+{

I think you could use a comment about why nowait might be useful for
local buffers in the future. It wouldn't make sense with synchronous
I/O, so it feels a bit weird without any comment.

+    if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
+    {
+        /* someone else already did the I/O */
+        UnlockBufHdr(bufHdr, buf_state);
+        return false;
+    }

UnlockBufHdr() explicitly says it should not be called for local
buffers. I know that code is unreachable right now, but it doesn't
feel quite right. I'm not sure what the architecture of AIO local
buffers will be like, but if other processes can't access these
buffers, I don't know why you would need BM_LOCKED. And if you will, I
think you need to edit the UnlockBufHdr() comment.

@@ -1450,13 +1450,11 @@ static inline bool
 WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
 {
     if (BufferIsLocal(buffer))
     else
-        return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
+        return StartBufferIO(GetBufferDescriptor(buffer - 1),
+                             true, nowait);

I'm not sure it is worth the diff in non-local buffer case to reflow
this. It is already confusing enough in this patch that you are adding
some code that is mostly unneeded.

- Melanie

  73. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-11T17:56:53Z

    Hi,

On 2025-03-11 11:31:18 -0400, Melanie Plageman wrote:
> On Mon, Mar 10, 2025 at 2:23 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > - 0016 to 0020 - cleanups for temp buffers code - I just wrote these to clean
> >   up the code before making larger changes, needs review
> 
> This is a review of 0016-0020
> 
> Commit messages for 0017-0020 are thin. I assume you will beef them up
> a bit before committing.

Yea. I wanted to get some feedback on whether these refactorings are a good
idea or not...


> Really, though, those matter much less than 0016 which is an actual bug (or
> pre-bug) fix. I called out the ones where I think you should really consider
> adding more detail to the commit message.
> 
> 0016:

Do you think we should backpatch that change? It's not really an active bug in
16+, but it's also not quite right. The other changes surely shouldn't be
backpatched...


>       * the case, write it out before reusing it!
>       */
> -    if (buf_state & BM_DIRTY)
> +    if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
>      {
> +        uint32        buf_state = pg_atomic_read_u32(&bufHdr->state);
> 
> I don't love that you fetch in the if statement and inside the if
> statement. You wouldn't normally do this, so it sticks out. I get that
> you want to avoid having the problem this commit fixes again, but
> maybe it is worth just fetching the buf_state above the if statement
> and adding a comment that it could have changed so you must do that.

It's seems way too easy to introduce new similar breakages if the scope of
buf_state is that wide - I yesterday did waste 90min because I did in another
similar place. The narrower scopes make that much less likely to be a problem.


> Anyway, I think your future patches make the local buf_state variable
> in this function obsolete, so perhaps it doesn't matter.

Leaving the defensive-programming aspect aside, it does seem like a better
intermediary state to me to have the local vars than to have to change more
lines when introducing FlushLocalBuffer() etc.


> Not related to this patch, but while reading this code, I noticed that
> this line of code is really weird:
>         LocalBufHdrGetBlock(bufHdr) = GetLocalBufferStorage();
> I actually don't understand what it is doing ... setting the result of
> the macro to the result of GetLocalBufferStorage()? I haven't seen
> anything like that before.

Yes, that's what it's doing. LocalBufferBlockPointers() evaluates to a value
that can be used as an lvalue in an assignment.

Not exactly pretty...


> Otherwise, this patch LGTM.
> 
> 0017:
> 
> +++ b/src/backend/storage/buffer/localbuf.c
> @@ -56,6 +56,7 @@ static int    NLocalPinnedBuffers = 0;
>  static Buffer GetLocalVictimBuffer(void);
> +static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
> 
> Technically this line is too long

Oh, do I love our line length limits. But, um, is it actually too long? It's
78 chars, which is exactly our limit, I think?


> + * InvalidateLocalBuffer -- mark a local buffer invalid.
> + *
> + * If check_unreferenced is true, error out if the buffer is still
> + * used. Passing false is appropriate when redesignating the buffer instead
> + * dropping it.
> + *
> + * See also InvalidateBuffer().
> + */
> +static void
> +InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
> +{
> 
> I was on the fence about the language "buffer is still used", since
> this is about the ref count and not the usage count. If this is the
> language used elsewhere perhaps it is fine.

I'll change it to "still pinned"
I guess I can make it "still pinned".


> I also was not sure what redesignate means here. If you mean to use
> this function in the future in other contexts than eviction and
> dropping buffers, fine. But otherwise, maybe just use a more obvious
> word (like eviction).

I was trying to reference changing the identity of the buffer as part of
buffer replacement, where we keep a pin to the buffer. Compared to the use of
InvalidateLocalBuffer() in DropRelationAllLocalBuffers() /
DropRelationLocalBuffers().

/*
 * InvalidateLocalBuffer -- mark a local buffer invalid.
 *
 * If check_unreferenced is true, error out if the buffer is still
 * pinned. Passing false is appropriate when calling InvalidateLocalBuffer()
 * as part of changing the identity of a buffer, instead of just dropping the
 * buffer.
 *
 * See also InvalidateBuffer().
 */


> 0018:
> 
> Compiler now warns that buf_state is unused in GetLocalVictimBuffer().

Oops. Missed that because it was then removed in a later commit...


> @@ -4564,8 +4548,7 @@ FlushRelationBuffers(Relation rel)
>                                          IOCONTEXT_NORMAL, IOOP_WRITE,
>                                          io_start, 1, BLCKSZ);
> 
> -                buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
> -                pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
> +                TerminateLocalBufferIO(bufHdr, true, 0);
> 
> FlushRelationBuffers() used to clear BM_JUST_DIRTIED, which it seems
> like wouldn't have been applicable to local buffers before, but,
> actually with async IO could perhaps happen in the future? Anyway,
> TerminateLocalBuffers() doesn't clear that flag, so you should call
> that out if it was intentional.

I think it'd be good to start using BM_JUST_DIRTIED, even if just to make the
code between local and shared buffers more similar. But that that's better
done separately.

I don't know why FlushRelationBuffers cleared it, it's neer set at the moment.

I'll add a note to the commit message.


> @@ -5652,8 +5635,11 @@ TerminateBufferIO(BufferDesc *buf, bool
> clear_dirty, uint32 set_flag_bits,
> +    buf_state &= ~BM_IO_IN_PROGRESS;
> +    buf_state &= ~BM_IO_ERROR;
> -    buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);
> 
> Is it worth mentioning in the commit message that you made a cosmetic
> change to TerminateBufferIO()?

Doesn't really seem worth calling out, but if you think it should, I will.


> 0020:
> This commit message is probably tooo thin. I think you need to at
> least say something about this being used by AIO in the future. Out of
> context of this patch set, it will be confusing.

Yep.


> +/*
> + * Like StartBufferIO, but for local buffers
> + */
> +bool
> +StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
> +{
> 
> I think you could use a comment about why nowait might be useful for
> local buffers in the future. It wouldn't make sense with synchronous
> I/O, so it feels a bit weird without any comment.

Hm, fair point. Another approach would be to defer adding the argument to a
later patch, it doesn't need to be added here.


> +    if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
> +    {
> +        /* someone else already did the I/O */
> +        UnlockBufHdr(bufHdr, buf_state);
> +        return false;
> +    }
> 
> UnlockBufHdr() explicitly says it should not be called for local
> buffers. I know that code is unreachable right now, but it doesn't
> feel quite right. I'm not sure what the architecture of AIO local
> buffers will be like, but if other processes can't access these
> buffers, I don't know why you would need BM_LOCKED. And if you will, I
> think you need to edit the UnlockBufHdr() comment.

You are right, this is a bug in my change. I started with a copy of
StartBufferIO() and whittled it down insufficiently. Thanks for catching that!

Wonder if we should add an assert against this to UnlockBufHdr()...


> @@ -1450,13 +1450,11 @@ static inline bool
>  WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
>  {
>      if (BufferIsLocal(buffer))
>      else
> -        return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
> +        return StartBufferIO(GetBufferDescriptor(buffer - 1),
> +                             true, nowait);
> 
> I'm not sure it is worth the diff in non-local buffer case to reflow
> this. It is already confusing enough in this patch that you are adding
> some code that is mostly unneeded.

Heh, you're right. I had to add a line break in the StartLocalBufferIO() and
it looked wrong to have the two lines formatted differently :)


Thanks for the review!

Greetings,

Andres Freund

  74. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-11T18:28:35Z

    On Tue, Mar 11, 2025 at 1:56 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-03-11 11:31:18 -0400, Melanie Plageman wrote:
> > Commit messages for 0017-0020 are thin. I assume you will beef them up
> > a bit before committing.
>
> Yea. I wanted to get some feedback on whether these refactorings are a good
> idea or not...

I'd say yes, they seem like a good idea.

> > Really, though, those matter much less than 0016 which is an actual bug (or
> > pre-bug) fix. I called out the ones where I think you should really consider
> > adding more detail to the commit message.
> >
> > 0016:
>
> Do you think we should backpatch that change? It's not really an active bug in
> 16+, but it's also not quite right. The other changes surely shouldn't be
> backpatched...

I don't feel strongly about it. PinLocalBuffer() is passed with
adjust_usagecount false and we have loads of other places where things
would just not work if we changed the boolean flag passed in to a
function called by it (bgwriter and SyncOneBuffer() with
skip_recently_used comes to mind).

On the other hand it's a straightforward fix that only needs to be
backpatched a couple versions, so it definitely doesn't hurt.

> > +++ b/src/backend/storage/buffer/localbuf.c
> > @@ -56,6 +56,7 @@ static int    NLocalPinnedBuffers = 0;
> >  static Buffer GetLocalVictimBuffer(void);
> > +static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
> >
> > Technically this line is too long
>
> Oh, do I love our line length limits. But, um, is it actually too long? It's
> 78 chars, which is exactly our limit, I think?

Teccchnically it's 79, which is why it showed up for me with this
handy line from the committing wiki page

git diff origin/master -- src/backend/storage/buffer/localbuf.c | grep
-E '^(\+|diff)' | sed 's/^+//' | expand -t4 | awk "length > 78 ||
/^diff/"

But anyway, it doesn't really matter. I only mentioned it because I
noticed it visually looked long.

> > +    if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
> > +    {
> > +        /* someone else already did the I/O */
> > +        UnlockBufHdr(bufHdr, buf_state);
> > +        return false;
> > +    }
> >
> > UnlockBufHdr() explicitly says it should not be called for local
> > buffers. I know that code is unreachable right now, but it doesn't
> > feel quite right. I'm not sure what the architecture of AIO local
> > buffers will be like, but if other processes can't access these
> > buffers, I don't know why you would need BM_LOCKED. And if you will, I
> > think you need to edit the UnlockBufHdr() comment.
>
> You are right, this is a bug in my change. I started with a copy of
> StartBufferIO() and whittled it down insufficiently. Thanks for catching that!
>
> Wonder if we should add an assert against this to UnlockBufHdr()...

Yea, I think that makes sense.

- Melanie

  75. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-11T19:41:08Z

    On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > For non-sync IO methods, I gather it's essential that a process other than the
> > IO definer be scanning for incomplete IOs and completing them.

> > Otherwise, deadlocks like this would happen:
> 
> > backend1 locks blk1 for non-IO reasons
> > backend2 locks blk2, starts AIO write
> > backend1 waits for lock on blk2 for non-IO reasons
> > backend2 waits for lock on blk1 for non-IO reasons
> >
> > If that's right, in worker mode, the IO worker resolves that deadlock.  What
> > resolves it under io_uring?  Another process that happens to do
> > pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> > make that happen systematically.
> 
> Yea, it's code that I haven't forward ported yet. I think basically
> LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
> immediately acquire the lock and if the buffer has IO going on.

I'm not finding that code in v2.6.  What function has it?


[I wrote a bunch of the subsequent comments against v2.5.  I may have missed
instances of v2.6 obsoleting them.]

On Tue, Mar 04, 2025 at 02:00:14PM -0500, Andres Freund wrote:
> Attached is v2.5 of the AIO patchset.

> - Added a proper commit message fo the main commit. I'd appreciate folks
>   reading through it. I'm sure I forgot a lot of folks and a lot of things.

Commit message looks fine.

> At this point I am not aware of anything significant left to do in the main
> AIO commit, safe some of the questions below.

That is a big milestone.

> Questions:
> 
> - My current thinking is that we'd set io_method = worker initially - so we
>   actually get some coverage - and then decide whether to switch to
>   io_method=sync by default for 18 sometime around beta1/2. Does that sound
>   reasonable?

Yes.

> - We could reduce memory usage a tiny bit if we made the mapping between
>   pgproc and per-backend-aio-state more complicated, i.e. not just indexed by
>   ProcNumber. Right now IO workers have the per-backend AIO state, but don't
>   actually need it.  I'm mildly inclined to think that the complexity isn't
>   worth it, but on the fence.

The max memory savings, for 32 IO workers, is like the difference between
max_connections=500 and max_connections=532, right?  If that's right, I
wouldn't bother in the foreseeable future.

> - Three of the commits in the series really are just precursor commits to
>   their subsequent commits, which I found helpful for development and review,
>   namely:
> 
>   - aio: Basic subsystem initialization
>   - aio: Skeleton IO worker infrastructure
>   - aio: Add liburing dependency
> 
>   Not sure if it's worth keeping these separate or whether they should just be
>   merged with their "real commit".

The split aided my review.  It's trivial to turn an unmerged stack of commits
into the merged equivalent, but unmerging is hard.

> - Thomas suggested renaming
>   COMPLETED_IO->COMPLETED,
>   COMPLETED_SHARED->TERMINATED_BY_COMPLETER,
>   COMPLETED_SHARED->TERMINATED_BY_SUBMITTER
>   in
>   https://www.postgresql.org/message-id/CA%2BhUKGLxH1tsUgzZfng4BU6GqnS6bKF2ThvxH1_w5c7-sLRKQw%40mail.gmail.com
> 
>   While the other things in the email were commented upon by others and
>   addressed in v2.4, the naming aspect wasn't further remarked upon by others.
>   I'm not personally in love with the suggested names, but I could live with
>   them.

I, too, could live with those.  None of these naming proposals bother me, and
I would not have raised the topic myself.  If I were changing it further, I'd
use these principles:

- use COMPLETED or TERMINATED, not both
- I like COMPLETED, because _complete_ works well in a function name.
  _terminate_ sounds more like an abnormal interruption.
- If one state name lacks a suffix, it should be the final state.

So probably one of:

{COMPLETED,TERMINATED,FINISHED,REAPED,DONE}_{KERN,RETURN,RETVAL,ERRNO}
{COMPLETED,TERMINATED,FINISHED,REAPED,DONE}_{SHMEM,SHARED}
{COMPLETED,TERMINATED,FINISHED,REAPED,DONE}{_SUBMITTER,}

If it were me picking today, I'd pick:

COMPLETED_RETURN
COMPLETED_SHMEM
COMPLETED

> - Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
>   but all the ereport()s add a noticeable amount of overhead at high IO
>   throughput (at multiple gigabytes/second), so that's probably not right
>   forever.  I'd leave this on initially and then change it to default to off
>   later.  I think that's ok?

Sure.  Perhaps make it depend on USE_ASSERT_CHECKING later?

> - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
>   automatically in/decrease active workers, the max number of workers (32) is
>   always allocated. That means we use more semaphores than before. I think
>   that's ok, it's not 1995 anymore.  Alternatively we can add a
>   "io_workers_max" GUC and probe for it in initdb.

Let's start as you have it.  If someone wants to make things perfect for
non-root BSD users, they can add the GUC later.  io_method=sync is a
sufficient backup plan indefinitely.

> - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
>   sure that's great?
> 
>   They could be an enum array or such too? That'd perhaps be a bit more
>   extensible? OTOH, we don't currently use enums in the catalogs and arrays
>   are somewhat annoying to conjure up from C.

An enum array does seem elegant and extensible, but it has the problems you
say.  (I would expect to lose time setting up pg_enum.oid values to not change
between releases.)  A possible compromise would be a text array like
heap_tuple_infomask_flags() does.  Overall, I'm not seeing a clear need to
change away from the bool columns.

> Todo:

> - Figure out how to deduplicate support for LockBufferForCleanup() in
>   TerminateBufferIO().

Yes, I agree there's an opportunity for a WakePinCountWaiter() or similar
subroutine.

> - Check if documentation for track_io_timing needs to be adjusted, after the
>   bufmgr.c changes we only track waiting for an IO.

Yes.


On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> Attached is v2.6 of the AIO patchset.

> - 0005, 0006 - io_uring support - close, but we need to do something about
>   set_max_fds(), which errors out spuriously in some cases

What do we know about those cases?  I don't see a set_max_fds(); is that
set_max_safe_fds(), or something else?

> - 0025 - tests for AIO - I think it's reasonable, unless somebody objects to
>   exporting a few bufmgr.c functions to the test

I'll essentially never object to that.


> +	 * AIO handles need be registered in critical sections and therefore
> +	 * cannot use the normal ResoureElem mechanism.

s/ResoureElem/ResourceElem/

> +      <varlistentry id="guc-io-method" xreflabel="io_method">
> +       <term><varname>io_method</varname> (<type>enum</type>)
> +       <indexterm>
> +        <primary><varname>io_method</varname> configuration parameter</primary>
> +       </indexterm>
> +       </term>
> +       <listitem>
> +        <para>
> +         Selects the method for executing asynchronous I/O.
> +         Possible values are:
> +         <itemizedlist>
> +          <listitem>
> +           <para>
> +            <literal>sync</literal> (execute asynchronous I/O synchronously)

The part in parentheses reads like a contradiction to me.  How about phrasing
it like one of these:

  (execute I/O synchronously, even I/O eligible for asynchronous execution)
  (execute asynchronous-eligible I/O synchronously)
  (execute I/O synchronously, even when asynchronous execution was feasible)

> + * This could be in aio_internal.h, as it is not pubicly referenced, but

typo -> publicly

> + * On what is IO being performed.

End sentence with question mark, probably.

> +	 * List of in-flight IOs. Also contains IOs that aren't strict speaking

s/strict/strictly/

> +	/*
> +	 * Start executing passed in IOs.
> +	 *
> +	 * Will not be called if ->needs_synchronous_execution() returned true.
> +	 *
> +	 * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
> +	 *

I recommend adding "Always called in a critical section." since at least
pgaio_worker_submit() subtly needs it.

> +	 */
> +	int			(*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);

> + * Each backend can only have one AIO handle that that has been "handed out"

s/that that/that/

> + * AIO, it typically will pass the handle to smgr., which will pass it on to

s/smgr.,/smgr.c,/ or just "smgr"

> +PgAioHandle *
> +pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
> +{
> +	if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
> +	{
> +		Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
> +		pgaio_submit_staged();

I'm seeing the "num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE" case uncovered in a
check-world coverage report.  I tried PGAIO_SUBMIT_BATCH_SIZE=2,
io_max_concurrency=1, and io_max_concurrency=64.  Do you already have a recipe
for reaching this case?

> +/*
> + * Stage IO for execution and, if necessary, submit it immediately.
> + *
> + * Should only be called from pgaio_io_prep_*().
> + */
> +void
> +pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
> +{

We've got closely-associated verbs "prepare", "prep", and "stage".  README.md
doesn't mention "stage".  Can one of the following two changes happen?

- README.md starts mentioning "stage" and how it differs from the others
- Code stops using "stage"

> +	 * locallbacks just before reclaiming at multiple callsites.

s/locallbacks/local callbacks/

> + * Check if the the referenced IO completed, without blocking.

s/the the/the/

> + * Batch submission mode needs to explicitly ended with
> + * pgaio_exit_batchmode(), but it is allowed to throw errors, in which case
> + * error recovery will end the batch.

This sentence needs some grammar help, I think.  Maybe use:

 * End batch submission mode with pgaio_exit_batchmode().  (Throwing errors is
 * allowed; error recovery will end the batch.)

>  Size
>  AioShmemSize(void)
>  {
>  	Size		sz = 0;
>  
> +	/*
> +	 * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
> +	 * However, if the DBA explicitly set wal_buffers = -1 in the config file,

s/wal_buffers/io_max_concurrency/

> +extern int	io_workers;

By the rule that GUC vars are PGDLLIMPORT, this should be PGDLLIMPORT.

> +static void
> +maybe_adjust_io_workers(void)

This also restarts workers that exit, so perhaps name it
start_io_workers_if_missing().

> +{
...
> +		/* Try to launch one. */
> +		child = StartChildProcess(B_IO_WORKER);
> +		if (child != NULL)
> +		{
> +			io_worker_children[id] = child;
> +			++io_worker_count;
> +		}
> +		else
> +			break;				/* XXX try again soon? */

Can LaunchMissingBackgroundProcesses() become the sole caller of this
function, replacing the current mix of callers?  That would be more conducive
to promptly doing the right thing after launch failure.

> --- a/src/backend/utils/init/miscinit.c
> +++ b/src/backend/utils/init/miscinit.c
> @@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
>  		case B_CHECKPOINTER:
>  			backendDesc = gettext_noop("checkpointer");
>  			break;
> +		case B_IO_WORKER:
> +			backendDesc = "io worker";

Wrap in gettext_noop() like B_CHECKPOINTER does.

> +         Only has an effect if <xref linkend="guc-max-wal-senders"/> is set to
> +         <literal>worker</literal>.

s/guc-max-wal-senders/guc-io-method/

> + * of IOs, wakeups "fan out"; each woken IO worker can wake two more.  qXXX

s/qXXX/XXX/

> +			/*
> +			 * It's very unlikely, but possible, that reopen fails. E.g. due
> +			 * to memory allocations failing or file permissions changing or
> +			 * such.  In that case we need to fail the IO.
> +			 *
> +			 * There's not really a good errno we can report here.
> +			 */
> +			error_errno = ENOENT;

Agreed there's not a good errno, but let's use a fake errno that we're mighty
unlikely to confuse with an actual case of libc returning that errno.  Like
one of EBADF or EOWNERDEAD.

> +	for (int contextno = 0; contextno < TotalProcs; contextno++)
> +	{
> +		PgAioUringContext *context = &pgaio_uring_contexts[contextno];
> +		int			ret;
> +
> +		/*
> +		 * XXX: Probably worth sharing the WQ between the different rings,
> +		 * when supported by the kernel. Could also cause additional
> +		 * contention, I guess?
> +		 */
> +#if 0
> +		if (!AcquireExternalFD())
> +			elog(ERROR, "No external FD available");
> +#endif

Probably remove the "#if 0" or add a comment on why it's here.

> +		ret = io_uring_submit(uring_instance);
> +		pgstat_report_wait_end();
> +
> +		if (ret == -EINTR)
> +		{
> +			pgaio_debug(DEBUG3,
> +						"aio method uring: submit EINTR, nios: %d",
> +						num_staged_ios);
> +		}
> +		else if (ret < 0)
> +			elog(PANIC, "failed: %d/%s",
> +				 ret, strerror(-ret));

I still think (see 2024-09-16 review) EAGAIN should do the documented
recommendation instead of PANIC:

  EAGAIN The kernel was unable to allocate memory for the request, or
  otherwise ran out of resources to handle it. The application should wait for
  some completions and try again.

At a minimum, it deserves a comment like "We accept PANIC on memory exhaustion
here."

> +			pgstat_report_wait_end();
> +
> +			if (ret == -EINTR)
> +			{
> +				continue;
> +			}
> +			else if (ret != 0)
> +			{
> +				elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));

I think errno isn't meaningful here, so %m doesn't belong.

> --- a/doc/src/sgml/config.sgml
> +++ b/doc/src/sgml/config.sgml
> @@ -2687,6 +2687,12 @@ include_dir 'conf.d'
>              <literal>worker</literal> (execute asynchronous I/O using worker processes)
>             </para>
>            </listitem>
> +          <listitem>
> +           <para>
> +            <literal>io_uring</literal> (execute asynchronous I/O using
> +            io_uring, if available)
> +           </para>
> +          </listitem>

Docs should eventually cover RLIMIT_MEMLOCK per
https://github.com/axboe/liburing "ulimit settings".  Maybe RLIMIT_NOFILE,
too.

> @@ -2498,6 +2529,12 @@ FilePathName(File file)
>  int
>  FileGetRawDesc(File file)
>  {
> +	int			returnCode;
> +
> +	returnCode = FileAccess(file);
> +	if (returnCode < 0)
> +		return returnCode;
> +
>  	Assert(FileIsValid(file));
>  	return VfdCache[file].fd;
>  }

What's the rationale for this function's change?

> +The main reason to want to use Direct IO are:

> +The main reason *not* to use Direct IO are:

x2 s/main reason/main reasons/

> +  and direct IO without O_DSYNC needs to issue a write and after the writes
> +  completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a

s/writes/write's/

> +  single FUA write).

I recommend including the acronym expansion: s/FUA/Force Unit Access (FUA)/

> +In an `EXEC_BACKEND` build backends executable code and other process local

s/backends/backends'/

> +state is not necessarily mapped to the same addresses in each process due to
> +ASLR. This means that the shared memory cannot contain pointer to callbacks.

s/pointer/pointers/

> +The "solution" to this the ability to associate multiple completion callbacks
> +with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
> +state and to verify the page and md.c. another callback to check if the IO
> +operation was successful.

One of these or similar:
s/md.c. another/md.c can have another/
s/md.c. /md.c /


I've got one high-level question that I felt could take too long to answer for
myself by code reading.  What's the cleanup story if process A does
elog(FATAL) with unfinished I/O?  Specifically:

- Suppose some other process B reuses the shared memory AIO data structures
  that pertained to process A.  After that, some process C completes the I/O
  in shmem.  Do we avoid confusing B by storing local callback data meant for
  A in shared memory now pertaining to B?

- Thinking more about this README paragraph:

    +In addition to completion, AIO callbacks also are called to "prepare" an
    +IO. This is, e.g., used to increase buffer reference counts to account for the
    +AIO subsystem referencing the buffer, which is required to handle the case
    +where the issuing backend errors out and releases its own pins while the IO is
    +still ongoing.

  Which function performs that reference count increase?  I'm not finding it
  today.  I wanted to look at how it ensures the issuing backend still exists
  as the function increases the reference count.


One later-patch item:

> +static PgAioResult
> +SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
> +{
...
> +	TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
> +									  tag.blockNum,
> +									  tag.spcOid,
> +									  tag.dbOid,
> +									  tag.relNumber,
> +									  INVALID_PROC_NUMBER,
> +									  false);

I wondered about whether the buffer-read-done probe should happen in the
process that calls the complete_shared callback or in the process that did the
buffer-read-start probe.  When I see dtrace examples, they usually involve
explicitly naming each PID to trace.  Assuming that's indeed the norm, I think
the local callback would be the better place, so a given trace contains both
probes.  If it were reasonable to dtrace all current and future postmaster
kids, that would argue for putting the probe in the complete_shared callback.
Alternatively, would could argue for separate probes buffer-read-done-shmem
and buffer-read-done.

Thanks,
nm

  76. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-11T23:55:35Z

    Hi,

On 2025-03-11 12:41:08 -0700, Noah Misch wrote:
> On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> > On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > > For non-sync IO methods, I gather it's essential that a process other than the
> > > IO definer be scanning for incomplete IOs and completing them.
>
> > > Otherwise, deadlocks like this would happen:
> >
> > > backend1 locks blk1 for non-IO reasons
> > > backend2 locks blk2, starts AIO write
> > > backend1 waits for lock on blk2 for non-IO reasons
> > > backend2 waits for lock on blk1 for non-IO reasons
> > >
> > > If that's right, in worker mode, the IO worker resolves that deadlock.  What
> > > resolves it under io_uring?  Another process that happens to do
> > > pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> > > make that happen systematically.
> >
> > Yea, it's code that I haven't forward ported yet. I think basically
> > LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
> > immediately acquire the lock and if the buffer has IO going on.
>
> I'm not finding that code in v2.6.  What function has it?

My local version now has it... Sorry, I was focusing on the earlier patches
until now.

What do we want to do for ConditionalLockBufferForCleanup() (I don't think
IsBufferCleanupOK() can matter)?  I suspect we should also make it wait for
the IO. See below:

Not for 18, but for full write support, we'll also need logic to wait for IO
in LockBuffer(BUFFER_LOCK_EXCLUSIVE) and answer the same question as for
ConditionalLockBufferForCleanup() for ConditionalLockBuffer().

It's not an issue with the current level of write support in the stack of
patches. But with v1 AIO, which had support for a lot more ways of doing
asynchronous writes, it turned out that not handling it in
ConditionalLockBuffer() triggers an endless loop. This can be
kind-of-reproduced today by just making ConditionalLockBuffer() always return
false - triggers a hang in the regression tests:

spginsert() loops around spgdoinsert() until it succeeds. spgdoinsert() locks
the child page with ConditionalLockBuffer() and gives up if it can't.

That seems like rather bad code in spgist, because, even without AIO, it'll
busy-loop until the buffer is unlocked. Which could take a while, given that
it'll conflict even with a share locker and thus synchronous writes.


Even if we fixed spgist, it seems rather likely that there's other code that
wouldn't tolerate "spurious" failures. Which leads me to think that causing
the IO to complete is probably the safest bet. Triggering IO completion never
requires acquiring new locks that could participate in a deadlock, so it'd be
safe.



> > At this point I am not aware of anything significant left to do in the main
> > AIO commit, safe some of the questions below.
>
> That is a big milestone.

Indeed!


> > - We could reduce memory usage a tiny bit if we made the mapping between
> >   pgproc and per-backend-aio-state more complicated, i.e. not just indexed by
> >   ProcNumber. Right now IO workers have the per-backend AIO state, but don't
> >   actually need it.  I'm mildly inclined to think that the complexity isn't
> >   worth it, but on the fence.
>
> The max memory savings, for 32 IO workers, is like the difference between
> max_connections=500 and max_connections=532, right?

Even less than that: Aux processes aren't always used as a multiplier in
places where max_connections etc are. E.g. max_locks_per_transaction is just
multiplied by MaxBackends, not MaxBackends+NUM_AUXILIARY_PROCS.


> If that's right, I wouldn't bother in the foreseeable future.

Cool.



> > - Three of the commits in the series really are just precursor commits to
> >   their subsequent commits, which I found helpful for development and review,
> >   namely:
> >
> >   - aio: Basic subsystem initialization
> >   - aio: Skeleton IO worker infrastructure
> >   - aio: Add liburing dependency
> >
> >   Not sure if it's worth keeping these separate or whether they should just be
> >   merged with their "real commit".
>
> The split aided my review.  It's trivial to turn an unmerged stack of commits
> into the merged equivalent, but unmerging is hard.

That's been the feedback so far, so I'll leave it split.



> > - Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
> >   but all the ereport()s add a noticeable amount of overhead at high IO
> >   throughput (at multiple gigabytes/second), so that's probably not right
> >   forever.  I'd leave this on initially and then change it to default to off
> >   later.  I think that's ok?
>
> Sure.  Perhaps make it depend on USE_ASSERT_CHECKING later?

Yea, that makes sense.


> > - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> >   automatically in/decrease active workers, the max number of workers (32) is
> >   always allocated. That means we use more semaphores than before. I think
> >   that's ok, it's not 1995 anymore.  Alternatively we can add a
> >   "io_workers_max" GUC and probe for it in initdb.
>
> Let's start as you have it.  If someone wants to make things perfect for
> non-root BSD users, they can add the GUC later.  io_method=sync is a
> sufficient backup plan indefinitely.

Cool.

I think we'll really need to do something about this for BSD users regardless
of AIO. Or maybe those OSs should fix something, but somehow I am not having
high hopes for an OS that claims to have POSIX confirming unnamed semaphores
due to having a syscall that always returns EPERM... [1].


> > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> >   sure that's great?
> >
> >   They could be an enum array or such too? That'd perhaps be a bit more
> >   extensible? OTOH, we don't currently use enums in the catalogs and arrays
> >   are somewhat annoying to conjure up from C.
>
> An enum array does seem elegant and extensible, but it has the problems you
> say.  (I would expect to lose time setting up pg_enum.oid values to not change
> between releases.)  A possible compromise would be a text array like
> heap_tuple_infomask_flags() does.  Overall, I'm not seeing a clear need to
> change away from the bool columns.

Yea, I think that's where I ended up too. If we get a dozen flags we can
reconsider.



> > Todo:
>
> > - Figure out how to deduplicate support for LockBufferForCleanup() in
> >   TerminateBufferIO().
>
> Yes, I agree there's an opportunity for a WakePinCountWaiter() or similar
> subroutine.

Done.


> > - Check if documentation for track_io_timing needs to be adjusted, after the
> >   bufmgr.c changes we only track waiting for an IO.
>
> Yes.

The relevant sentences seem to be:

- "Enables timing of database I/O calls."

  s/calls/waits/

- "Time spent in {read,write,writeback,extend,fsync} operations"

  s/in/waiting for/

  Even though not all of these will use AIO, the "waiting for" formulation
  seems just as accurate.

- "Columns tracking I/O time will only be non-zero when <xref
  linkend="guc-track-io-timing"/> is enabled."

  s/time/wait time/


> On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> > Attached is v2.6 of the AIO patchset.
>
> > - 0005, 0006 - io_uring support - close, but we need to do something about
> >   set_max_fds(), which errors out spuriously in some cases
>
> What do we know about those cases?  I don't see a set_max_fds(); is that
> set_max_safe_fds(), or something else?

Sorry, yes, set_max_safe_fds(). The problem basically is that with io_uring we
will have a large number of FDs already allocated by the time
set_max_safe_fds() is called. set_max_safe_fds() subtracts already_open from
max_files_per_process allowing few, and even negative, IOs.

I think we should redefine max_files_per_process to be about the number of
files each *backend* will additionally open.  Jelte was working on related
patches, see [2]

> > +	 * AIO handles need be registered in critical sections and therefore
> > +	 * cannot use the normal ResoureElem mechanism.
>
> s/ResoureElem/ResourceElem/

Oops, fixed.


> > +      <varlistentry id="guc-io-method" xreflabel="io_method">
> > +       <term><varname>io_method</varname> (<type>enum</type>)
> > +       <indexterm>
> > +        <primary><varname>io_method</varname> configuration parameter</primary>
> > +       </indexterm>
> > +       </term>
> > +       <listitem>
> > +        <para>
> > +         Selects the method for executing asynchronous I/O.
> > +         Possible values are:
> > +         <itemizedlist>
> > +          <listitem>
> > +           <para>
> > +            <literal>sync</literal> (execute asynchronous I/O synchronously)
>
> The part in parentheses reads like a contradiction to me.

There's something to that...


> How about phrasing it like one of these:
>
>   (execute I/O synchronously, even I/O eligible for asynchronous execution)
>   (execute asynchronous-eligible I/O synchronously)
>   (execute I/O synchronously, even when asynchronous execution was feasible)

I like the second one best, adopted.


> [..]
> End sentence with question mark, probably.
> [..]
> s/strict/strictly/
> [..]
> I recommend adding "Always called in a critical section." since at least
> pgaio_worker_submit() subtly needs it.
> [..]
> s/that that/that/
> [..]
> s/smgr.,/smgr.c,/ or just "smgr"
> [..]
> s/locallbacks/local callbacks/
> [..]
> s/the the/the/

All adopted.


> > +PgAioHandle *
> > +pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
> > +{
> > +	if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
> > +	{
> > +		Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
> > +		pgaio_submit_staged();
>
> I'm seeing the "num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE" case uncovered in a
> check-world coverage report.  I tried PGAIO_SUBMIT_BATCH_SIZE=2,
> io_max_concurrency=1, and io_max_concurrency=64.  Do you already have a recipe
> for reaching this case?

With the default server settings it's hard to hit due to read_stream.c
limiting how much IO it issues:

1) The default io_combine_limit=16 makes reads larger, reducing the queue
   depth, at least for sequential scans

2) The default shared_buffers/max_connections settings limit the number of
   buffers that can be pinned to 86, which will only allow a small number of
   IOs due to 86/io_combine_limit = ~5

3) The default effective_io_concurrency only allows one IO in flight

Melanie has a patch to adjust effective_io_concurrency:
https://www.postgresql.org/message-id/CAAKRu_Z4ekRbfTacYYVrvu9xRqS6G4DMbZSbN_1usaVtj%2Bbv2w%40mail.gmail.com

If I increase shared_buffers and decrease io_combine_limit and put an
elog(PANIC) in that branch, it's rather quickly hit.



> > +/*
> > + * Stage IO for execution and, if necessary, submit it immediately.
> > + *
> > + * Should only be called from pgaio_io_prep_*().
> > + */
> > +void
> > +pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
> > +{
>
> We've got closely-associated verbs "prepare", "prep", and "stage".  README.md
> doesn't mention "stage".  Can one of the following two changes happen?
>
> - README.md starts mentioning "stage" and how it differs from the others
> - Code stops using "stage"

I'll try to add something to README.md. To me the sequence is prepare->stage.


> > + * Batch submission mode needs to explicitly ended with
> > + * pgaio_exit_batchmode(), but it is allowed to throw errors, in which case
> > + * error recovery will end the batch.
>
> This sentence needs some grammar help, I think.

Indeed.

> Maybe use:
>
>  * End batch submission mode with pgaio_exit_batchmode().  (Throwing errors is
>  * allowed; error recovery will end the batch.)

I like it.


> >  Size
> >  AioShmemSize(void)
> >  {
> >  	Size		sz = 0;
> >
> > +	/*
> > +	 * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
> > +	 * However, if the DBA explicitly set wal_buffers = -1 in the config file,
>
> s/wal_buffers/io_max_concurrency/

Ooops.


> > +extern int	io_workers;
>
> By the rule that GUC vars are PGDLLIMPORT, this should be PGDLLIMPORT.

Indeed. I wish we had something finding violations of this automatically...


> > +static void
> > +maybe_adjust_io_workers(void)
>
> This also restarts workers that exit, so perhaps name it
> start_io_workers_if_missing().

But it also stops IO workers if necessary?


> > +{
> ...
> > +		/* Try to launch one. */
> > +		child = StartChildProcess(B_IO_WORKER);
> > +		if (child != NULL)
> > +		{
> > +			io_worker_children[id] = child;
> > +			++io_worker_count;
> > +		}
> > +		else
> > +			break;				/* XXX try again soon? */
>
> Can LaunchMissingBackgroundProcesses() become the sole caller of this
> function, replacing the current mix of callers?  That would be more conducive
> to promptly doing the right thing after launch failure.

I'm not sure that'd be a good idea - right now IO workers are started before
the startup process, as the startup process might need to perform IO. If we
started it only later in ServerLoop() we'd potentially do a fair bit of work,
including starting checkpointer, bgwriter, bgworkers before we started IO
workers.  That shouldn't actively break anything, but it would likely make
things slower.

I rather dislike the code around when we start what. Leaving AIO aside, during
a normal startup we start checkpointer, bgwriter before the startup
process. But during a crash restart we don't explicitly start them. Why make
things uniform when it coul also be exciting :)


> > --- a/src/backend/utils/init/miscinit.c
> > +++ b/src/backend/utils/init/miscinit.c
> > @@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
> >  		case B_CHECKPOINTER:
> >  			backendDesc = gettext_noop("checkpointer");
> >  			break;
> > +		case B_IO_WORKER:
> > +			backendDesc = "io worker";
>
> Wrap in gettext_noop() like B_CHECKPOINTER does.
>
> > +         Only has an effect if <xref linkend="guc-max-wal-senders"/> is set to
> > +         <literal>worker</literal>.
>
> s/guc-max-wal-senders/guc-io-method/
>
> > + * of IOs, wakeups "fan out"; each woken IO worker can wake two more.  qXXX
>
> s/qXXX/XXX/

All fixed.


> > +			/*
> > +			 * It's very unlikely, but possible, that reopen fails. E.g. due
> > +			 * to memory allocations failing or file permissions changing or
> > +			 * such.  In that case we need to fail the IO.
> > +			 *
> > +			 * There's not really a good errno we can report here.
> > +			 */
> > +			error_errno = ENOENT;
>
> Agreed there's not a good errno, but let's use a fake errno that we're mighty
> unlikely to confuse with an actual case of libc returning that errno.  Like
> one of EBADF or EOWNERDEAD.

Can we rely on that to be present on all platforms, including windows?


> > +	for (int contextno = 0; contextno < TotalProcs; contextno++)
> > +	{
> > +		PgAioUringContext *context = &pgaio_uring_contexts[contextno];
> > +		int			ret;
> > +
> > +		/*
> > +		 * XXX: Probably worth sharing the WQ between the different rings,
> > +		 * when supported by the kernel. Could also cause additional
> > +		 * contention, I guess?
> > +		 */
> > +#if 0
> > +		if (!AcquireExternalFD())
> > +			elog(ERROR, "No external FD available");
> > +#endif
>
> Probably remove the "#if 0" or add a comment on why it's here.

Will do. It was an attempt at dealing with the set_max_safe_fds() issue above,
but it turned out to not work at all, given how fd.c currently works.


> > +		ret = io_uring_submit(uring_instance);
> > +		pgstat_report_wait_end();
> > +
> > +		if (ret == -EINTR)
> > +		{
> > +			pgaio_debug(DEBUG3,
> > +						"aio method uring: submit EINTR, nios: %d",
> > +						num_staged_ios);
> > +		}
> > +		else if (ret < 0)
> > +			elog(PANIC, "failed: %d/%s",
> > +				 ret, strerror(-ret));
>
> I still think (see 2024-09-16 review) EAGAIN should do the documented
> recommendation instead of PANIC:
>
>   EAGAIN The kernel was unable to allocate memory for the request, or
>   otherwise ran out of resources to handle it. The application should wait for
>   some completions and try again.

I don't think this can be hit in a recoverable way. We'd likely just end up
with an untested path that quite possibly would be wrong.

What wait time would be appropriate?  What problems would it cause if we just
slept while holding critical lwlocks? I think it'd typically just delay the
crash-restart if we did, making it harder to recover from the problem.

Because we are careful to limit how many outstanding IO requests there are on
an io_uring instance, the kernel has to have run *severely* out of memory to
hit this.

I suspect it might currently be *impossible* to hit this due to ENOMEM,
because io_uring will fall back to allocating individual request, if the batch
allocation it normally does, fails. My understanding is that for small
allocations the kernel will try to reclaim memory forever, only large ones can
fail.

Even if it were possible to hit, the likelihood that postgres can continue to
work ok if the kernel can't allocate ~250 bytes seems very low.

How about adding a dedicated error message for EAGAIN? IMO io_uring_enter()'s
meaning of EAGAIN is, uhm, unconvential, so a better error message than
strerror() might be good?


Proposed comment:
			/*
			 * The io_uring_enter() manpage suggests that the appropriate
			 * reaction to EAGAIN is:
			 *
			 * "The application should wait for some completions and try
			 * again"
			 *
			 * However, it seems unlikely that that would help in our case, as
			 * we apply a low limit to the number of outstanding IOs and thus
			 * also outstanding completions, making it unlikely that we'd get
			 * EAGAIN while the OS is in good working order.
			 *
			 * Additionally, it would be problematic to just wait here, our
			 * caller might hold critical locks. It'd possibly lead to
			 * delaying the crash-restart that seems likely to occur when the
			 * kernel is under such heavy memory pressure.
			 */




> > +			pgstat_report_wait_end();
> > +
> > +			if (ret == -EINTR)
> > +			{
> > +				continue;
> > +			}
> > +			else if (ret != 0)
> > +			{
> > +				elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
>
> I think errno isn't meaningful here, so %m doesn't belong.

You're right.  I wonder if we should make errno meaningful though (by setting
it), the elog.c machinery captures it and I know that there are logging hooks
that utilize that fact.  That'd also avoid the need to use strerror() here.


> > --- a/doc/src/sgml/config.sgml
> > +++ b/doc/src/sgml/config.sgml
> > @@ -2687,6 +2687,12 @@ include_dir 'conf.d'
> >              <literal>worker</literal> (execute asynchronous I/O using worker processes)
> >             </para>
> >            </listitem>
> > +          <listitem>
> > +           <para>
> > +            <literal>io_uring</literal> (execute asynchronous I/O using
> > +            io_uring, if available)
> > +           </para>
> > +          </listitem>
>
> Docs should eventually cover RLIMIT_MEMLOCK per
> https://github.com/axboe/liburing "ulimit settings".

The way we currently use io_uring (i.e. no registered buffers), the
RLIMIT_MEMLOCK advice only applies to linux <= 5.11.  I'm not sure that's
worth documenting?


> Maybe RLIMIT_NOFILE, too.

Yea, we probably need to. Depends a bit on where we go with [2] though.


>
> > @@ -2498,6 +2529,12 @@ FilePathName(File file)
> >  int
> >  FileGetRawDesc(File file)
> >  {
> > +	int			returnCode;
> > +
> > +	returnCode = FileAccess(file);
> > +	if (returnCode < 0)
> > +		return returnCode;
> > +
> >  	Assert(FileIsValid(file));
> >  	return VfdCache[file].fd;
> >  }
>
> What's the rationale for this function's change?

It flatly didn't work before.  I guess I can make that a separate commit.


> > +The main reason to want to use Direct IO are:
>
> > +The main reason *not* to use Direct IO are:
>
> x2 s/main reason/main reasons/
>
> > +  and direct IO without O_DSYNC needs to issue a write and after the writes
> > +  completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
>
> s/writes/write's/
>
> > +  single FUA write).
>
> I recommend including the acronym expansion: s/FUA/Force Unit Access (FUA)/
>
> > +In an `EXEC_BACKEND` build backends executable code and other process local
>
> s/backends/backends'/
>
> > +state is not necessarily mapped to the same addresses in each process due to
> > +ASLR. This means that the shared memory cannot contain pointer to callbacks.
>
> s/pointer/pointers/
>
> > +The "solution" to this the ability to associate multiple completion callbacks
> > +with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
> > +state and to verify the page and md.c. another callback to check if the IO
> > +operation was successful.
>
> One of these or similar:
> s/md.c. another/md.c can have another/
> s/md.c. /md.c /

All applied.


> I've got one high-level question that I felt could take too long to answer for
> myself by code reading.  What's the cleanup story if process A does
> elog(FATAL) with unfinished I/O?  Specifically:

It's a good question. Luckily there's a relatively easy answer:

pgaio_shutdown() is registered via before_shmem_exit() in pgaio_init_backend()
and pgaio_shutdown() waits for all IOs to finish.

The main reason this exists is that the AIO mechanism in various OSs, at least
in some OS versions, don't like it if the issuing process exits while the IO
is in flight.  IIRC that was the case with in v1 with posix_aio (which we
don't support in v2, and probably should never use) and I think also with
io_uring in some kernel versions.

Another reason is that those requests would show up in pg_aios (or whatever we
end up naming it) until they're reused, which doesn't seem great.


> - Suppose some other process B reuses the shared memory AIO data structures
>   that pertained to process A.  After that, some process C completes the I/O
>   in shmem.  Do we avoid confusing B by storing local callback data meant for
>   A in shared memory now pertaining to B?

This will, before pgaio_shutdown() gets involved, also be prevented by local
callbacks being cleared by resowner cleanup. We take care that that resowner
cleanup happens before process exit. That's important, because the backend
local pointer could be invalidated by an ERROR


> - Thinking more about this README paragraph:
>
>     +In addition to completion, AIO callbacks also are called to "prepare" an
>     +IO. This is, e.g., used to increase buffer reference counts to account for the
>     +AIO subsystem referencing the buffer, which is required to handle the case
>     +where the issuing backend errors out and releases its own pins while the IO is
>     +still ongoing.
>
>   Which function performs that reference count increase?  I'm not finding it
>   today.

Ugh, I just renamed the relevant functions in my local branch, while trying to
reduce the code duplication between shared and local buffers ;).

In <= v2.6 it's shared_buffer_stage_common() and local_buffer_readv_stage().

In v2.7-to-be it is buffer_stage_common(), which now supports both shared and
local buffers.


> I wanted to look at how it ensures the issuing backend still exists as the
> function increases the reference count.

The reference count is increased solely in the BufferDesc, *not* in the
backend-local pin tracking.  Earlier I had tracked the pin in BufferDesc for
shared buffers (as the pin needs to be released upon completion, which might
be in another backend), but in LocalRefCount[] for temp buffers.  But that
turned out to not work when the backend errors out, as it would make
CheckForLocalBufferLeaks() complain.


>
> One later-patch item:
>
> > +static PgAioResult
> > +SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
> > +{
> ...
> > +	TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
> > +									  tag.blockNum,
> > +									  tag.spcOid,
> > +									  tag.dbOid,
> > +									  tag.relNumber,
> > +									  INVALID_PROC_NUMBER,
> > +									  false);
>
> I wondered about whether the buffer-read-done probe should happen in the
> process that calls the complete_shared callback or in the process that did the
> buffer-read-start probe.

Yea, that's a good point. I should at least have added a comment pointing out
that it's a choice with pros and cons.

The reason I went for doing it in the completion callback is that it seemed
better to get the READ_DONE event as soon as possible, even if the issuer of
the IO is currently busy doing other things. The shared completion callback is
after all where the buffer state is updated for shared buffers.

But I think you have a point too.


> When I see dtrace examples, they usually involve explicitly naming each PID
> to trace

TBH, i've only ever used our tracepoints via perf and bpftrace, not dtrace
itself. For those it's easy to trace more than just a single pid and to
monitor system-wide. I don't really know enough about using dtrace itself.


> Assuming that's indeed the norm, I think the local callback would
> be the better place, so a given trace contains both probes.

Seems like a shame to add an extra indirect function call for a tracing
feature that afaict approximately nobody ever uses (IIRC we several times have
passed wrong things to tracepoints without that being noticed).


TBH, the tracepoints are so poorly documented and maintained that I was
tempted to suggest removing them a couple times.


This was an awesome review, thanks!

Andres Freund

[1] https://man.openbsd.org/sem_init.3#STANDARDS
[2] https://postgr.es/m/D80MHNSG4EET.6MSV5G9P130F%40jeltef.nl

  77. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-12T03:57:43Z

    On Tue, Mar 11, 2025 at 07:55:35PM -0400, Andres Freund wrote:
> On 2025-03-11 12:41:08 -0700, Noah Misch wrote:
> > On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> > > On 2024-09-16 07:43:49 -0700, Noah Misch wrote:

> What do we want to do for ConditionalLockBufferForCleanup() (I don't think
> IsBufferCleanupOK() can matter)?  I suspect we should also make it wait for
> the IO. See below:

I agree IsBufferCleanupOK() can't matter.  It asserts that the caller already
holds the exclusive buffer lock, and it doesn't loop or wait.

> [...] leads me to think that causing
> the IO to complete is probably the safest bet. Triggering IO completion never
> requires acquiring new locks that could participate in a deadlock, so it'd be
> safe.

Yes, that decision looks right to me.  I scanned the callers, and none of them
clearly prefers a different choice.  If we someday find one caller prefers a
false return over blocking on I/O completion, we can always introduce a new
ConditionalLock* variant for that.

> > > - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> > >   automatically in/decrease active workers, the max number of workers (32) is
> > >   always allocated. That means we use more semaphores than before. I think
> > >   that's ok, it's not 1995 anymore.  Alternatively we can add a
> > >   "io_workers_max" GUC and probe for it in initdb.
> >
> > Let's start as you have it.  If someone wants to make things perfect for
> > non-root BSD users, they can add the GUC later.  io_method=sync is a
> > sufficient backup plan indefinitely.
> 
> Cool.
> 
> I think we'll really need to do something about this for BSD users regardless
> of AIO. Or maybe those OSs should fix something, but somehow I am not having
> high hopes for an OS that claims to have POSIX confirming unnamed semaphores
> due to having a syscall that always returns EPERM... [1].

I won't mind a project making things better for non-root BSD users.  I do
think such a project should not block other projects making things better for
everything else (like $SUBJECT).

> > > - Check if documentation for track_io_timing needs to be adjusted, after the
> > >   bufmgr.c changes we only track waiting for an IO.
> >
> > Yes.
> 
> The relevant sentences seem to be:
> 
> - "Enables timing of database I/O calls."
> 
>   s/calls/waits/
> 
> - "Time spent in {read,write,writeback,extend,fsync} operations"
> 
>   s/in/waiting for/
> 
>   Even though not all of these will use AIO, the "waiting for" formulation
>   seems just as accurate.
> 
> - "Columns tracking I/O time will only be non-zero when <xref
>   linkend="guc-track-io-timing"/> is enabled."
> 
>   s/time/wait time/

Sounds good.

> > On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> > > Attached is v2.6 of the AIO patchset.
> >
> > > - 0005, 0006 - io_uring support - close, but we need to do something about
> > >   set_max_fds(), which errors out spuriously in some cases
> >
> > What do we know about those cases?  I don't see a set_max_fds(); is that
> > set_max_safe_fds(), or something else?
> 
> Sorry, yes, set_max_safe_fds(). The problem basically is that with io_uring we
> will have a large number of FDs already allocated by the time
> set_max_safe_fds() is called. set_max_safe_fds() subtracts already_open from
> max_files_per_process allowing few, and even negative, IOs.
> 
> I think we should redefine max_files_per_process to be about the number of
> files each *backend* will additionally open.  Jelte was working on related
> patches, see [2]

Got it.  max_files_per_process is a quaint setting, documented as follows (I
needed the reminder):

        If the kernel is enforcing
        a safe per-process limit, you don't need to worry about this setting.
        But on some platforms (notably, most BSD systems), the kernel will
        allow individual processes to open many more files than the system
        can actually support if many processes all try to open
        that many files. If you find yourself seeing <quote>Too many open
        files</quote> failures, try reducing this setting.

I could live with
v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but would lean
against it since it feels unduly novel to have a setting where we use the
postgresql.conf value to calculate a value that becomes the new SHOW-value of
the same setting.  Options I'd consider before that:

- Like you say, "redefine max_files_per_process to be about the number of
  files each *backend* will additionally open".  It will become normal that
  each backend's actual FD list length is max_files_per_process + MaxBackends
  if io_method=io_uring.  Outcome is not unlike
  v6-0002-Bump-postmaster-soft-open-file-limit-RLIMIT_NOFIL.patch +
  v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but we don't
  mutate max_files_per_process.  Benchmark results should not change beyond
  the inter-major-version noise level unless one sets io_method=io_uring.  I'm
  feeling best about this one, but I've not been thinking about it long.

- When building with io_uring, make the max_files_per_process default
  something like 10000 instead.  Disadvantages: FD usage grows even if you
  don't use io_uring.  Merely rebuilding with io_uring (not enabling it at
  runtime) will change benchmark results.  High MaxBackends still needs to
  override the value.

> > > +static void
> > > +maybe_adjust_io_workers(void)
> >
> > This also restarts workers that exit, so perhaps name it
> > start_io_workers_if_missing().
> 
> But it also stops IO workers if necessary?

Good point.  Maybe just add a comment like "start or stop IO workers to close
the gap between the running count and the configured count intent".

> > > +{
> > ...
> > > +		/* Try to launch one. */
> > > +		child = StartChildProcess(B_IO_WORKER);
> > > +		if (child != NULL)
> > > +		{
> > > +			io_worker_children[id] = child;
> > > +			++io_worker_count;
> > > +		}
> > > +		else
> > > +			break;				/* XXX try again soon? */
> >
> > Can LaunchMissingBackgroundProcesses() become the sole caller of this
> > function, replacing the current mix of callers?  That would be more conducive
> > to promptly doing the right thing after launch failure.
> 
> I'm not sure that'd be a good idea - right now IO workers are started before
> the startup process, as the startup process might need to perform IO. If we
> started it only later in ServerLoop() we'd potentially do a fair bit of work,
> including starting checkpointer, bgwriter, bgworkers before we started IO
> workers.  That shouldn't actively break anything, but it would likely make
> things slower.

I missed that.  How about keeping the two calls associated with PM_STARTUP but
replacing the assign_io_workers() and process_pm_child_exit() calls with one
in LaunchMissingBackgroundProcesses()?  In the event of a launch failure, I
think that would retry the launch quickly, as opposed to maybe-never.

> I rather dislike the code around when we start what. Leaving AIO aside, during
> a normal startup we start checkpointer, bgwriter before the startup
> process. But during a crash restart we don't explicitly start them. Why make
> things uniform when it coul also be exciting :)

It's become some artisanal code! :)

> > > +			/*
> > > +			 * It's very unlikely, but possible, that reopen fails. E.g. due
> > > +			 * to memory allocations failing or file permissions changing or
> > > +			 * such.  In that case we need to fail the IO.
> > > +			 *
> > > +			 * There's not really a good errno we can report here.
> > > +			 */
> > > +			error_errno = ENOENT;
> >
> > Agreed there's not a good errno, but let's use a fake errno that we're mighty
> > unlikely to confuse with an actual case of libc returning that errno.  Like
> > one of EBADF or EOWNERDEAD.
> 
> Can we rely on that to be present on all platforms, including windows?

I expect EBADF is universal.  EBADF would be fine.

EOWNERDEAD is from 2006.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/errno-constants?view=msvc-140
says VS2015 had EOWNERDEAD (the page doesn't have links for older Visual
Studio versions, so I consider them unknown).
https://github.com/coreutils/gnulib/blob/master/doc/posix-headers/errno.texi
lists some OSs not having it, the newest of which looks like NetBSD 9.3
(2022).  We could use it and add a #define for platforms lacking it.

> > > +		ret = io_uring_submit(uring_instance);
> > > +		pgstat_report_wait_end();
> > > +
> > > +		if (ret == -EINTR)
> > > +		{
> > > +			pgaio_debug(DEBUG3,
> > > +						"aio method uring: submit EINTR, nios: %d",
> > > +						num_staged_ios);
> > > +		}
> > > +		else if (ret < 0)
> > > +			elog(PANIC, "failed: %d/%s",
> > > +				 ret, strerror(-ret));
> >
> > I still think (see 2024-09-16 review) EAGAIN should do the documented
> > recommendation instead of PANIC:
> >
> >   EAGAIN The kernel was unable to allocate memory for the request, or
> >   otherwise ran out of resources to handle it. The application should wait for
> >   some completions and try again.
> 
> I don't think this can be hit in a recoverable way. We'd likely just end up
> with an untested path that quite possibly would be wrong.
> 
> What wait time would be appropriate?  What problems would it cause if we just
> slept while holding critical lwlocks? I think it'd typically just delay the
> crash-restart if we did, making it harder to recover from the problem.

I might use 30s like pgwin32_open_handle(), but 30s wouldn't be principled.

> Because we are careful to limit how many outstanding IO requests there are on
> an io_uring instance, the kernel has to have run *severely* out of memory to
> hit this.
> 
> I suspect it might currently be *impossible* to hit this due to ENOMEM,
> because io_uring will fall back to allocating individual request, if the batch
> allocation it normally does, fails. My understanding is that for small
> allocations the kernel will try to reclaim memory forever, only large ones can
> fail.
> 
> Even if it were possible to hit, the likelihood that postgres can continue to
> work ok if the kernel can't allocate ~250 bytes seems very low.
> 
> How about adding a dedicated error message for EAGAIN? IMO io_uring_enter()'s
> meaning of EAGAIN is, uhm, unconvential, so a better error message than
> strerror() might be good?

I'm fine with the present error message.

> Proposed comment:
> 			/*
> 			 * The io_uring_enter() manpage suggests that the appropriate
> 			 * reaction to EAGAIN is:
> 			 *
> 			 * "The application should wait for some completions and try
> 			 * again"
> 			 *
> 			 * However, it seems unlikely that that would help in our case, as
> 			 * we apply a low limit to the number of outstanding IOs and thus
> 			 * also outstanding completions, making it unlikely that we'd get
> 			 * EAGAIN while the OS is in good working order.
> 			 *
> 			 * Additionally, it would be problematic to just wait here, our
> 			 * caller might hold critical locks. It'd possibly lead to
> 			 * delaying the crash-restart that seems likely to occur when the
> 			 * kernel is under such heavy memory pressure.
> 			 */

That works for me.  No retry needed, then.

> > > +			pgstat_report_wait_end();
> > > +
> > > +			if (ret == -EINTR)
> > > +			{
> > > +				continue;
> > > +			}
> > > +			else if (ret != 0)
> > > +			{
> > > +				elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
> >
> > I think errno isn't meaningful here, so %m doesn't belong.
> 
> You're right.  I wonder if we should make errno meaningful though (by setting
> it), the elog.c machinery captures it and I know that there are logging hooks
> that utilize that fact.  That'd also avoid the need to use strerror() here.

That's better still.

> > > --- a/doc/src/sgml/config.sgml
> > > +++ b/doc/src/sgml/config.sgml
> > > @@ -2687,6 +2687,12 @@ include_dir 'conf.d'
> > >              <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > >             </para>
> > >            </listitem>
> > > +          <listitem>
> > > +           <para>
> > > +            <literal>io_uring</literal> (execute asynchronous I/O using
> > > +            io_uring, if available)
> > > +           </para>
> > > +          </listitem>
> >
> > Docs should eventually cover RLIMIT_MEMLOCK per
> > https://github.com/axboe/liburing "ulimit settings".
> 
> The way we currently use io_uring (i.e. no registered buffers), the
> RLIMIT_MEMLOCK advice only applies to linux <= 5.11.  I'm not sure that's
> worth documenting?

Kernel 5.11 will be 5.5 years old by the time v18 is out.  Yeah, no need for
doc coverage of that.

> > One later-patch item:
> >
> > > +static PgAioResult
> > > +SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
> > > +{
> > ...
> > > +	TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
> > > +									  tag.blockNum,
> > > +									  tag.spcOid,
> > > +									  tag.dbOid,
> > > +									  tag.relNumber,
> > > +									  INVALID_PROC_NUMBER,
> > > +									  false);
> >
> > I wondered about whether the buffer-read-done probe should happen in the
> > process that calls the complete_shared callback or in the process that did the
> > buffer-read-start probe.
> 
> Yea, that's a good point. I should at least have added a comment pointing out
> that it's a choice with pros and cons.
> 
> The reason I went for doing it in the completion callback is that it seemed
> better to get the READ_DONE event as soon as possible, even if the issuer of
> the IO is currently busy doing other things. The shared completion callback is
> after all where the buffer state is updated for shared buffers.
> 
> But I think you have a point too.
> 
> 
> > When I see dtrace examples, they usually involve explicitly naming each PID
> > to trace
> 
> TBH, i've only ever used our tracepoints via perf and bpftrace, not dtrace
> itself. For those it's easy to trace more than just a single pid and to
> monitor system-wide. I don't really know enough about using dtrace itself.

Perhaps just a comment, then.

> > Assuming that's indeed the norm, I think the local callback would
> > be the better place, so a given trace contains both probes.
> 
> Seems like a shame to add an extra indirect function call

Yep.

> This was an awesome review, thanks!

Glad it helped.

> [1] https://man.openbsd.org/sem_init.3#STANDARDS
> [2] https://postgr.es/m/D80MHNSG4EET.6MSV5G9P130F%40jeltef.nl

  78. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-12T16:10:42Z

    Hi,

On 2025-03-11 19:55:35 -0400, Andres Freund wrote:
> On 2025-03-11 12:41:08 -0700, Noah Misch wrote:
> > On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> > > On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > > > For non-sync IO methods, I gather it's essential that a process other than the
> > > > IO definer be scanning for incomplete IOs and completing them.
> >
> > > > Otherwise, deadlocks like this would happen:
> > >
> > > > backend1 locks blk1 for non-IO reasons
> > > > backend2 locks blk2, starts AIO write
> > > > backend1 waits for lock on blk2 for non-IO reasons
> > > > backend2 waits for lock on blk1 for non-IO reasons
> > > >
> > > > If that's right, in worker mode, the IO worker resolves that deadlock.  What
> > > > resolves it under io_uring?  Another process that happens to do
> > > > pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> > > > make that happen systematically.
> > >
> > > Yea, it's code that I haven't forward ported yet. I think basically
> > > LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
> > > immediately acquire the lock and if the buffer has IO going on.
> >
> > I'm not finding that code in v2.6.  What function has it?
> 
> My local version now has it... Sorry, I was focusing on the earlier patches
> until now.

Looking more at my draft, I don't think it was race-free.  I had a race-free
way of doing it in the v1 patch (by making lwlocks extensible, so the check
for IO could happen between enqueueing on the lwlock wait queue and sleeping
on the semaphore), but that obviously requires that infrastructure.

I want to focus on reads for now, so I'll add FIXMEs to the relevant places in
the patch to support AIO writes and focus on the rest of the patch for now.

Greetings,

Andres Freund

  79. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-12T17:06:03Z

    Hi,

On 2025-03-11 20:57:43 -0700, Noah Misch wrote:
> > I think we'll really need to do something about this for BSD users regardless
> > of AIO. Or maybe those OSs should fix something, but somehow I am not having
> > high hopes for an OS that claims to have POSIX confirming unnamed semaphores
> > due to having a syscall that always returns EPERM... [1].
> 
> I won't mind a project making things better for non-root BSD users.  I do
> think such a project should not block other projects making things better for
> everything else (like $SUBJECT).

Oh, I strongly agree.  The main reason I would like it to be addressed that
I'm pretty tired of having to think about open/netbsd whenever we update some
default setting.


> > > On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> > > > Attached is v2.6 of the AIO patchset.
> > >
> > > > - 0005, 0006 - io_uring support - close, but we need to do something about
> > > >   set_max_fds(), which errors out spuriously in some cases
> > >
> > > What do we know about those cases?  I don't see a set_max_fds(); is that
> > > set_max_safe_fds(), or something else?
> > 
> > Sorry, yes, set_max_safe_fds(). The problem basically is that with io_uring we
> > will have a large number of FDs already allocated by the time
> > set_max_safe_fds() is called. set_max_safe_fds() subtracts already_open from
> > max_files_per_process allowing few, and even negative, IOs.
> > 
> > I think we should redefine max_files_per_process to be about the number of
> > files each *backend* will additionally open.  Jelte was working on related
> > patches, see [2]
> 
> Got it.  max_files_per_process is a quaint setting, documented as follows (I
> needed the reminder):
> 
>         If the kernel is enforcing
>         a safe per-process limit, you don't need to worry about this setting.
>         But on some platforms (notably, most BSD systems), the kernel will
>         allow individual processes to open many more files than the system
>         can actually support if many processes all try to open
>         that many files. If you find yourself seeing <quote>Too many open
>         files</quote> failures, try reducing this setting.
> 
> I could live with
> v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but would lean
> against it since it feels unduly novel to have a setting where we use the
> postgresql.conf value to calculate a value that becomes the new SHOW-value of
> the same setting.

I think we may update some other GUCs, but not sure.


> Options I'd consider before that:

> - Like you say, "redefine max_files_per_process to be about the number of
>   files each *backend* will additionally open".  It will become normal that
>   each backend's actual FD list length is max_files_per_process + MaxBackends
>   if io_method=io_uring.  Outcome is not unlike
>   v6-0002-Bump-postmaster-soft-open-file-limit-RLIMIT_NOFIL.patch +
>   v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but we don't
>   mutate max_files_per_process.  Benchmark results should not change beyond
>   the inter-major-version noise level unless one sets io_method=io_uring.  I'm
>   feeling best about this one, but I've not been thinking about it long.

Yea, I think that's something probably worth doing separately from Jelte's
patch.  I do think that it'd be rather helpful to have jelte's patch to
increase NOFILE in addition though.


> > > > +static void
> > > > +maybe_adjust_io_workers(void)
> > >
> > > This also restarts workers that exit, so perhaps name it
> > > start_io_workers_if_missing().
> > 
> > But it also stops IO workers if necessary?
> 
> Good point.  Maybe just add a comment like "start or stop IO workers to close
> the gap between the running count and the configured count intent".

It's now
/*
 * Start or stop IO workers, to close the gap between the number of running
 * workers and the number of configured workers.  Used to respond to change of
 * the io_workers GUC (by increasing and decreasing the number of workers), as
 * well as workers terminating in response to errors (by starting
 * "replacement" workers).
 */


> > > > +{
> > > ...
> > > > +		/* Try to launch one. */
> > > > +		child = StartChildProcess(B_IO_WORKER);
> > > > +		if (child != NULL)
> > > > +		{
> > > > +			io_worker_children[id] = child;
> > > > +			++io_worker_count;
> > > > +		}
> > > > +		else
> > > > +			break;				/* XXX try again soon? */
> > >
> > > Can LaunchMissingBackgroundProcesses() become the sole caller of this
> > > function, replacing the current mix of callers?  That would be more conducive
> > > to promptly doing the right thing after launch failure.
> > 
> > I'm not sure that'd be a good idea - right now IO workers are started before
> > the startup process, as the startup process might need to perform IO. If we
> > started it only later in ServerLoop() we'd potentially do a fair bit of work,
> > including starting checkpointer, bgwriter, bgworkers before we started IO
> > workers.  That shouldn't actively break anything, but it would likely make
> > things slower.
> 
> I missed that.  How about keeping the two calls associated with PM_STARTUP but
> replacing the assign_io_workers() and process_pm_child_exit() calls with one
> in LaunchMissingBackgroundProcesses()?

I think replacing the call in assign_io_workers() is a good idea, that way we
don't need assign_io_workers().

Less convinced it's a good idea to do the same for process_pm_child_exit() -
if IO workers errored out we'll launch backends etc before we get to
LaunchMissingBackgroundProcesses(). That's not a fundamental problem, but
seems a bit odd.

I think LaunchMissingBackgroundProcesses() should be split into one that
starts aux processes and one that starts bgworkers. The one maintaining aux
processes should be called before we start backends, the latter not.


> In the event of a launch failure, I think that would retry the launch
> quickly, as opposed to maybe-never.

That's a fair point.


> > > > +			/*
> > > > +			 * It's very unlikely, but possible, that reopen fails. E.g. due
> > > > +			 * to memory allocations failing or file permissions changing or
> > > > +			 * such.  In that case we need to fail the IO.
> > > > +			 *
> > > > +			 * There's not really a good errno we can report here.
> > > > +			 */
> > > > +			error_errno = ENOENT;
> > >
> > > Agreed there's not a good errno, but let's use a fake errno that we're mighty
> > > unlikely to confuse with an actual case of libc returning that errno.  Like
> > > one of EBADF or EOWNERDEAD.
> > 
> > Can we rely on that to be present on all platforms, including windows?
> 
> I expect EBADF is universal.  EBADF would be fine.

Hm, that's actually an error that could happen for other reasons, and IMO
would be more confusing than ENOENT. The latter describes the issue to a
reasonable extent, EBADFD seems like it would be more confusing.

I'm not sure it's worth investing time in this - it really shouldn't happen,
and we probably have bigger problems than the error code if it does. But if we
do want to do something, I think I can see a way to report a dedicated error
message for this.


> EOWNERDEAD is from 2006.
> https://learn.microsoft.com/en-us/cpp/c-runtime-library/errno-constants?view=msvc-140
> says VS2015 had EOWNERDEAD (the page doesn't have links for older Visual
> Studio versions, so I consider them unknown).

Oh, that's a larger list than I'd have though.


> https://github.com/coreutils/gnulib/blob/master/doc/posix-headers/errno.texi
> lists some OSs not having it, the newest of which looks like NetBSD 9.3
> (2022).  We could use it and add a #define for platforms lacking it.

What would we define it as?  I guess we could just pick a high value, but...


Greetings,

Andres Freund

  80. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-12T23:23:47Z

    Hi,

Attached is v2.7, with the following changes:

- Significantly deduplicated AIO related code bufmgr.c

  Previously the code for temp and shared buffers was duplicated to an
  uncomfortable degree. Now there is a common helper to implements the
  behaviour for both cases.

  The BM_PIN_COUNT_WAITER supporting code was also deduplicated, by
  introducing a helper function.


- Fixed typos / improved phrasing, per Noah's review


- Add comment explaining why retries for EAGAIN for io_uring_enter syscall
  failures don't seem to make sense, improve related error messages slightly


- Added a comment to aio.h explaining that aio_types.h might suffice for
  function declarations and aio_init.h for initialization related code.


- Added and expanded comments for PgAioHandleState, explaining the state
  machine in more detail.


- Updated README to mention the stage callback (instead of the outdated
  "prepare"), plus some other minor cleanups.


- Added a commit rephrasing track_io_timing related docs to talk about waits


- Added FIXME to method_uring.c about the set_max_safe_fds() issue. Depending
  on when/how that is resolved, the relevant commits can be reordered relative
  to the rest.


- Improved localbuf: patches and commit messages, as per Melanie's review


- Added FIXMEs to the bufmgr.c write support (only in later commit, unlikely
  to be realistic for 18) denoting that deadlock risk needs to be
  addressed. We probably need some lwlock.c improvements to make that
  race-free, otherwise I'd just have fixed this.


- Added a comment discussing the placement of the
  TRACE_POSTGRESQL_BUFFER_READ_DONE callback


- removed a few debug ereports() from the StartReadBuffers patch


Unresolved:

- Whether to continue starting new workers in process_pm_child_exit()


- What to name the view (currently pg_aios). I'm inclined to go for
  pg_io_handles right now.


- set_max_safe_fds() related issues for the io_uring backend


Greetings,

Andres Freund

  81. Re: AIO v2.5

    Antonin Houska <ah@cybertec.at> — 2025-03-13T10:53:03Z

    Andres Freund <andres@anarazel.de> wrote:

> Attached is v2.7, with the following changes:

Attached are a few proposals for minor comment fixes.

Besides that, it occurred to me when I was trying to get familiar with the
patch set (respectable work, btw) that an additional Assert() statement could
make sense:

diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index a9c351eb0dc..325688f0f23 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -413,6 +413,7 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
        bool            needs_synchronous;
 
        Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+       Assert(pgaio_my_backend->handed_out_io == ioh);
        Assert(pgaio_io_has_target(ioh));
 
        ioh->op = op;

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

  82. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-14T19:43:15Z

    Hi,

On 2025-03-13 11:53:03 +0100, Antonin Houska wrote:
> Attached are a few proposals for minor comment fixes.

Thanks, applied.


> Besides that, it occurred to me when I was trying to get familiar with the
> patch set (respectable work, btw) that an additional Assert() statement could
> make sense:

Yea, it does. I added it to another place as well.


Attached is v2.8 with the following changes:

- I wasn't happy with the way StartReadBuffers(), WaitReadBuffers() and
  AsyncReadBuffers() interacted. The io_method=sync path in particular was too
  cute by half, calling WaitReadBuffers() from within WaitReadBuffers().

  I think the new state considerably better.

  Plenty other smaller changes as part of that. One worth calling out is that
  ReadBuffersCanStartIO() now submits staged IO before actually blocking. Not
  the prettiest code, but I think it's ok.


- Added a function to assert the sanitiy of a ReadBuffersOperation

  While doing the refactoring for the prior point, I temporarily had a bug
  that returned buffers for which IO wasn't actually performed. Surprisingly
  the only assertion that triggered was when that buffer was read again by
  another operation, because it had been marked dirty, despite never being
  valid.

  Now there's a function that can be used to check that the buffers referenced
  by a ReadBuffersOperation are in a sane state.

  I guess this could be committed independently, but it'd not be entirely
  trivial to extract, so I'm currently leaning against doing that.


- Previously VacuumCostActive accounting happened after IO completion. But
  that doesn't seem quite right, it'd allow to submit a lot of IO at
  once. It's now moved to AsyncReadBuffers()


- With io_method=sync or with worker and temp tables, smgrstartreadv() would
  actually execute the IO. But the time accounting was done entirely around
  pgaio_wref_wait(). Now it's done in both places.


- Rebased onto newer version of Thomas' read_stream.c changes

  With that newer version the integration with read stream for actually doing
  AIO is a bit simpler.  There's one FIXME in the patch, because I don't
  really understand what a comment is referring to.

  I also split out a more experimental patch to make more efficient use of
  batching in read stream, the heuristics are more complicated, and it works
  well enough without.


- I added a commit to clean up the buffer access accounting for the case that
  a buffer was read in concurrently. That IMO is somewhat bogus on master, and
  it seemed to get more bogus with AIO.


- Integrated Antonin Houska's fixes and Assert suggestion


- Added a patch to address the smgr.c/md.c interrupt issue (a problem in master), see
  https://postgr.es/m/3vae7l5ozvqtxmd7rr7zaeq3qkuipz365u3rtim5t5wdkr6f4g@vkgf2fogjirl



I think the reasonable next steps are:

- Commit "localbuf: *" commits


- Commit temp table tests, likely after lowering the minimum temp_buffers setting


- Pursue a fix of the smgr interupt issue on the referenced thread

  This can happen in parallel with AIO patches up to
  "aio: Implement support for reads in smgr/md/fd"


- Commit the core AIO infrastructure patch after doing a few more passes


- Commit IO worker support


- In parallel: Find a way to deal with the set_max_safe_fds() issue that we've
  been discussing on this thread recently. As that only affects io_uring, it
  doesn't have to block other patches going in.


- Do a round of review of the read_stream changes Thomas recently posted (and
  that are also included here)


- Try to get some more review for the bufmgr.c related changes. I've whacked
  them around a fair bit lately.


- Try to get Thomas to review my read_stream.c changes



Open items:

- The upstream BAS_BULKREAD is so small that throughput is substantially worse
  once a table reaches 1/4 shared_buffers. That patch in the patchset as-is is
  probably not good enough, although I am not sure about that


- The set_max_safe_fds() issue for io_uring


- Right now effective_io_concurrency cannot be set > 0 on Windows and other
  platforms that lack posix_fadvise. But with AIO we can read ahead without
  posix_fadvise().

  It'd not really make anything worse than today to not remove the limit, but
  it'd be pretty weird to prevent windows etc from benefiting from AIO.  Need
  to look around and see whether it would require anything other than doc
  changes.


Greetings,

Andres Freund

  83. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-14T19:58:43Z

    Hi,

On 2025-03-14 15:43:15 -0400, Andres Freund wrote:
> Open items:
>
> - The upstream BAS_BULKREAD is so small that throughput is substantially worse
>   once a table reaches 1/4 shared_buffers. That patch in the patchset as-is is
>   probably not good enough, although I am not sure about that
>
>
> - The set_max_safe_fds() issue for io_uring
>
>
> - Right now effective_io_concurrency cannot be set > 0 on Windows and other
>   platforms that lack posix_fadvise. But with AIO we can read ahead without
>   posix_fadvise().
>
>   It'd not really make anything worse than today to not remove the limit, but
>   it'd be pretty weird to prevent windows etc from benefiting from AIO.  Need
>   to look around and see whether it would require anything other than doc
>   changes.

A fourth, smaller, question:

- Should the docs for debug_io_direct be rephrased and if so, how?

  Without read-stream-AIO debug_io_direct=data has completely unusable
  performance if there's ever any data IO - and if there's no IO there's no
  point in using the option.

  Now there is a certain set of workloads where performance with
  debug_io_direct=data can be better than master, sometimes substantially
  so. But at the same time, without support for at least:

  - AIO writes for at least checkpointer, bgwriter

    doing one synchronous IO for each buffer is ... slow.


  - read-streamified index vacuuming


  And probably also:
  - AIO-ified writes for writes executed by backends, e.g. due to strategies

    Doing one synchronous IO for each buffer is ... slow. And e.g. with COPY
    we do a *lot* of those. OTOH, it could be fine if most modifications are
    done via INSERTs instead of COPY.


  - prefetching for non-BHS index accesses

    Without prefetching, a well correlated index-range scan will be orders of
    magnitude slower with DIO.


  - Anything bypassing shared_buffers, like RelationCopyStorage() or
    bulk_write.c will be extremely slow

    The only saving grace is that these aren't all *that* common.


Due to those constraints I think it's pretty clear we can't remove the debug_
prefix at this time.

Perhaps it's worth going from

       <para>
        Currently this feature reduces performance, and is intended for
        developer testing only.
       </para>
to
       <para>
        Currently this feature reduces performance in many workloads, and is
        intended for testing only.
       </para>

I.e. qualify the downside with "many workloads" and widen the audience ever so
slightly?

Greetings,

Andres Freund

  84. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-17T20:00:53Z

    Hi,

Attached is v2.9 with the following, fairly small, changes:
- Rebased ontop of the additional committed read stream patches

- Committed the localbuf: refactorings (but not the change to expand
  refcounting of local buffers, that seems a bit more dependent on the rest)

- Committed the temp table test after some annoying debugging
  https://postgr.es/m/w5wr26ijzp7xz2qrxkt6dzvmmn2tn6tn5fp64y6gq5iuvg43hw%40v4guo6x776dq

- Some rephrasing and moving of comments in the first two commits

- There was a small bug in the smgr reopen call I found when reviewing, the
  PgAioOpData->read.fd field was referenced for both reads and writes, which
  failed to fail because both read/write use a compatible struct layout.


Unless I hear otherwise, I plan to commit the first two patches fairly soon,
followed by the worker support patches a few buildfarm cycles later.

I'm sure there's a bunch of stuff worth improving in the AIO infrastructure
and I can't imagine a project of this size not having bugs. But I think it's
in a state where that's better worked out in tree, with broader exposure and
testing.

Greetings,

Andres Freund

  85. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-17T20:53:57Z

    On Fri, Mar 14, 2025 at 3:43 PM Andres Freund <andres@anarazel.de> wrote:
>
> Open items:
>
> - Right now effective_io_concurrency cannot be set > 0 on Windows and other
>   platforms that lack posix_fadvise. But with AIO we can read ahead without
>   posix_fadvise().
>
>   It'd not really make anything worse than today to not remove the limit, but
>   it'd be pretty weird to prevent windows etc from benefiting from AIO.  Need
>   to look around and see whether it would require anything other than doc
>   changes.

I've attached a patch that removes the limit for
effective_io_concurrency and maintenance_io_concurrency. I tested both
GUCs with fadvise manually disabled on my system and I think it is
working for those read stream users I tried (vacuum and BHS).

I checked around to make sure no one was using only the value of the
guc to guard prefetches, and it seems like we're safe.

The one thing I am wondering about with the docs is whether or not we
need to make it more clear that only a subset of the "simultaneous
I/O" behavior controlled by eic/mic is available if your system
doesn't have fadvise. I tried to do that a bit, but I avoided getting
into too many details.

- Melanie

  86. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-18T20:12:18Z

    Hi,

Attached is v2.10, with the following changes:

- committed core AIO infrastructure patch

  A few cosmetic changes


- committed io_method=worker

  Two non-cosmetic changes:

  - The shmem allocation functions over-estimated the amount of shared memory
    required.

  - pgaio_worker_shmem_init() should initialize up to MAX_IO_WORKERS, not just
    io_workers, since the latter is intentionally PGC_SIGHUP (found by Thomas)


- Bunch of typo fixes found by searching for repeated words

  Thomas found one and then I searched for more.


- Added Melanie's patch to allow effective_io_concurrency to be set on windows etc


- Fixed a reference to an outdated function reference (thanks to Bilal)


- Reordered patches to be a bit more in dependency order

  E.g. "bufmgr: Implement AIO read support" doesn't depend on Thomas' "buffer
  forwarding" patches and thus can be commited before those go in.


Next steps:

- Decide what to do about the smgr interrupt issue

  I guess it could be deferred, based on the argument it only matters with a
  sufficiently high debug level, but I don't feel comfortable with that.

  I think it'd be reasonable to just go with the patch I sent on the other
  thread (and included here).


- Get somebody to look at
  "bufmgr: Improve stats when buffer was read in concurrently"

  This arguably fixes a bug, or just weird behaviour, on master.


- Address the set_max_safe_fds() issue and once resolved, commit io_uring
  method

  Can happen concurrently with next steps


- Commit "aio: Implement support for reads in smgr/md/fd"


- Get somebody to do one more pass at bufmgr related commits, I think they
  could use a less in-the-weeds eye.

  That's the following commits:
  - localbuf: Track pincount in BufferDesc as well
  - bufmgr: Implement AIO read support
  - bufmgr: Use AIO in StartReadBuffers()
  - aio: Basic read_stream adjustments for real AIO


Questions / Unresolved:

- Write support isn't going to land in 18, but there is a tiny bit of code
  regarding writes in the code for bufmgr IO. I guess I could move that to a
  later commit?

  I'm inclined to leave it, the structure of the code only really makes
  knowing that it's going to be shared between reads & writes.


- pg_aios view name


Greetings,

Andres Freund

  87. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-19T01:00:17Z

    On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
> Attached is v2.10,

This is a review of 0008:     bufmgr: Implement AIO read support

I'm afraid it is more of a cosmetic review than a sign-off on the
patch's correctness, but perhaps it will help future readers who may
have the same questions I did.

In the commit message:
    bufmgr: Implement AIO read support

    This implements the following:
    - Addition of callbacks to maintain buffer state on completion of a readv
    - Addition of a wait reference to BufferDesc, to allow backends to wait for
      IOs
    - StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO

I think it might be nice to say something about allowing backends to
complete IOs issued by other backends.

@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
    CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),

    CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+   CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+   CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
 #undef CALLBACK_ENTRY
 };

I personally can't quite figure out why the read and write callbacks
are defined differently than the stage, complete, and report
callbacks. I know there is a comment above PgAioHandleCallbackID about
something about this, but it didn't really clarify it for me. Maybe
you can put a block comment at the top of aio_callback.c? Or perhaps I
just need to study it more...

@@ -5482,10 +5503,19 @@ WaitIO(BufferDesc *buf)
+       if (pgaio_wref_valid(&iow))
+       {
+           pgaio_wref_wait(&iow);
+           ConditionVariablePrepareToSleep(cv);
+           continue;
+       }

I'd add some comment above this.  I reread it many times, and I still
only _think_ I understand what it does. I think the reason we need
ConditionVariablePrepareToSleep() again is because pgaio_io_wait() may
have called ConditionVariableCancelSleep() so we need to
ConditionVariablePrepareToSleep() again (it was done already at the
top of Wait())?

I'll admit I find the CV API quite confusing, so maybe I'm just
misunderstanding it.

Maybe worth mentioning in the commit message about why WaitIO() has to
work differently for AIO than sync IO.

    /*
     * Support LockBufferForCleanup()
     *
     * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
     * Most of the time the current backend will hold another pin preventing
     * that from happening, but that's e.g. not the case when completing an IO
     * another backend started.
     */

I found this wording a bit confusing. what about this:

We may have just released the last pin other than the waiter's. In most cases,
this backend holds another pin on the buffer. But, if, for example, this
backend is completing an IO issued by another backend, it may be time to wake
the waiter.

/*
 * Helper for AIO staging callback for both reads and writes as well as temp
 * and shared buffers.
 */
static pg_attribute_always_inline void
buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)

I think buffer_stage_common() needs the function comment to explain
what unit it is operating on.
It is called "buffer_" singular but then it loops through io_data
which appears to contain multiple buffers

        /*
         * Check that all the buffers are actually ones that could conceivably
         * be done in one IO, i.e. are sequential.
         */
        if (i == 0)
            first = buf_hdr->tag;
        else
        {
            Assert(buf_hdr->tag.relNumber == first.relNumber);
            Assert(buf_hdr->tag.blockNum == first.blockNum + i);
        }

So it is interesting to me that this validation is done at this level.
Enforcing sequentialness doesn't seem like it would be intrinsically
related to or required to stage IOs. And there isn't really anything
in this function that seems like it would require it either. Usually
an assert is pretty close to the thing it is protecting.

Oh and I think the end of the loop in buffer_stage_common() would look
nicer with a small refactor with the resulting code looking like this:

        /* temp buffers don't use BM_IO_IN_PROGRESS */
        Assert(!is_temp || (buf_state & BM_IO_IN_PROGRESS));

        /* we better have ensured the buffer is present until now */
        Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);

        /*
         * Reflect that the buffer is now owned by the subsystem.
         *
         * For local buffers: This can't be done just in LocalRefCount as one
         * might initially think, as this backend could error out while AIO is
         * still in progress, releasing all the pins by the backend itself.
         */
        buf_state += BUF_REFCOUNT_ONE;
        buf_hdr->io_wref = io_ref;

        if (is_temp)
        {
            pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
            continue;
        }

        UnlockBufHdr(buf_hdr, buf_state);

        if (is_write)
        {
            LWLock       *content_lock;

            content_lock = BufferDescriptorGetContentLock(buf_hdr);

            Assert(LWLockHeldByMe(content_lock));

            /*
             * Lock is now owned by AIO subsystem.
             */
            LWLockDisown(content_lock);
        }

        /*
         * Stop tracking this buffer via the resowner - the AIO system now
         * keeps track.
         */
        ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
    }

In buffer_readv_complete(), this comment

    /*
     * Iterate over all the buffers affected by this IO and call appropriate
     * per-buffer completion function for each buffer.
     */

makes it sound like we might invoke different completion functions (like invoke
the completion callback), but that isn't what happens here.

        failed =
            prior_result.status == ARS_ERROR
            || prior_result.result <= buf_off;

Though not introduced in this commit, I will say that I find the ARS prefix not
particularly helpful. Though not as brief, something like AIO_RS_ERROR would
probably be more clear.

@@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
  * Like StartBufferIO, but for local buffers
  */
 bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
 {
-   uint32      buf_state = pg_atomic_read_u32(&bufHdr->state);
+   uint32      buf_state;
+
+   /*
+    * The buffer could have IO in progress, e.g. when there are two scans of
+    * the same relation. Either wait for the other IO or return false.
+    */
+   if (pgaio_wref_valid(&bufHdr->io_wref))
+   {
+       PgAioWaitRef iow = bufHdr->io_wref;
+
+       if (nowait)
+           return false;
+
+       pgaio_wref_wait(&iow);
+   }

+   buf_state = pg_atomic_read_u32(&bufHdr->state);
    if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
    {
        /* someone else already did the I/O */

I'd move this comment ("someone else already did") outside of the if
statement so it kind of separates it into three clear cases:
1) the IO is in progress and you can wait on it if you want, 2) the IO
is completed by someone else (is this possible for local buffers
without AIO?) 3) you can start the IO

- Melanie

  88. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-19T01:50:11Z

    Hi,

On 2025-03-18 21:00:17 -0400, Melanie Plageman wrote:
> On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
> > Attached is v2.10,
> 
> This is a review of 0008:     bufmgr: Implement AIO read support
> 
> I'm afraid it is more of a cosmetic review than a sign-off on the
> patch's correctness, but perhaps it will help future readers who may
> have the same questions I did.

I think that's actually an important level of review. I'm, as odd as that
sounds, more confident about the architectural stuff than about
"understandability" etc.


> In the commit message:
>     bufmgr: Implement AIO read support
> 
>     This implements the following:
>     - Addition of callbacks to maintain buffer state on completion of a readv
>     - Addition of a wait reference to BufferDesc, to allow backends to wait for
>       IOs
>     - StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
> 
> I think it might be nice to say something about allowing backends to
> complete IOs issued by other backends.

Hm, I'd have said that's basically implied by the way AIO works (as outlined
in the added README.md), but I can think of a way to mention it here.


> @@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
>     CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
> 
>     CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
> +
> +   CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
> +
> +   CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
>  #undef CALLBACK_ENTRY
>  };
> 
> I personally can't quite figure out why the read and write callbacks
> are defined differently than the stage, complete, and report
> callbacks. I know there is a comment above PgAioHandleCallbackID about
> something about this, but it didn't really clarify it for me. Maybe
> you can put a block comment at the top of aio_callback.c? Or perhaps I
> just need to study it more...

They're not implemented differently - PgAioHandleCallbacks (which is what is
contained in aio_handle_cbs, just with a name added) all have a stage,
complete and report callbacks.

E.g. for SHARED_BUFFER_READV you have a stage (to transfer the buffer pins to
the AIO subsystem), a shared completion (to verify the page, to remove
BM_IO_IN_PROGRESS and set BM_VALID/BM_IO_ERROR, as appropriate) and a report
callback (to report a page validation error).

Maybe more of the relevant types and functions should have been plural, but
then it becomes very awkward to talk about the separate registrations of
multiple callbacks (i.e. the set of callbacks for md.c and the set of
callbacks for bufmgr.c).


> @@ -5482,10 +5503,19 @@ WaitIO(BufferDesc *buf)
> +       if (pgaio_wref_valid(&iow))
> +       {
> +           pgaio_wref_wait(&iow);
> +           ConditionVariablePrepareToSleep(cv);
> +           continue;
> +       }
> 
> I'd add some comment above this.  I reread it many times, and I still
> only _think_ I understand what it does. I think the reason we need
> ConditionVariablePrepareToSleep() again is because pgaio_io_wait() may
> have called ConditionVariableCancelSleep() so we need to
> ConditionVariablePrepareToSleep() again (it was done already at the
> top of Wait())?

Oh, yes, that definitely needs a comment. I've been marinating in this for so
long that it seems obvious, but if I take a step back, it's not at all
obvious.

The issue is that pgaio_wref_wait() internally waits on a *different*
condition variable than the BufferDesc's CV.  The consequences of not doing
this would be fairly mild - the next ConditionVariableSleep would prepare to
sleep and return immediately - but it's unnecessary.


> Maybe worth mentioning in the commit message about why WaitIO() has to
> work differently for AIO than sync IO.

K.


>     /*
>      * Support LockBufferForCleanup()
>      *
>      * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
>      * Most of the time the current backend will hold another pin preventing
>      * that from happening, but that's e.g. not the case when completing an IO
>      * another backend started.
>      */
> 
> I found this wording a bit confusing. what about this:
> 
> We may have just released the last pin other than the waiter's. In most cases,
> this backend holds another pin on the buffer. But, if, for example, this
> backend is completing an IO issued by another backend, it may be time to wake
> the waiter.

WFM.


> /*
>  * Helper for AIO staging callback for both reads and writes as well as temp
>  * and shared buffers.
>  */
> static pg_attribute_always_inline void
> buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
> 
> I think buffer_stage_common() needs the function comment to explain
> what unit it is operating on.
> It is called "buffer_" singular but then it loops through io_data
> which appears to contain multiple buffers

Hm. Yea. Originally it was just for readv and was duplicated for writes. The
vectorized bit hinted at being for multiple buffers.


>         /*
>          * Check that all the buffers are actually ones that could conceivably
>          * be done in one IO, i.e. are sequential.
>          */
>         if (i == 0)
>             first = buf_hdr->tag;
>         else
>         {
>             Assert(buf_hdr->tag.relNumber == first.relNumber);
>             Assert(buf_hdr->tag.blockNum == first.blockNum + i);
>         }
> 
> So it is interesting to me that this validation is done at this level.
> Enforcing sequentialness doesn't seem like it would be intrinsically
> related to or required to stage IOs. And there isn't really anything
> in this function that seems like it would require it either. Usually
> an assert is pretty close to the thing it is protecting.

Staging is the last buffer-aware thing that happens before IO is actually
executed. If you were to do a readv() into *non* buffers that aren't for
sequential blocks, you would get bogus buffer pool contents, because obviously
it doesn't make sense to read data for block N+1 into the buffer for block N+3
or whatnot.

The assertions did find bugs during development, fwiw.


> Oh and I think the end of the loop in buffer_stage_common() would look
> nicer with a small refactor with the resulting code looking like this:
> 
>         /* temp buffers don't use BM_IO_IN_PROGRESS */
>         Assert(!is_temp || (buf_state & BM_IO_IN_PROGRESS));
> 
>         /* we better have ensured the buffer is present until now */
>         Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
> 
>         /*
>          * Reflect that the buffer is now owned by the subsystem.
>          *
>          * For local buffers: This can't be done just in LocalRefCount as one
>          * might initially think, as this backend could error out while AIO is
>          * still in progress, releasing all the pins by the backend itself.
>          */
>         buf_state += BUF_REFCOUNT_ONE;
>         buf_hdr->io_wref = io_ref;
> 
>         if (is_temp)
>         {
>             pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
>             continue;
>         }
> 
>         UnlockBufHdr(buf_hdr, buf_state);
> 
>         if (is_write)
>         {
>             LWLock       *content_lock;
> 
>             content_lock = BufferDescriptorGetContentLock(buf_hdr);
> 
>             Assert(LWLockHeldByMe(content_lock));
> 
>             /*
>              * Lock is now owned by AIO subsystem.
>              */
>             LWLockDisown(content_lock);
>         }
> 
>         /*
>          * Stop tracking this buffer via the resowner - the AIO system now
>          * keeps track.
>          */
>         ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
>     }

I don't particularly like this, I'd like to make the logic for shared and
local buffers more similar over time. E.g. by also tracking local buffer IO
via resowner.


> In buffer_readv_complete(), this comment
> 
>     /*
>      * Iterate over all the buffers affected by this IO and call appropriate
>      * per-buffer completion function for each buffer.
>      */
> 
> makes it sound like we might invoke different completion functions (like invoke
> the completion callback), but that isn't what happens here.

Oops, that's how it used to work, but it doesn't anymore, because it ended up
with too much duplication.


>         failed =
>             prior_result.status == ARS_ERROR
>             || prior_result.result <= buf_off;
> 
> Though not introduced in this commit, I will say that I find the ARS prefix not
> particularly helpful. Though not as brief, something like AIO_RS_ERROR would
> probably be more clear.

Fair enough. I'd go for PGAIO_RS_ERROR etc though.


> @@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
>   * Like StartBufferIO, but for local buffers
>   */
>  bool
> -StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
> +StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
>  {
> -   uint32      buf_state = pg_atomic_read_u32(&bufHdr->state);
> +   uint32      buf_state;
> +
> +   /*
> +    * The buffer could have IO in progress, e.g. when there are two scans of
> +    * the same relation. Either wait for the other IO or return false.
> +    */
> +   if (pgaio_wref_valid(&bufHdr->io_wref))
> +   {
> +       PgAioWaitRef iow = bufHdr->io_wref;
> +
> +       if (nowait)
> +           return false;
> +
> +       pgaio_wref_wait(&iow);
> +   }
> 
> +   buf_state = pg_atomic_read_u32(&bufHdr->state);
>     if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
>     {
>         /* someone else already did the I/O */
> 
> I'd move this comment ("someone else already did") outside of the if
> statement so it kind of separates it into three clear cases:

FWIW it's inside because that's how StartBufferIOs comment has been for a fair
while...


> 1) the IO is in progress and you can wait on it if you want,
> 2) the IO is completed by someone else (is this possible for local buffers
> without AIO?)

No, that's not possible without AIO.


> 3) you can start the IO

I'll give it a go.


Thanks for the review!


Greetings,

Andres Freund

  89. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-19T14:47:50Z

    On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
>
> Attached is v2.10

This is a review of 0002:  bufmgr: Improve stats when buffer is read
in concurrently

In the commit message, it might be worth distinguishing that
pg_stat_io and vacuum didn't double count reads, they under-counted
hits. pgBufferUsage and relation-level stats (pg_stat_all_tables etc)
overcounted reads and undercounted hits.

Quick example:
On master, if we try to read 7 blocks and 3 were hits and 2 were
completed by someone else then
- pg_stat_io and VacuumCostBalance would record 3 hits and 2 reads,
which looks like 2 misses.
- pgBufferUsage would record 3 hits and 4 reads, which looks like 4 misses.
- pg_stat_all_tables would record 3 hits and 7 reads, which looks like 4 misses.

The correct number of misses is 2 misses comprising 5 hits and 2 reads
(or 7 reads and 5 hits for pg_stat_all_tables which does the math
later).

@@ -1463,8 +1450,13 @@ WaitReadBuffers(ReadBuffersOperation *operation)
        if (!WaitReadBuffersCanStartIO(buffers[i], false))
        {
            /*
-            * Report this as a 'hit' for this backend, even though it must
-            * have started out as a miss in PinBufferForBlock().
+            * Report and track this as a 'hit' for this backend, even though
+            * it must have started out as a miss in PinBufferForBlock().
+            *
+            * Some of the accesses would otherwise never be counted (e.g.
+            * pgBufferUsage) or counted as a miss (e.g.
+            * pgstat_count_buffer_hit(), as we always call
+            * pgstat_count_buffer_read()).
             */

I think this comment should be changed. It reads like something
written when discovering this problem and not like something useful in
the future. I think you can probably drop the whole second paragraph.

You could make it even more clear by mentioning that the other backend
will count it as a read.

Otherwise, LGTM


- Melanie

  90. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-19T17:20:17Z

    On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
>
> Attached is v2.10,

I noticed a few comments could be improved in  0011: bufmgr: Use AIO
in StartReadBuffers()

In WaitReadBuffers(), this comment is incomplete:

        /*
-        * Skip this block if someone else has already completed it.  If an
-        * I/O is already in progress in another backend, this will wait for
-        * the outcome: either done, or something went wrong and we will
-        * retry.
+        * If there is an IO associated with the operation, we may need to
+        * wait for it. It's possible for there to be no IO if
         */

In WaitReadBuffers(), too many thes

        /*
         * Most of the the the one IO we started will read in everything.  But
         * we need to deal with short reads and buffers not needing IO
         * anymore.
         */

In ReadBuffersCanStartIO()

+       /*
+        * Unfortunately a false returned StartBufferIO() doesn't allow to
+        * distinguish between the buffer already being valid and IO already
+        * being in progress. Since IO already being in progress is quite
+        * rare, this approach seems fine.
+        */

maybe reword "a false returned StartBufferIO()"

Above and in AsyncReadBuffers()

 * To support retries after short reads, the first operation->nblocks_done is
 * buffers are skipped.

can't quite understand this

+ * On return *nblocks_progres is updated to reflect the number of buffers
progress spelled wrong

     * A secondary benefit is that this would allows us to measure the time in
     * pgaio_io_acquire() without causing undue timer overhead in the common,
     * non-blocking, case.  However, currently the pgstats infrastructure
     * doesn't really allow that, as it a) asserts that an operation can't
     * have time without operations b) doesn't have an API to report
     * "accumulated" time.
     */

allows->allow

What would the time spent in pgaio_io_acquire() be reported as? Time
submitting IOs? Time waiting for a handle? And what is "accumulated"
time here? It seems like you just add the time to the running total
and that is already accumulated.

- Melanie

  91. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-19T19:28:39Z

    Hi,

On 2025-03-19 13:20:17 -0400, Melanie Plageman wrote:
> On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > Attached is v2.10,
> 
> I noticed a few comments could be improved in  0011: bufmgr: Use AIO
> in StartReadBuffers()
> [...]

Yep.


> Above and in AsyncReadBuffers()
> 
>  * To support retries after short reads, the first operation->nblocks_done is
>  * buffers are skipped.
> 
> can't quite understand this

Heh, yea, it's easy to misunderstand. "short read" in the sense of a partial
read, i.e. a preadv() that only read some of the blocks, not all. I'm
replacing the "short" with partial.

(also removed the superfluous "is")



>      * A secondary benefit is that this would allows us to measure the time in
>      * pgaio_io_acquire() without causing undue timer overhead in the common,
>      * non-blocking, case.  However, currently the pgstats infrastructure
>      * doesn't really allow that, as it a) asserts that an operation can't
>      * have time without operations b) doesn't have an API to report
>      * "accumulated" time.
>      */
> 
> allows->allow
> 
> What would the time spent in pgaio_io_acquire() be reported as?

I'd report it as additional time for the IO we're trying to start, as that
wait would otherwise not happen.


> And what is "accumulated" time here? It seems like you just add the time to
> the running total and that is already accumulated.

Afaict there currently is no way to report a time delta to
pgstat. pgstat_count_io_op_time() computes the time since
pgstat_prepare_io_time(). Due to the assertions that time cannot be reported
for an operation with a zero count, we can't just do two
  pgstat_prepare_io_time(); ...; pgstat_count_io_op_time();
twice, with the first one passing cnt=0.

Greetings,

Andres Freund

  92. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-19T21:25:30Z

    On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote:
> On 2025-03-11 20:57:43 -0700, Noah Misch wrote:
> > - Like you say, "redefine max_files_per_process to be about the number of
> >   files each *backend* will additionally open".  It will become normal that
> >   each backend's actual FD list length is max_files_per_process + MaxBackends
> >   if io_method=io_uring.  Outcome is not unlike
> >   v6-0002-Bump-postmaster-soft-open-file-limit-RLIMIT_NOFIL.patch +
> >   v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but we don't
> >   mutate max_files_per_process.  Benchmark results should not change beyond
> >   the inter-major-version noise level unless one sets io_method=io_uring.  I'm
> >   feeling best about this one, but I've not been thinking about it long.
> 
> Yea, I think that's something probably worth doing separately from Jelte's
> patch.  I do think that it'd be rather helpful to have jelte's patch to
> increase NOFILE in addition though.

Agreed.

> > > > > +static void
> > > > > +maybe_adjust_io_workers(void)
> > > >
> > > > This also restarts workers that exit, so perhaps name it
> > > > start_io_workers_if_missing().
> > > 
> > > But it also stops IO workers if necessary?
> > 
> > Good point.  Maybe just add a comment like "start or stop IO workers to close
> > the gap between the running count and the configured count intent".
> 
> It's now
> /*
>  * Start or stop IO workers, to close the gap between the number of running
>  * workers and the number of configured workers.  Used to respond to change of
>  * the io_workers GUC (by increasing and decreasing the number of workers), as
>  * well as workers terminating in response to errors (by starting
>  * "replacement" workers).
>  */

Excellent.

> > > > > +{
> > > > ...
> > > > > +		/* Try to launch one. */
> > > > > +		child = StartChildProcess(B_IO_WORKER);
> > > > > +		if (child != NULL)
> > > > > +		{
> > > > > +			io_worker_children[id] = child;
> > > > > +			++io_worker_count;
> > > > > +		}
> > > > > +		else
> > > > > +			break;				/* XXX try again soon? */
> > > >
> > > > Can LaunchMissingBackgroundProcesses() become the sole caller of this
> > > > function, replacing the current mix of callers?  That would be more conducive
> > > > to promptly doing the right thing after launch failure.
> > > 
> > > I'm not sure that'd be a good idea - right now IO workers are started before
> > > the startup process, as the startup process might need to perform IO. If we
> > > started it only later in ServerLoop() we'd potentially do a fair bit of work,
> > > including starting checkpointer, bgwriter, bgworkers before we started IO
> > > workers.  That shouldn't actively break anything, but it would likely make
> > > things slower.
> > 
> > I missed that.  How about keeping the two calls associated with PM_STARTUP but
> > replacing the assign_io_workers() and process_pm_child_exit() calls with one
> > in LaunchMissingBackgroundProcesses()?
> 
> I think replacing the call in assign_io_workers() is a good idea, that way we
> don't need assign_io_workers().
> 
> Less convinced it's a good idea to do the same for process_pm_child_exit() -
> if IO workers errored out we'll launch backends etc before we get to
> LaunchMissingBackgroundProcesses(). That's not a fundamental problem, but
> seems a bit odd.

Works for me.

> I think LaunchMissingBackgroundProcesses() should be split into one that
> starts aux processes and one that starts bgworkers. The one maintaining aux
> processes should be called before we start backends, the latter not.

That makes sense, though I've not thought about it much.

> > > > > +			/*
> > > > > +			 * It's very unlikely, but possible, that reopen fails. E.g. due
> > > > > +			 * to memory allocations failing or file permissions changing or
> > > > > +			 * such.  In that case we need to fail the IO.
> > > > > +			 *
> > > > > +			 * There's not really a good errno we can report here.
> > > > > +			 */
> > > > > +			error_errno = ENOENT;
> > > >
> > > > Agreed there's not a good errno, but let's use a fake errno that we're mighty
> > > > unlikely to confuse with an actual case of libc returning that errno.  Like
> > > > one of EBADF or EOWNERDEAD.
> > > 
> > > Can we rely on that to be present on all platforms, including windows?
> > 
> > I expect EBADF is universal.  EBADF would be fine.
> 
> Hm, that's actually an error that could happen for other reasons, and IMO
> would be more confusing than ENOENT. The latter describes the issue to a
> reasonable extent, EBADFD seems like it would be more confusing.
> 
> I'm not sure it's worth investing time in this - it really shouldn't happen,
> and we probably have bigger problems than the error code if it does. But if we
> do want to do something, I think I can see a way to report a dedicated error
> message for this.

I agree it's not worth much investment.  Let's leave that one as-is.  We can
always change it further if the not-really-good errno shows up too much.

> > https://github.com/coreutils/gnulib/blob/master/doc/posix-headers/errno.texi
> > lists some OSs not having it, the newest of which looks like NetBSD 9.3
> > (2022).  We could use it and add a #define for platforms lacking it.
> 
> What would we define it as?  I guess we could just pick a high value, but...

Some second-best value, but I withdraw that idea.

On Wed, Mar 12, 2025 at 07:23:47PM -0400, Andres Freund wrote:
> Attached is v2.7, with the following changes:

> Unresolved:
> 
> - Whether to continue starting new workers in process_pm_child_exit()

I'm fine with that continuing.  It's hurting ~nothing.

> - What to name the view (currently pg_aios). I'm inclined to go for
>   pg_io_handles right now.

I like pg_aios mildly better than pg_io_handles, since "handle" sounds
implementation-centric.

On Fri, Mar 14, 2025 at 03:43:15PM -0400, Andres Freund wrote:
> Attached is v2.8 with the following changes:

> - In parallel: Find a way to deal with the set_max_safe_fds() issue that we've
>   been discussing on this thread recently. As that only affects io_uring, it
>   doesn't have to block other patches going in.

As above, I like the "redefine" option.

> - Right now effective_io_concurrency cannot be set > 0 on Windows and other
>   platforms that lack posix_fadvise. But with AIO we can read ahead without
>   posix_fadvise().
> 
>   It'd not really make anything worse than today to not remove the limit, but
>   it'd be pretty weird to prevent windows etc from benefiting from AIO.  Need
>   to look around and see whether it would require anything other than doc
>   changes.

Worth changing, but non-blocking.

On Fri, Mar 14, 2025 at 03:58:43PM -0400, Andres Freund wrote:
> - Should the docs for debug_io_direct be rephrased and if so, how?

> Perhaps it's worth going from
> 
>        <para>
>         Currently this feature reduces performance, and is intended for
>         developer testing only.
>        </para>
> to
>        <para>
>         Currently this feature reduces performance in many workloads, and is
>         intended for testing only.
>        </para>
> 
> I.e. qualify the downside with "many workloads" and widen the audience ever so
> slightly?

Yes, that's good.


Other than the smgr patch review sent on its own thread, I've not yet reviewed
any of these patches comprehensively.  Given the speed of change, I felt it
was time to flush comments buffered since 2025-03-11:

commit 0284401 wrote:
>     aio: Basic subsystem initialization

> @@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
>  		 */
>  		LWLockReleaseAll();
>  		pgstat_report_wait_end();
> +		pgaio_error_cleanup();

AutoVacLauncherMain(), BackgroundWriterMain(), CheckpointerMain(), and
WalWriterMain() call AtEOXact_Buffers() but not AtEOXact_Aio().  Is that
proper?  They do call pgaio_error_cleanup() as seen here, so the only loss is
some asserts.  (The load-bearing part does get done.)

commit da72269 wrote:
>     aio: Add core asynchronous I/O infrastructure

> + * This could be in aio_internal.h, as it is not pubicly referenced, but

typo -> publicly

commit 55b454d wrote:
>     aio: Infrastructure for io_method=worker

> +		/* Try to launch one. */
> +		child = StartChildProcess(B_IO_WORKER);
> +		if (child != NULL)
> +		{
> +			io_worker_children[id] = child;
> +			++io_worker_count;
> +		}
> +		else
> +			break;				/* XXX try again soon? */

I'd change the comment to something like one of:

  retry after DetermineSleepTime()
  next LaunchMissingBackgroundProcesses() will retry in <60s

On Tue, Mar 18, 2025 at 04:12:18PM -0400, Andres Freund wrote:
> - Decide what to do about the smgr interrupt issue

Replied on that thread.  It's essentially ready.

> Questions / Unresolved:
> 
> - Write support isn't going to land in 18, but there is a tiny bit of code
>   regarding writes in the code for bufmgr IO. I guess I could move that to a
>   later commit?
> 
>   I'm inclined to leave it, the structure of the code only really makes
>   knowing that it's going to be shared between reads & writes.

Fine to leave it.

> - pg_aios view name

Covered above.

> Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support

Some comments about BM_IO_IN_PROGRESS may need updates.  This paragraph:

* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
per-buffer LWLock).  The process doing a read or write sets the flag for the
duration, and processes that need to wait for it to be cleared sleep on a
condition variable.

And these individual lines from "git grep BM_IO_IN_PROGRESS":
		 * I/O already in progress.  We already hold BM_IO_IN_PROGRESS for the
	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
 *	i.e at most one BM_IO_IN_PROGRESS bit is set per proc.

The last especially.  For the other three lines and the paragraph, the notion
of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
being the process "doing a read" becomes less significant when one process
starts the IO and another completes it.

> +		/* we better have ensured the buffer is present until now */
> +		Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);

I'd delete that comment; to me, the assertion alone is clearer.

> +			ereport(LOG,
> +					(errcode(ERRCODE_DATA_CORRUPTED),
> +					 errmsg("invalid page in block %u of relation %s; zeroing out page",

This is changing level s/WARNING/LOG/.  That seems orthogonal to the patch's
goals; is it needed?  If so, I recommend splitting it out as a preliminary
patch, to highlight the behavior change for release notes.

> +/*
> + * Perform completion handling of a single AIO read. This read may cover
> + * multiple blocks / buffers.
> + *
> + * Shared between shared and local buffers, to reduce code duplication.
> + */
> +static pg_attribute_always_inline PgAioResult
> +buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
> +					  uint8 cb_data, bool is_temp)
> +{
> +	PgAioResult result = prior_result;
> +	PgAioTargetData *td = pgaio_io_get_target_data(ioh);
> +	uint64	   *io_data;
> +	uint8		handle_data_len;
> +
> +	if (is_temp)
> +	{
> +		Assert(td->smgr.is_temp);
> +		Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
> +	}
> +	else
> +		Assert(!td->smgr.is_temp);
> +
> +	/*
> +	 * Iterate over all the buffers affected by this IO and call appropriate
> +	 * per-buffer completion function for each buffer.
> +	 */
> +	io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
> +	for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
> +	{
> +		Buffer		buf = io_data[buf_off];
> +		PgAioResult buf_result;
> +		bool		failed;
> +
> +		Assert(BufferIsValid(buf));
> +
> +		/*
> +		 * If the entire failed on a lower-level, each buffer needs to be

Missing word, probably fix like:
s,entire failed on a lower-level,entire I/O failed on a lower level,

> +		 * marked as failed. In case of a partial read, some buffers may be
> +		 * ok.
> +		 */
> +		failed =
> +			prior_result.status == ARS_ERROR
> +			|| prior_result.result <= buf_off;

I didn't run an experiment to check the following, but I think this should be
s/<=/</.  Suppose we requested two blocks and read some amount of bytes
[1*BLCKSZ, 2*BLSCKSZ - 1].  md_readv_complete will store result=1.  buf_off==0
should compute failed=false here, but buf_off==1 should compute failed=true.

I see this relies on md_readv_complete having converted "result" to blocks.
Was there some win from doing that as opposed to doing the division here?
Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
to follow, to me.

> +
> +		buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
> +											   is_temp);
> +
> +		/*
> +		 * If there wasn't any prior error and the IO for this page failed in
> +		 * some form, set the whole IO's to the page's result.

s/the IO for this page/page verification/
s/IO's/IO's result/

> +		 */
> +		if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
> +		{
> +			result = buf_result;
> +			pgaio_result_report(result, td, LOG);
> +		}
> +	}
> +
> +	return result;
> +}

  93. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-19T22:17:37Z

    Hi,

On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote:
> > - Right now effective_io_concurrency cannot be set > 0 on Windows and other
> >   platforms that lack posix_fadvise. But with AIO we can read ahead without
> >   posix_fadvise().
> >
> >   It'd not really make anything worse than today to not remove the limit, but
> >   it'd be pretty weird to prevent windows etc from benefiting from AIO.  Need
> >   to look around and see whether it would require anything other than doc
> >   changes.
>
> Worth changing, but non-blocking.

Thankfully Melanie submitted a patch for that...


> Other than the smgr patch review sent on its own thread, I've not yet reviewed
> any of these patches comprehensively.  Given the speed of change, I felt it
> was time to flush comments buffered since 2025-03-11:

Thanks!


> commit 0284401 wrote:
> >     aio: Basic subsystem initialization
>
> > @@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
> >  		 */
> >  		LWLockReleaseAll();
> >  		pgstat_report_wait_end();
> > +		pgaio_error_cleanup();
>
> AutoVacLauncherMain(), BackgroundWriterMain(), CheckpointerMain(), and
> WalWriterMain() call AtEOXact_Buffers() but not AtEOXact_Aio().  Is that
> proper?  They do call pgaio_error_cleanup() as seen here, so the only loss is
> some asserts.  (The load-bearing part does get done.)

I don't think it's particularly good that we use the AtEOXact_* functions in
the sigsetjmp blocks, that feels like a weird mixup of infrastructure to
me. So this was intentional.


> commit da72269 wrote:
> >     aio: Add core asynchronous I/O infrastructure
>
> > + * This could be in aio_internal.h, as it is not pubicly referenced, but
>
> typo -> publicly

/me has a red face.


> commit 55b454d wrote:
> >     aio: Infrastructure for io_method=worker
>
> > +		/* Try to launch one. */
> > +		child = StartChildProcess(B_IO_WORKER);
> > +		if (child != NULL)
> > +		{
> > +			io_worker_children[id] = child;
> > +			++io_worker_count;
> > +		}
> > +		else
> > +			break;				/* XXX try again soon? */
>
> I'd change the comment to something like one of:
>
>   retry after DetermineSleepTime()
>   next LaunchMissingBackgroundProcesses() will retry in <60s

Hm, we retry more frequently that that if there are new connections...  Maybe
just "try again next time"?


> On Tue, Mar 18, 2025 at 04:12:18PM -0400, Andres Freund wrote:
> > - Decide what to do about the smgr interrupt issue
>
> Replied on that thread.  It's essentially ready.

Cool, will reply there in a bit.


> > Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support
>
> Some comments about BM_IO_IN_PROGRESS may need updates.  This paragraph:
>
> * The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
> buffer to complete (and in releases before 14, it was accompanied by a
> per-buffer LWLock).  The process doing a read or write sets the flag for the
> duration, and processes that need to wait for it to be cleared sleep on a
> condition variable.

First draft:
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
per-buffer LWLock).  The process start a read or write sets the flag. When the
I/O is completed, be it by the process that initiated the I/O or by another
process, the flag is removed and the Buffer's condition variable is signalled.
Processes that need to wait for the I/O to complete can wait for asynchronous
I/O to using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be unset by
sleeping on the buffer's condition variable.


> And these individual lines from "git grep BM_IO_IN_PROGRESS":
>
>  *	i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
>
> The last especially.

Huh - yea.  This isn't a "new" issue, I think I missed this comment in 16's
12f3867f5534.  I think the comment can just be deleted?


> 		 * I/O already in progress.  We already hold BM_IO_IN_PROGRESS for the
> 	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> 	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.

> For the other three lines and the paragraph, the notion
> of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
> being the process "doing a read" becomes less significant when one process
> starts the IO and another completes it.

Hm. I think they'd be ok as-is, but we can probably improve them. Maybe


	 * Now it's safe to write buffer to disk. Note that no one else should
	 * have been able to write it while we were busy with log flushing because
	 * we got the exclusive right to perform I/O by setting the
	 * BM_IO_IN_PROGRESS bit.



> > +		/* we better have ensured the buffer is present until now */
> > +		Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
>
> I'd delete that comment; to me, the assertion alone is clearer.

Ok.


> > +			ereport(LOG,
> > +					(errcode(ERRCODE_DATA_CORRUPTED),
> > +					 errmsg("invalid page in block %u of relation %s; zeroing out page",
>
> This is changing level s/WARNING/LOG/.  That seems orthogonal to the patch's
> goals; is it needed?  If so, I recommend splitting it out as a preliminary
> patch, to highlight the behavior change for release notes.

No, it's not needed. I think I looked over the patch at some point and
considered the log-level wrong according to our guidelines and thought I'd
broken it.


> > +		/*
> > +		 * If the entire failed on a lower-level, each buffer needs to be
>
> Missing word, probably fix like:
> s,entire failed on a lower-level,entire I/O failed on a lower level,


Yep.


> > +		 * marked as failed. In case of a partial read, some buffers may be
> > +		 * ok.
> > +		 */
> > +		failed =
> > +			prior_result.status == ARS_ERROR
> > +			|| prior_result.result <= buf_off;
>
> I didn't run an experiment to check the following, but I think this should be
> s/<=/</.  Suppose we requested two blocks and read some amount of bytes
> [1*BLCKSZ, 2*BLSCKSZ - 1].  md_readv_complete will store result=1.  buf_off==0
> should compute failed=false here, but buf_off==1 should compute failed=true.

Huh, you might be right. I thought I wrote a test for this, I wonder why it
didn't catch the problem...


> I see this relies on md_readv_complete having converted "result" to blocks.
> Was there some win from doing that as opposed to doing the division here?
> Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> to follow, to me.

It seemed like that would be wrong layering - what if we had an smgr that
could store data in a compressed format? The raw read would be of a smaller
size. The smgr API deals in BlockNumbers, only the md.c layer should know
about bytes.


> > +
> > +		buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
> > +											   is_temp);
> > +
> > +		/*
> > +		 * If there wasn't any prior error and the IO for this page failed in
> > +		 * some form, set the whole IO's to the page's result.
>
> s/the IO for this page/page verification/
> s/IO's/IO's result/

Agreed.

Thanks for the review!

Greetings,

Andres Freund

  94. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-20T01:11:18Z

    On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > commit 55b454d wrote:
> > >     aio: Infrastructure for io_method=worker
> >
> > > +		/* Try to launch one. */
> > > +		child = StartChildProcess(B_IO_WORKER);
> > > +		if (child != NULL)
> > > +		{
> > > +			io_worker_children[id] = child;
> > > +			++io_worker_count;
> > > +		}
> > > +		else
> > > +			break;				/* XXX try again soon? */
> >
> > I'd change the comment to something like one of:
> >
> >   retry after DetermineSleepTime()
> >   next LaunchMissingBackgroundProcesses() will retry in <60s
> 
> Hm, we retry more frequently that that if there are new connections...  Maybe
> just "try again next time"?

Works for me.

> > On Tue, Mar 18, 2025 at 04:12:18PM -0400, Andres Freund wrote:
> > > Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support
> >
> > Some comments about BM_IO_IN_PROGRESS may need updates.  This paragraph:
> >
> > * The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
> > buffer to complete (and in releases before 14, it was accompanied by a
> > per-buffer LWLock).  The process doing a read or write sets the flag for the
> > duration, and processes that need to wait for it to be cleared sleep on a
> > condition variable.
> 
> First draft:
> * The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
> buffer to complete (and in releases before 14, it was accompanied by a
> per-buffer LWLock).  The process start a read or write sets the flag. When the
s/start/starting/
> I/O is completed, be it by the process that initiated the I/O or by another
> process, the flag is removed and the Buffer's condition variable is signalled.
> Processes that need to wait for the I/O to complete can wait for asynchronous
> I/O to using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be unset by
s/to using/by using/
> sleeping on the buffer's condition variable.

Sounds good.

> > And these individual lines from "git grep BM_IO_IN_PROGRESS":
> >
> >  *	i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
> >
> > The last especially.
> 
> Huh - yea.  This isn't a "new" issue, I think I missed this comment in 16's
> 12f3867f5534.  I think the comment can just be deleted?

Hmm, yes, it's orthogonal to $SUBJECT and deletion works fine.

> > 		 * I/O already in progress.  We already hold BM_IO_IN_PROGRESS for the
> > 	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> > 	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> 
> > For the other three lines and the paragraph, the notion
> > of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
> > being the process "doing a read" becomes less significant when one process
> > starts the IO and another completes it.
> 
> Hm. I think they'd be ok as-is, but we can probably improve them. Maybe

Looking again, I agree they're okay.

> 
> 	 * Now it's safe to write buffer to disk. Note that no one else should
> 	 * have been able to write it while we were busy with log flushing because
> 	 * we got the exclusive right to perform I/O by setting the
> 	 * BM_IO_IN_PROGRESS bit.

That's fine too.  Maybe s/perform/stage/ or s/perform/start/.

> > I see this relies on md_readv_complete having converted "result" to blocks.
> > Was there some win from doing that as opposed to doing the division here?
> > Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> > to follow, to me.
> 
> It seemed like that would be wrong layering - what if we had an smgr that
> could store data in a compressed format? The raw read would be of a smaller
> size. The smgr API deals in BlockNumbers, only the md.c layer should know
> about bytes.

I hadn't thought of that.  That's a good reason.

  95. Re: AIO v2.5

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-03-20T08:39:32Z

    On Tue, Mar 18, 2025 at 9:12 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> Attached is v2.10, with the following changes:
>
> - committed core AIO infrastructure patch

Hi, yay, It's happening.jpg ;)

Some thoughts about 2.10-0004:
What do you think about putting there into (io_uring patch)  info
about the need to ensure that kernel.io_uring_disabled sysctl is on ?
(some distros might shut it down) E.g. in doc/src/sgml/config.sgml
after io_method = <listitems>... there could be

--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
            <literal>io_uring</literal> (execute asynchronous I/O using
            io_uring, if available)
[..]
and then add something like:
+ "At present io_method=io_uring is supported only on Linux and
requires Linux's sysctl kernel.io_uring_disabled (if present) to be at
value 0 (enabled) or 1 (with kernel.io_uring_group set to PostgreSQL's
GID)."

Rationale: it seems that at least RHEL 9.x will have this knob present
(but e.g. RHEL 8.10 even kernel-ml 6.4.2 doesn't, as this seems to
come with 6.6+ but saw somewhere that somebody had issues with this on
probably backported kernel in Rocky 9.x ). Also further googling for I
have found also that mysql can throw - when executed from
podmap/docker: "mysqld: io_uring_queue_init() failed with ENOSYS:
check seccomp filters, and the kernel version (newer than 5.1
required)"

and this leaves this with two probable follow-up questions when
adjusting this sentence:
a. shouldn't we add some sentence about containers/namespaces/seccomp
allowing this ?
b. and/or shouldn't we reference in docs a minimum kernel version
(this is somewhat wild, liburing could be installed and compiled
against, but runtime kernel would be < 5.1 ?)

-J.

  96. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-20T17:05:05Z

    Hi,

On 2025-03-19 18:17:37 -0400, Andres Freund wrote:
> On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > +		 * marked as failed. In case of a partial read, some buffers may be
> > > +		 * ok.
> > > +		 */
> > > +		failed =
> > > +			prior_result.status == ARS_ERROR
> > > +			|| prior_result.result <= buf_off;
> >
> > I didn't run an experiment to check the following, but I think this should be
> > s/<=/</.  Suppose we requested two blocks and read some amount of bytes
> > [1*BLCKSZ, 2*BLSCKSZ - 1].  md_readv_complete will store result=1.  buf_off==0
> > should compute failed=false here, but buf_off==1 should compute failed=true.
> 
> Huh, you might be right. I thought I wrote a test for this, I wonder why it
> didn't catch the problem...

It was correct as-is. With result=1 you get precisely the result you describe
as the desired outcome, no?
   prior_result.result <= buf_off
   ->
   1 <= 0 -> failed = 0
   1 <= 1 -> failed = 1

but if it were < as you suggest:

   prior_result.result < buf_off
   ->
   1 < 0 -> failed = 0
   1 < 1 -> failed = 0

I.e. we would assume that the second buffer also completed.


What does concern me is that the existing tests do *not* catch the problem if
I turn "<=" into "<".  The second buffer in this case wrongly gets marked as
valid. We do retry the read (because bufmgr.c thinks only one block was read),
but find the buffer to already be valid.

The reason the test doesn't fail, is that the way I set up the "short read"
tests. The injection point runs after the IO completed and just modifies the
result. However, the actual buffer contents still got modified.


The easiest way around that seems to be to have the injection point actually
zero out the remaining memory. Not pretty, but it'd be harder to just submit
shortend IOs in multiple IO methods.  It'd be even better if we could
trivially use something like randomize_mem(), but it's only conditionally
compiled...

Greetings,

Andres Freund

  97. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-20T18:19:04Z

    On Thu, Mar 20, 2025 at 01:05:05PM -0400, Andres Freund wrote:
> On 2025-03-19 18:17:37 -0400, Andres Freund wrote:
> > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > > +		 * marked as failed. In case of a partial read, some buffers may be
> > > > +		 * ok.
> > > > +		 */
> > > > +		failed =
> > > > +			prior_result.status == ARS_ERROR
> > > > +			|| prior_result.result <= buf_off;
> > >
> > > I didn't run an experiment to check the following, but I think this should be
> > > s/<=/</.  Suppose we requested two blocks and read some amount of bytes
> > > [1*BLCKSZ, 2*BLSCKSZ - 1].  md_readv_complete will store result=1.  buf_off==0
> > > should compute failed=false here, but buf_off==1 should compute failed=true.
> > 
> > Huh, you might be right. I thought I wrote a test for this, I wonder why it
> > didn't catch the problem...
> 
> It was correct as-is. With result=1 you get precisely the result you describe
> as the desired outcome, no?
>    prior_result.result <= buf_off
>    ->
>    1 <= 0 -> failed = 0
>    1 <= 1 -> failed = 1
> 
> but if it were < as you suggest:
> 
>    prior_result.result < buf_off
>    ->
>    1 < 0 -> failed = 0
>    1 < 1 -> failed = 0
> 
> I.e. we would assume that the second buffer also completed.

That's right.  I see it now.  My mistake.

> What does concern me is that the existing tests do *not* catch the problem if
> I turn "<=" into "<".  The second buffer in this case wrongly gets marked as
> valid. We do retry the read (because bufmgr.c thinks only one block was read),
> but find the buffer to already be valid.
> 
> The reason the test doesn't fail, is that the way I set up the "short read"
> tests. The injection point runs after the IO completed and just modifies the
> result. However, the actual buffer contents still got modified.
> 
> 
> The easiest way around that seems to be to have the injection point actually
> zero out the remaining memory.

Sounds reasonable and sufficient.

FYI, I've resumed the comprehensive review.  That's still ongoing.

  98. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-20T18:54:14Z

    Hi,

On 2025-03-19 18:11:18 -0700, Noah Misch wrote:
> On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:

> > Hm, we retry more frequently that that if there are new connections...  Maybe
> > just "try again next time"?
> 
> Works for me.
> 

> > > And these individual lines from "git grep BM_IO_IN_PROGRESS":
> > >
> > >  *	i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
> > >
> > > The last especially.
> > 
> > Huh - yea.  This isn't a "new" issue, I think I missed this comment in 16's
> > 12f3867f5534.  I think the comment can just be deleted?
> 
> Hmm, yes, it's orthogonal to $SUBJECT and deletion works fine.
> 
> > > 		 * I/O already in progress.  We already hold BM_IO_IN_PROGRESS for the
> > > 	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> > > 	 * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> > 
> > > For the other three lines and the paragraph, the notion
> > > of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
> > > being the process "doing a read" becomes less significant when one process
> > > starts the IO and another completes it.
> > 
> > Hm. I think they'd be ok as-is, but we can probably improve them. Maybe
> 
> Looking again, I agree they're okay.
> 
> > 
> > 	 * Now it's safe to write buffer to disk. Note that no one else should
> > 	 * have been able to write it while we were busy with log flushing because
> > 	 * we got the exclusive right to perform I/O by setting the
> > 	 * BM_IO_IN_PROGRESS bit.
> 
> That's fine too.  Maybe s/perform/stage/ or s/perform/start/.

I put these comment changes into their own patch, as it seemed confusing to
change them as part of one of the already queued commits.


> > > I see this relies on md_readv_complete having converted "result" to blocks.
> > > Was there some win from doing that as opposed to doing the division here?
> > > Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> > > to follow, to me.
> > 
> > It seemed like that would be wrong layering - what if we had an smgr that
> > could store data in a compressed format? The raw read would be of a smaller
> > size. The smgr API deals in BlockNumbers, only the md.c layer should know
> > about bytes.
> 
> I hadn't thought of that.  That's a good reason.

I thought that was better documented, but alas, it wasn't. How about updating
the documentation of smgrstartreadv to the following:

/*
 * smgrstartreadv() -- asynchronous version of smgrreadv()
 *
 * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
 * `ioh` all parameters are the same as smgrreadv().
 *
 * Completion callbacks above smgr will be passed the result as the number of
 * successfully read blocks if the read [partially] succeeds. This maintains
 * the abstraction that smgr operates on the level of blocks, rather than
 * bytes.
 */


I briefly had a bug in test_aio's injection point that lead to *increasing*
the number of bytes successfully read. That triggered an assertion failure in
bufmgr.c, but not closer to the problem.  Is it worth adding an assert against
that to md_readv_complete? Can't quite decide.

Greetings,

Andres Freund

  99. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-20T19:52:45Z

    On Thu, Mar 20, 2025 at 02:54:14PM -0400, Andres Freund wrote:
> On 2025-03-19 18:11:18 -0700, Noah Misch wrote:
> > On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> > > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > > I see this relies on md_readv_complete having converted "result" to blocks.
> > > > Was there some win from doing that as opposed to doing the division here?
> > > > Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> > > > to follow, to me.
> > > 
> > > It seemed like that would be wrong layering - what if we had an smgr that
> > > could store data in a compressed format? The raw read would be of a smaller
> > > size. The smgr API deals in BlockNumbers, only the md.c layer should know
> > > about bytes.
> > 
> > I hadn't thought of that.  That's a good reason.
> 
> I thought that was better documented, but alas, it wasn't. How about updating
> the documentation of smgrstartreadv to the following:
> 
> /*
>  * smgrstartreadv() -- asynchronous version of smgrreadv()
>  *
>  * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
>  * `ioh` all parameters are the same as smgrreadv().
>  *
>  * Completion callbacks above smgr will be passed the result as the number of
>  * successfully read blocks if the read [partially] succeeds. This maintains
>  * the abstraction that smgr operates on the level of blocks, rather than
>  * bytes.
>  */

That's good.  Possibly add "(Buffers for blocks not successfully read might
bear unspecified modifications, up to the full nblocks.)"

In a bit of over-thinking this, I wondered if shared_buffer_readv_complete
would be better named shared_buffer_smgrreadv_complete, to emphasize the
smgrreadv semantics.  PGAIO_HCB_SHARED_BUFFER_READV likewise.  But I tend to
think not.  smgrreadv() has no "result" concept, so the symmetry is limited.

> I briefly had a bug in test_aio's injection point that lead to *increasing*
> the number of bytes successfully read. That triggered an assertion failure in
> bufmgr.c, but not closer to the problem.  Is it worth adding an assert against
> that to md_readv_complete? Can't quite decide.

I'd lean yes, if in doubt.

  100. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-21T01:58:37Z

    Hi,

Attached v2.11, with the following changes:


- Pushed the smgr interrupt change, as discussed on the dedicated thread


- Pushed "bufmgr: Improve stats when a buffer is read in concurrently"

  It was reviewed by Melanie and there didn't seem to be any reason to wait
  further.


- Addressed feedback from Melanie


- Addressed feedback from Noah


- Added a new commit: aio: Change prefix of PgAioResultStatus values to PGAIO_RS_

  As suggested/requested by Melanie. I think she's unfortunately right.


- Added a patch for some comment fixups for code that's either older or
  already pushed


- Added an error check for FileStartReadV() failing

  FileStartReadV() actually can fail, if the file can't be re-opened. I
  thought it'd be important for the error message to differ from the one
  that's issued for read actually failing, so I went with:

  "could not start reading blocks %u..%u in file \"%s\": %m"

  but I'm not sure how good that is.


- Added a new commit to redefine set_max_safe_fds() to not subtract
  already_open fds from max_files_per_process

  This prevents io_method=io_uring from failing when RLIMIT_NOFILE is high
  enough, but more than max_files_per_process io_uring instances need to be
  created.


- Improved error message if io_uring_queue_init() fails

  Added errhint()s for likely cases of failure.

  Added errcode().  I was tempted to use errcode_for_file_access(), but that
  doesn't support ENOSYS - perhaps I should add that instead?


- Disable io_uring method when using EXEC_BACKEND, they're not compatible

  I chose to do this with a define aio.h, but I guess we could also do it at
  configure time? That seems more complicated though - how would we even know
  that EXEC_BACKEND is used on non-windows?

  Not sure yet how to best disable testing io_uring in this case. We can't
  just query EXEC_BACKEND from pg_config.h unfortunately.  I guess making the
  initdb not fail and checking the error log would work, but that doesn't work
  nicely with Cluster.pm.


- Changed test_aio injection short-read injection point to zero out the rest
  of the IO, otherwise some tests fail to fail even if a bug in retries of
  partial reads is introduced


- Improved method_io_uring.c includes a bit (no pgstat.h)



Questions:


- We only "look" at BM_IO_ERROR for writes, isn't that somewhat weird?

  See AbortBufferIO(Buffer buffer)

  It doesn't really matter for the patchset, but it just strikes me as an oddity.



Greetings,

Andres Freund

  101. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-23T00:20:56Z

    On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11, with the following changes:

> - Added an error check for FileStartReadV() failing
> 
>   FileStartReadV() actually can fail, if the file can't be re-opened. I
>   thought it'd be important for the error message to differ from the one
>   that's issued for read actually failing, so I went with:
> 
>   "could not start reading blocks %u..%u in file \"%s\": %m"
> 
>   but I'm not sure how good that is.

Message looks good.

> - Improved error message if io_uring_queue_init() fails
> 
>   Added errhint()s for likely cases of failure.
> 
>   Added errcode().  I was tempted to use errcode_for_file_access(), but that
>   doesn't support ENOSYS - perhaps I should add that instead?

Either way is fine with me.  ENOSYS -> ERRCODE_FEATURE_NOT_SUPPORTED is a good
general mapping to have in errcode_for_file_access(), but it's also not a
problem to keep it the way v2.11 has it.

> - Disable io_uring method when using EXEC_BACKEND, they're not compatible
> 
>   I chose to do this with a define aio.h, but I guess we could also do it at
>   configure time? That seems more complicated though - how would we even know
>   that EXEC_BACKEND is used on non-windows?

Agreed, "make PROFILE=-DEXEC_BACKEND" is a valid way to get EXEC_BACKEND.

>   Not sure yet how to best disable testing io_uring in this case. We can't
>   just query EXEC_BACKEND from pg_config.h unfortunately.  I guess making the
>   initdb not fail and checking the error log would work, but that doesn't work
>   nicely with Cluster.pm.

How about "postgres -c io_method=io_uring -C <anything>":

--- a/src/test/modules/test_aio/t/001_aio.pl
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -29,7 +29,13 @@ $node_worker->stop();
 # Test io_method=io_uring
 ###
 
-if ($ENV{with_liburing} eq 'yes')
+sub have_io_uring
+{
+	local %ENV = $node_worker->_get_env();  # any node works
+	return run_log [qw(postgres -c io_method=io_uring -C io_method)];
+}
+
+if (have_io_uring())
 {
 	my $node_uring = create_node('io_uring');
 	$node_uring->start();

> Questions:
> 
> 
> - We only "look" at BM_IO_ERROR for writes, isn't that somewhat weird?
> 
>   See AbortBufferIO(Buffer buffer)
> 
>   It doesn't really matter for the patchset, but it just strikes me as an oddity.

That caught my attention in an earlier review round, but I didn't find it
important enough to raise.  It's mildly unfortunate to be setting BM_IO_ERROR
for reads when the only thing BM_IO_ERROR drives is message "Multiple failures
--- write error might be permanent."  It's minor, so let's leave it that way
for the foreseeable future.

> Subject: [PATCH v2.11 01/27] aio, bufmgr: Comment fixes

Ready to commit, though other comment fixes might come up in later reviews.
One idea so far is to comment on valid states after some IoMethodOps
callbacks:

--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -310,6 +310,9 @@ typedef struct IoMethodOps
 	/*
 	 * Start executing passed in IOs.
 	 *
+	 * Shall advance state to PGAIO_HS_SUBMITTED.  (By the time this returns,
+	 * other backends might have advanced the state further.)
+	 *
 	 * Will not be called if ->needs_synchronous_execution() returned true.
 	 *
 	 * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -321,6 +324,12 @@ typedef struct IoMethodOps
 	/*
 	 * Wait for the IO to complete. Optional.
 	 *
+	 * On return, state shall be PGAIO_HS_COMPLETED_IO,
+	 * PGAIO_HS_COMPLETED_SHARED or PGAIO_HS_COMPLETED_LOCAL.  (The callback
+	 * need not change the state if it's already one of those.)  If state is
+	 * PGAIO_HS_COMPLETED_IO, state will reach PGAIO_HS_COMPLETED_SHARED
+	 * without further intervention.
+	 *
 	 * If not provided, it needs to be guaranteed that the IO method calls
 	 * pgaio_io_process_completion() without further interaction by the
 	 * issuing backend.

> Subject: [PATCH v2.11 02/27] aio: Change prefix of PgAioResultStatus values to
>  PGAIO_RS_

Ready to commit

> Subject: [PATCH v2.11 03/27] Redefine max_files_per_process to control
>  additionally opened files

Ready to commit

> Subject: [PATCH v2.11 04/27] aio: Add liburing dependency

> --- a/meson.build
> +++ b/meson.build
> @@ -944,6 +944,18 @@ endif
>  
>  
>  
> +###############################################################
> +# Library: liburing
> +###############################################################
> +
> +liburingopt = get_option('liburing')
> +liburing = dependency('liburing', required: liburingopt)
> +if liburing.found()
> +  cdata.set('USE_LIBURING', 1)
> +endif

This is a different style from other deps; is it equivalent to our standard
style?  Example for lz4:

lz4opt = get_option('lz4')
if not lz4opt.disabled()
  lz4 = dependency('liblz4', required: false)
  # Unfortunately the dependency is named differently with cmake
  if not lz4.found() # combine with above once meson 0.60.0 is required
    lz4 = dependency('lz4', required: lz4opt,
                     method: 'cmake', modules: ['LZ4::lz4_shared'],
                    )
  endif

  if lz4.found()
    cdata.set('USE_LZ4', 1)
    cdata.set('HAVE_LIBLZ4', 1)
  endif

else
  lz4 = not_found_dep
endif

> --- a/configure.ac
> +++ b/configure.ac
> @@ -975,6 +975,14 @@ AC_SUBST(with_readline)
>  PGAC_ARG_BOOL(with, libedit-preferred, no,
>                [prefer BSD Libedit over GNU Readline])
>  
> +#
> +# liburing
> +#
> +AC_MSG_CHECKING([whether to build with liburing support])
> +PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],

Fourth arg generally starts with "build" for args like this.  I suggest "build
with io_uring support, for asynchronous I/O".  Comparable options:

  --with-llvm             build with LLVM based JIT support
  --with-tcl              build Tcl modules (PL/Tcl)
  --with-perl             build Perl modules (PL/Perl)
  --with-python           build Python modules (PL/Python)
  --with-gssapi           build with GSSAPI support
  --with-pam              build with PAM support
  --with-bsd-auth         build with BSD Authentication support
  --with-ldap             build with LDAP support
  --with-bonjour          build with Bonjour support
  --with-selinux          build with SELinux support
  --with-systemd          build with systemd support
  --with-libcurl          build with libcurl support
  --with-libxml           build with XML support
  --with-libxslt          use XSLT support when building contrib/xml2
  --with-lz4              build with LZ4 support
  --with-zstd             build with ZSTD support

> +              [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
> +AC_MSG_RESULT([$with_liburing])
> +AC_SUBST(with_liburing)
>  
>  #
>  # UUID library
> @@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
>  fi
>  AC_SUBST(UUID_LIBS)
>  
> +if test "$with_liburing" = yes; then
> +  PKG_CHECK_MODULES(LIBURING, liburing)
> +fi

We usually put this right after the AC_MSG_CHECKING ... AC_SUBST block.  This
currently has unrelated stuff separating them.  Also, with the exception of
icu, we follow PKG_CHECK_MODULES uses by absorbing flags from pkg-config and
use AC_CHECK_LIB to add the actual "-l".  By not absorbing flags, I think a
liburing in a nonstandard location would require --with-libraries and
--with-includes, unlike the other PKG_CHECK_MODULES-based dependencies.  lz4
is a representative example of our standard:

```
AC_MSG_CHECKING([whether to build with LZ4 support])
PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],
              [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])
AC_MSG_RESULT([$with_lz4])
AC_SUBST(with_lz4)

if test "$with_lz4" = yes; then
  PKG_CHECK_MODULES(LZ4, liblz4)
  # We only care about -I, -D, and -L switches;
  # note that -llz4 will be added by AC_CHECK_LIB below.
  for pgac_option in $LZ4_CFLAGS; do
    case $pgac_option in
      -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
    esac
  done
  for pgac_option in $LZ4_LIBS; do
    case $pgac_option in
      -L*) LDFLAGS="$LDFLAGS $pgac_option";;
    esac
  done
fi

# ... later in file ...

if test "$with_lz4" = yes ; then
  AC_CHECK_LIB(lz4, LZ4_compress_default, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
fi
```

I think it's okay to not use the AC_CHECK_LIB and rely on explicit
src/backend/Makefile code like you've done, but we shouldn't miss
CPPFLAGS/LDFLAGS (or should have a comment on why missing them is right).

> --- a/doc/src/sgml/installation.sgml
> +++ b/doc/src/sgml/installation.sgml

lz4 and other deps have a mention in <sect1 id="install-requirements">, in
addition to sections edited here.

> Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring

(Still reviewing this one.)

  102. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-23T02:09:55Z

    On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11

> Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring

Apart from some isolated cosmetic points, this is ready to commit:

> +			ereport(ERROR,
> +					errcode(err),
> +					errmsg("io_uring_queue_init failed: %m"),
> +					hint != NULL ? errhint("%s", hint) : 0);

https://www.postgresql.org/docs/current/error-style-guide.html gives the example:

BAD:    open() failed: %m
BETTER: could not open file %s: %m

Hence, this errmsg should change, perhaps to:
"could not setup io_uring queues: %m".

> +		pgaio_debug_io(DEBUG3, ioh,
> +					   "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
> +					   (long long unsigned) ref_generation,
> +					   (long long unsigned) ioh->generation,

In the message string, io_gen appears before ref_gen.  In the subsequent args,
the order is swapped relative to the message string.

> --- a/src/backend/utils/activity/wait_event_names.txt
> +++ b/src/backend/utils/activity/wait_event_names.txt
> @@ -192,6 +192,8 @@ ABI_compatibility:
>  
>  Section: ClassName - WaitEventIO
>  
> +AIO_IO_URING_SUBMIT	"Waiting for IO submission via io_uring."
> +AIO_IO_URING_COMPLETION	"Waiting for IO completion via io_uring."
>  AIO_IO_COMPLETION	"Waiting for IO completion."

I'm wondering if there's an opportunity to enrich the last two wait event
names and/or descriptions.  The current descriptions suggest to me more
similarity than is actually there.  Inputs to the decision:

- AIO_IO_COMPLETION waits for an IO in PGAIO_HS_DEFINED, PGAIO_HS_STAGED, or
  PGAIO_HS_COMPLETED_IO to reach PGAIO_HS_COMPLETED_SHARED.  The three
  starting states are the states where some other backend owns the next
  action, so the current backend can only wait to be signaled.

- AIO_IO_URING_COMPLETION waits for the kernel to do enough so we can move
  from PGAIO_HS_SUBMITTED to PGAIO_HS_COMPLETED_IO.

Possible names and descriptions, based on PgAioHandleState enum names and
comments:

AIO_IO_URING_COMPLETED_IO	"Waiting for IO result via io_uring."
AIO_COMPLETED_SHARED	"Waiting for IO shared completion callback."

If "shared completion callback" is too internals-focused, perhaps this:

AIO_IO_URING_COMPLETED_IO	"Waiting for IO result via io_uring."
AIO_COMPLETED_SHARED	"Waiting for IO completion to update shared memory."

> --- a/doc/src/sgml/config.sgml
> +++ b/doc/src/sgml/config.sgml
> @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
>              <literal>worker</literal> (execute asynchronous I/O using worker processes)
>             </para>
>            </listitem>
> +          <listitem>
> +           <para>
> +            <literal>io_uring</literal> (execute asynchronous I/O using
> +            io_uring, if available)

I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
there to reduce surprise on other platforms.

> Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd

(Still reviewing this one.)

  103. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-23T15:11:53Z

    Hi,

On 2025-03-22 17:20:56 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> >   Not sure yet how to best disable testing io_uring in this case. We can't
> >   just query EXEC_BACKEND from pg_config.h unfortunately.  I guess making the
> >   initdb not fail and checking the error log would work, but that doesn't work
> >   nicely with Cluster.pm.
> 
> How about "postgres -c io_method=io_uring -C <anything>":
> 
> --- a/src/test/modules/test_aio/t/001_aio.pl
> +++ b/src/test/modules/test_aio/t/001_aio.pl
> @@ -29,7 +29,13 @@ $node_worker->stop();
>  # Test io_method=io_uring
>  ###
>  
> -if ($ENV{with_liburing} eq 'yes')
> +sub have_io_uring
> +{
> +	local %ENV = $node_worker->_get_env();  # any node works
> +	return run_log [qw(postgres -c io_method=io_uring -C io_method)];
> +}
> +
> +if (have_io_uring())
>  {
>  	my $node_uring = create_node('io_uring');
>  	$node_uring->start();

Yea, that's a good idea.

One thing that doesn't seem great is that it requires a prior node - what if
we do -c io_method=invalid' that would report the list of valid GUC options,
so we could just grep for io_uring?

It's too bad that postgres --describe-config
a) doesn't report the possible enum values
b) doesn't apply/validate -c options


> > Subject: [PATCH v2.11 01/27] aio, bufmgr: Comment fixes
> 
> Ready to commit, though other comment fixes might come up in later reviews.

I'll reorder it to a bit later in the series, to accumulate a few more.


> One idea so far is to comment on valid states after some IoMethodOps
> callbacks:
> 
> --- a/src/include/storage/aio_internal.h
> +++ b/src/include/storage/aio_internal.h
> @@ -310,6 +310,9 @@ typedef struct IoMethodOps
>  	/*
>  	 * Start executing passed in IOs.
>  	 *
> +	 * Shall advance state to PGAIO_HS_SUBMITTED.  (By the time this returns,
> +	 * other backends might have advanced the state further.)
> +	 *
>  	 * Will not be called if ->needs_synchronous_execution() returned true.
>  	 *
>  	 * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
> @@ -321,6 +324,12 @@ typedef struct IoMethodOps
>  	/*
>  	 * Wait for the IO to complete. Optional.
>  	 *
> +	 * On return, state shall be PGAIO_HS_COMPLETED_IO,
> +	 * PGAIO_HS_COMPLETED_SHARED or PGAIO_HS_COMPLETED_LOCAL.  (The callback
> +	 * need not change the state if it's already one of those.)  If state is
> +	 * PGAIO_HS_COMPLETED_IO, state will reach PGAIO_HS_COMPLETED_SHARED
> +	 * without further intervention.
> +	 *
>  	 * If not provided, it needs to be guaranteed that the IO method calls
>  	 * pgaio_io_process_completion() without further interaction by the
>  	 * issuing backend.

I think these are a good idea. I added those to the copy-edit patch, with a
few more tweaks:

@@ -315,6 +315,9 @@ typedef struct IoMethodOps
     /*
      * Start executing passed in IOs.
      *
+     * Shall advance state to at least PGAIO_HS_SUBMITTED.  (By the time this
+     * returns, other backends might have advanced the state further.)
+     *
      * Will not be called if ->needs_synchronous_execution() returned true.
      *
      * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -323,12 +326,24 @@ typedef struct IoMethodOps
      */
     int         (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
 
-    /*
+    /* ---
      * Wait for the IO to complete. Optional.
      *
+     * On return, state shall be on of
+     * - PGAIO_HS_COMPLETED_IO
+     * - PGAIO_HS_COMPLETED_SHARED
+     * - PGAIO_HS_COMPLETED_LOCAL
+     *
+     * The callback must not block if the handle is already in one of those
+     * states, or has been reused (see pgaio_io_was_recycled()).  If, on
+     * return, the state is PGAIO_HS_COMPLETED_IO, state will reach
+     * PGAIO_HS_COMPLETED_SHARED without further intervention by the IO
+     * method.
+     *
      * If not provided, it needs to be guaranteed that the IO method calls
      * pgaio_io_process_completion() without further interaction by the
      * issuing backend.
+     * ---
      */
     void        (*wait_one) (PgAioHandle *ioh,
                              uint64 ref_generation);



> > Subject: [PATCH v2.11 03/27] Redefine max_files_per_process to control
> >  additionally opened files
> 
> Ready to commit

Cool!


> > Subject: [PATCH v2.11 04/27] aio: Add liburing dependency
> 
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -944,6 +944,18 @@ endif
> >  
> >  
> >  
> > +###############################################################
> > +# Library: liburing
> > +###############################################################
> > +
> > +liburingopt = get_option('liburing')
> > +liburing = dependency('liburing', required: liburingopt)
> > +if liburing.found()
> > +  cdata.set('USE_LIBURING', 1)
> > +endif
> 
> This is a different style from other deps; is it equivalent to our standard
> style?

Yes - the only reason to be more complicated in the lz4 case is that we want
to fall back to other ways of looking up the dependency (primarily because of
windows. But that's not required for liburing, which oviously is linux only.


> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -975,6 +975,14 @@ AC_SUBST(with_readline)
> >  PGAC_ARG_BOOL(with, libedit-preferred, no,
> >                [prefer BSD Libedit over GNU Readline])
> >  
> > +#
> > +# liburing
> > +#
> > +AC_MSG_CHECKING([whether to build with liburing support])
> > +PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
> 
> Fourth arg generally starts with "build" for args like this.  I suggest "build
> with io_uring support, for asynchronous I/O".

WFM.


> > +              [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
> > +AC_MSG_RESULT([$with_liburing])
> > +AC_SUBST(with_liburing)
> >  
> >  #
> >  # UUID library
> > @@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
> >  fi
> >  AC_SUBST(UUID_LIBS)
> >  
> > +if test "$with_liburing" = yes; then
> > +  PKG_CHECK_MODULES(LIBURING, liburing)
> > +fi
> 
> We usually put this right after the AC_MSG_CHECKING ... AC_SUBST block.

We don't really seem to do that for "dependency checks" in general, e.g.
PGAC_CHECK_PERL_CONFIGS, PGAC_CHECK_PYTHON_EMBED_SETUP, PGAC_CHECK_READLINE,
dependency dependent AC_CHECK_LIB calls, .. later in configure.ac than the
defnition of the option.  TBH, I've always struggled trying to discern what
the organizing principle of configure.ac is.

But you're right that the PKG_CHECK_MODULES calls are closer-by. And I'm happy
to move towards having the code for each dep all in one place, so moved.


A related thing: We seem to have no order of the $with_ checks that I can
discern. Should the liburing check be at a different place?


> This currently has unrelated stuff separating them.  Also, with the
> exception of icu, we follow PKG_CHECK_MODULES uses by absorbing flags from
> pkg-config and use AC_CHECK_LIB to add the actual "-l".

I think for liburing I was trying to follow ICU's example - injecting CFLAGS
and LIBS just in the parts of the build dir that needs them.

For LIBS I think I did so:

diff --git a/src/backend/Makefile b/src/backend/Makefile
...
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)

But ugh, for some reason I didn't do that for LIBURING_CFLAGS. In the v1.x
version of aio I had
aio:src/backend/storage/aio/Makefile:override CPPFLAGS += $(LIBURING_CFLAGS)

but somehow lost that somewhere along the way to v2.x


I think I like targetting where ${LIB}_LIBS and ${LIB}_CFLAGS are applied more
narrowly better than just adding to the global CFLAGS, CPPFLAGS, LDFLAGS.  I'm
somewhat inclined to add it LIBURING_CFLAGS in src/backend rather than
src/backend/storage/aio/ though.

But I'm also willing to do it entirely differently.


> > --- a/doc/src/sgml/installation.sgml
> > +++ b/doc/src/sgml/installation.sgml
> 
> lz4 and other deps have a mention in <sect1 id="install-requirements">, in
> addition to sections edited here.

Good point.

Although once more I feel defeated by the ordering used :)

Hm, that list is rather incomplete. At least libxml, libxslt, selinux, curl,
uuid, systemd, selinux and bonjour aren't listed.

Not sure if it makes sense to add liburing, given that?

Greetings,

Andres Freund

  104. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-23T15:55:29Z

    On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote:
> On 2025-03-22 17:20:56 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > >   Not sure yet how to best disable testing io_uring in this case. We can't
> > >   just query EXEC_BACKEND from pg_config.h unfortunately.  I guess making the
> > >   initdb not fail and checking the error log would work, but that doesn't work
> > >   nicely with Cluster.pm.
> > 
> > How about "postgres -c io_method=io_uring -C <anything>":
> > 
> > --- a/src/test/modules/test_aio/t/001_aio.pl
> > +++ b/src/test/modules/test_aio/t/001_aio.pl
> > @@ -29,7 +29,13 @@ $node_worker->stop();
> >  # Test io_method=io_uring
> >  ###
> >  
> > -if ($ENV{with_liburing} eq 'yes')
> > +sub have_io_uring
> > +{
> > +	local %ENV = $node_worker->_get_env();  # any node works
> > +	return run_log [qw(postgres -c io_method=io_uring -C io_method)];
> > +}
> > +
> > +if (have_io_uring())
> >  {
> >  	my $node_uring = create_node('io_uring');
> >  	$node_uring->start();
> 
> Yea, that's a good idea.
> 
> One thing that doesn't seem great is that it requires a prior node - what if
> we do -c io_method=invalid' that would report the list of valid GUC options,
> so we could just grep for io_uring?

Works for me.

> > One idea so far is to comment on valid states after some IoMethodOps
> > callbacks:

> I think these are a good idea. I added those to the copy-edit patch, with a
> few more tweaks:

The tweaks made it better.

> > > Subject: [PATCH v2.11 04/27] aio: Add liburing dependency

> > > +              [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
> > > +AC_MSG_RESULT([$with_liburing])
> > > +AC_SUBST(with_liburing)
> > >  
> > >  #
> > >  # UUID library
> > > @@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
> > >  fi
> > >  AC_SUBST(UUID_LIBS)
> > >  
> > > +if test "$with_liburing" = yes; then
> > > +  PKG_CHECK_MODULES(LIBURING, liburing)
> > > +fi
> > 
> > We usually put this right after the AC_MSG_CHECKING ... AC_SUBST block.
> 
> We don't really seem to do that for "dependency checks" in general, e.g.
> PGAC_CHECK_PERL_CONFIGS, PGAC_CHECK_PYTHON_EMBED_SETUP, PGAC_CHECK_READLINE,
> dependency dependent AC_CHECK_LIB calls, .. later in configure.ac than the
> defnition of the option.

AC_CHECK_LIB stays far away, yes.

> But you're right that the PKG_CHECK_MODULES calls are closer-by. And I'm happy
> to move towards having the code for each dep all in one place, so moved.
> 
> 
> A related thing: We seem to have no order of the $with_ checks that I can
> discern. Should the liburing check be at a different place?

No opinion on that one.  It's fine.

> > This currently has unrelated stuff separating them.  Also, with the
> > exception of icu, we follow PKG_CHECK_MODULES uses by absorbing flags from
> > pkg-config and use AC_CHECK_LIB to add the actual "-l".
> 
> I think for liburing I was trying to follow ICU's example - injecting CFLAGS
> and LIBS just in the parts of the build dir that needs them.
> 
> For LIBS I think I did so:
> 
> diff --git a/src/backend/Makefile b/src/backend/Makefile
> ...
> +# The backend conditionally needs libraries that most executables don't need.
> +LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
> 
> But ugh, for some reason I didn't do that for LIBURING_CFLAGS. In the v1.x
> version of aio I had
> aio:src/backend/storage/aio/Makefile:override CPPFLAGS += $(LIBURING_CFLAGS)
> 
> but somehow lost that somewhere along the way to v2.x
> 
> 
> I think I like targetting where ${LIB}_LIBS and ${LIB}_CFLAGS are applied more
> narrowly better than just adding to the global CFLAGS, CPPFLAGS, LDFLAGS.

Agreed.

> somewhat inclined to add it LIBURING_CFLAGS in src/backend rather than
> src/backend/storage/aio/ though.
> 
> But I'm also willing to do it entirely differently.

The CPPFLAGS addition, located wherever makes sense, resolves that point.

> > > --- a/doc/src/sgml/installation.sgml
> > > +++ b/doc/src/sgml/installation.sgml
> > 
> > lz4 and other deps have a mention in <sect1 id="install-requirements">, in
> > addition to sections edited here.
> 
> Good point.
> 
> Although once more I feel defeated by the ordering used :)
> 
> Hm, that list is rather incomplete. At least libxml, libxslt, selinux, curl,
> uuid, systemd, selinux and bonjour aren't listed.
> 
> Not sure if it makes sense to add liburing, given that?

That's a lot of preexisting incompleteness.  I withdraw the point about <sect1
id="install-requirements">.


Unrelated to the above, another question about io_uring:

commit da722699 wrote:
> +/*
> + * Need to submit staged but not yet submitted IOs using the fd, otherwise
> + * the IO would end up targeting something bogus.
> + */
> +void
> +pgaio_closing_fd(int fd)

An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
PGAIO_HS_COMPLETED_IO clearly doesn't block that close.  For io_method=worker,
closing in PGAIO_HS_SUBMITTED is okay.  For io_method=io_uring, is there a
reference about it being okay to close during PGAIO_HS_SUBMITTED?  I looked
awhile for an authoritative view on that, but I didn't find one.  If we can
rely on io_uring_submit() returning only after the kernel has given the
io_uring its own reference to all applicable file descriptors, I expect it's
okay to close the process's FD.  If the io_uring acquires its reference later
than that, I expect we shouldn't close before that later time.

  105. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-23T15:57:48Z

    Hi,

On 2025-03-22 19:09:55 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Attached v2.11
>
> > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
>
> Apart from some isolated cosmetic points, this is ready to commit:
>
> > +			ereport(ERROR,
> > +					errcode(err),
> > +					errmsg("io_uring_queue_init failed: %m"),
> > +					hint != NULL ? errhint("%s", hint) : 0);
>
> https://www.postgresql.org/docs/current/error-style-guide.html gives the example:
>
> BAD:    open() failed: %m
> BETTER: could not open file %s: %m
>
> Hence, this errmsg should change, perhaps to:
> "could not setup io_uring queues: %m".

You're right. I didn't intentionally "violate" the policy, but I do have to
admit, I'm not a huge fan of that aspect, it just obfuscates what actually
failed, forcing one to look at the code or strace to figure out what precisely
failed.

(Changed)


> > +		pgaio_debug_io(DEBUG3, ioh,
> > +					   "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
> > +					   (long long unsigned) ref_generation,
> > +					   (long long unsigned) ioh->generation,
>
> In the message string, io_gen appears before ref_gen.  In the subsequent args,
> the order is swapped relative to the message string.

Oops, you're right.


> > --- a/src/backend/utils/activity/wait_event_names.txt
> > +++ b/src/backend/utils/activity/wait_event_names.txt
> > @@ -192,6 +192,8 @@ ABI_compatibility:
> >
> >  Section: ClassName - WaitEventIO
> >
> > +AIO_IO_URING_SUBMIT	"Waiting for IO submission via io_uring."
> > +AIO_IO_URING_COMPLETION	"Waiting for IO completion via io_uring."
> >  AIO_IO_COMPLETION	"Waiting for IO completion."
>
> I'm wondering if there's an opportunity to enrich the last two wait event
> names and/or descriptions.  The current descriptions suggest to me more
> similarity than is actually there.  Inputs to the decision:
>
> - AIO_IO_COMPLETION waits for an IO in PGAIO_HS_DEFINED, PGAIO_HS_STAGED, or
>   PGAIO_HS_COMPLETED_IO to reach PGAIO_HS_COMPLETED_SHARED.  The three
>   starting states are the states where some other backend owns the next
>   action, so the current backend can only wait to be signaled.
>
> - AIO_IO_URING_COMPLETION waits for the kernel to do enough so we can move
>   from PGAIO_HS_SUBMITTED to PGAIO_HS_COMPLETED_IO.
>
> Possible names and descriptions, based on PgAioHandleState enum names and
> comments:
>
> AIO_IO_URING_COMPLETED_IO	"Waiting for IO result via io_uring."
> AIO_COMPLETED_SHARED	"Waiting for IO shared completion callback."
>
> If "shared completion callback" is too internals-focused, perhaps this:
>
> AIO_IO_URING_COMPLETED_IO	"Waiting for IO result via io_uring."
> AIO_COMPLETED_SHARED	"Waiting for IO completion to update shared memory."

Hm, right now AIO_IO_COMPLETION also covers the actual "raw" execution of the
IO with io_method=worker/sync. For that AIO_COMPLETED_SHARED would be
inappropriate.

We could use a different wait event if wait for an IO via CV in
PGAIO_HS_SUBMITTED, with a small refactoring of pgaio_io_wait().  But I'm not
sure that would get you that far - we don't broadcast the CV when
transitioning from PGAIO_HS_SUBMITTED -> PGAIO_HS_COMPLETED_IO, so the wait
event would stay the same, now wrong, wait event until the shared callback
completes. Obviously waking everyone up just so they can use a differen wait
event doesn't make sense.

A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
"execution" or something like that, to hint at a separation between the raw IO
being completed and the IO, including the callbacks completing.


> > --- a/doc/src/sgml/config.sgml
> > +++ b/doc/src/sgml/config.sgml
> > @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> >              <literal>worker</literal> (execute asynchronous I/O using worker processes)
> >             </para>
> >            </listitem>
> > +          <listitem>
> > +           <para>
> > +            <literal>io_uring</literal> (execute asynchronous I/O using
> > +            io_uring, if available)
>
> I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
> Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
> there to reduce surprise on other platforms.

You're right, the if available can be misunderstood. But not mentioning that
it's an optional dependency seems odd too. What about something like

           <para>
            <literal>io_uring</literal> (execute asynchronous I/O using
            io_uring, requires postgres to have been built with
            <link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
            <link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
           </para>

Should the docs for --with-liburing/-Dliburing mention it's linux only? We
don't seem to do that for things like systemd (linux), selinux (linux) and
only kinda for bonjour (macos).

Greetings,

Andres Freund

  106. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-23T16:32:48Z

    On Sun, Mar 23, 2025 at 11:57:48AM -0400, Andres Freund wrote:
> On 2025-03-22 19:09:55 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Attached v2.11
> >
> > > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring

> > > --- a/src/backend/utils/activity/wait_event_names.txt
> > > +++ b/src/backend/utils/activity/wait_event_names.txt
> > > @@ -192,6 +192,8 @@ ABI_compatibility:
> > >
> > >  Section: ClassName - WaitEventIO
> > >
> > > +AIO_IO_URING_SUBMIT	"Waiting for IO submission via io_uring."
> > > +AIO_IO_URING_COMPLETION	"Waiting for IO completion via io_uring."
> > >  AIO_IO_COMPLETION	"Waiting for IO completion."
> >
> > I'm wondering if there's an opportunity to enrich the last two wait event
> > names and/or descriptions.  The current descriptions suggest to me more
> > similarity than is actually there.  Inputs to the decision:
> >
> > - AIO_IO_COMPLETION waits for an IO in PGAIO_HS_DEFINED, PGAIO_HS_STAGED, or
> >   PGAIO_HS_COMPLETED_IO to reach PGAIO_HS_COMPLETED_SHARED.  The three
> >   starting states are the states where some other backend owns the next
> >   action, so the current backend can only wait to be signaled.
> >
> > - AIO_IO_URING_COMPLETION waits for the kernel to do enough so we can move
> >   from PGAIO_HS_SUBMITTED to PGAIO_HS_COMPLETED_IO.
> >
> > Possible names and descriptions, based on PgAioHandleState enum names and
> > comments:
> >
> > AIO_IO_URING_COMPLETED_IO	"Waiting for IO result via io_uring."
> > AIO_COMPLETED_SHARED	"Waiting for IO shared completion callback."
> >
> > If "shared completion callback" is too internals-focused, perhaps this:
> >
> > AIO_IO_URING_COMPLETED_IO	"Waiting for IO result via io_uring."
> > AIO_COMPLETED_SHARED	"Waiting for IO completion to update shared memory."
> 
> Hm, right now AIO_IO_COMPLETION also covers the actual "raw" execution of the
> IO with io_method=worker/sync.

Right, it could start with the IO in PGAIO_HS_DEFINED and end with the IO in
PGAIO_HS_COMPLETED_SHARED.  So another part of the wait may be the definer
doing work before exiting batch mode.

> For that AIO_COMPLETED_SHARED would be
> inappropriate.

The concept I had in mind was "waiting to reach PGAIO_HS_COMPLETED_SHARED,
whatever obstacles that involves".

Another candidate description string:

AIO_COMPLETED_SHARED	"Waiting for another process to complete IO."

> We could use a different wait event if wait for an IO via CV in
> PGAIO_HS_SUBMITTED, with a small refactoring of pgaio_io_wait().  But I'm not
> sure that would get you that far - we don't broadcast the CV when
> transitioning from PGAIO_HS_SUBMITTED -> PGAIO_HS_COMPLETED_IO, so the wait
> event would stay the same, now wrong, wait event until the shared callback
> completes. Obviously waking everyone up just so they can use a differen wait
> event doesn't make sense.

Agreed.  The mapping of code ranges to wait events seems fine to me.  I'm mainly
trying to optimize the wait event description strings to fit those code ranges.

> A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
> "execution" or something like that, to hint at a separation between the raw IO
> being completed and the IO, including the callbacks completing.

Yes, that would work for me.

> > > --- a/doc/src/sgml/config.sgml
> > > +++ b/doc/src/sgml/config.sgml
> > > @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> > >              <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > >             </para>
> > >            </listitem>
> > > +          <listitem>
> > > +           <para>
> > > +            <literal>io_uring</literal> (execute asynchronous I/O using
> > > +            io_uring, if available)
> >
> > I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
> > Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
> > there to reduce surprise on other platforms.
> 
> You're right, the if available can be misunderstood. But not mentioning that
> it's an optional dependency seems odd too. What about something like
> 
>            <para>
>             <literal>io_uring</literal> (execute asynchronous I/O using
>             io_uring, requires postgres to have been built with
>             <link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
>             <link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
>            </para>

I'd change s/postgres to have been built/a build with/ since the SGML docs
don't use the term "postgres" that way.  Otherwise, that works for me.

> Should the docs for --with-liburing/-Dliburing mention it's linux only? We
> don't seem to do that for things like systemd (linux), selinux (linux) and
> only kinda for bonjour (macos).

No need, I think.

  107. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-23T16:59:43Z

    Hi,

On 2025-03-23 08:55:29 -0700, Noah Misch wrote:
> On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote:
> Unrelated to the above, another question about io_uring:
> 
> commit da722699 wrote:
> > +/*
> > + * Need to submit staged but not yet submitted IOs using the fd, otherwise
> > + * the IO would end up targeting something bogus.
> > + */
> > +void
> > +pgaio_closing_fd(int fd)
> 
> An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
> PGAIO_HS_COMPLETED_IO clearly doesn't block that close.  For io_method=worker,
> closing in PGAIO_HS_SUBMITTED is okay.  For io_method=io_uring, is there a
> reference about it being okay to close during PGAIO_HS_SUBMITTED?  I looked
> awhile for an authoritative view on that, but I didn't find one.  If we can
> rely on io_uring_submit() returning only after the kernel has given the
> io_uring its own reference to all applicable file descriptors, I expect it's
> okay to close the process's FD.  If the io_uring acquires its reference later
> than that, I expect we shouldn't close before that later time.

I'm fairly sure io_uring has its own reference for the file descriptor by the
time io_uring_enter() returns [1].  What io_uring does *not* reliably tolerate
is the issuing process *exiting* before the IO completes, even if there are
other processes attached to the same io_uring instance.

AIO v1 had a posix_aio backend, which, on several platforms, did *not*
tolerate the FD being closed before the IO completes. Because of that
IoMethodOps had a closing_fd callback, which posix_aio used to wait for the
IO's completion [2].


I've added a test case exercising this path for all io methods. But I can't
think of a way that would catch io_uring not actually holding a reference to
the fd with a high likelihood - the IO will almost always complete quickly
enough to not be able to catch that. But it still seems better than not at all
testing the path - it does catch at least the problem of pgaio_closing_fd()
not doing anything.

Greetings,

Andres Freund

[1] See
  https://github.com/torvalds/linux/blob/586de92313fcab8ed84ac5f78f4d2aae2db92c59/io_uring/io_uring.c#L1728
  called from
  https://github.com/torvalds/linux/blob/586de92313fcab8ed84ac5f78f4d2aae2db92c59/io_uring/io_uring.c#L2204
  called from
  https://github.com/torvalds/linux/blob/586de92313fcab8ed84ac5f78f4d2aae2db92c59/io_uring/io_uring.c#L3372
  in the io_uring_enter() syscall

[2] https://github.com/anarazel/postgres/blob/a08cd717b5af4e51afb25ec86623973158a72ab9/src/backend/storage/aio/aio_posix.c#L738

  108. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-24T00:29:39Z

    commit 247ce06b wrote:
> +			pgaio_io_reopen(ioh);
> +
> +			/*
> +			 * To be able to exercise the reopen-fails path, allow injection
> +			 * points to trigger a failure at this point.
> +			 */
> +			pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
> +
> +			error_errno = 0;
> +			error_ioh = NULL;
> +
> +			/*
> +			 * We don't expect this to ever fail with ERROR or FATAL, no need
> +			 * to keep error_ioh set to the IO.
> +			 * pgaio_io_perform_synchronously() contains a critical section to
> +			 * ensure we don't accidentally fail.
> +			 */
> +			pgaio_io_perform_synchronously(ioh);

A CHECK_FOR_INTERRUPTS() could close() the FD that pgaio_io_reopen() callee
smgr_aio_reopen() stores.  Hence, I think smgrfd() should assert that
interrupts are held instead of doing its own HOLD_INTERRUPTS(), and a
HOLD_INTERRUPTS() should surround the above region of code.  It's likely hard
to reproduce a problem, because pgaio_io_call_inj() does nothing in many
builds, and pgaio_io_perform_synchronously() starts by entering a critical
section.

On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11

> Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd

> +int
> +FileStartReadV(PgAioHandle *ioh, File file,
> +			   int iovcnt, off_t offset,
> +			   uint32 wait_event_info)
> +{
> +	int			returnCode;
> +	Vfd		   *vfdP;
> +
> +	Assert(FileIsValid(file));
> +
> +	DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
> +			   file, VfdCache[file].fileName,
> +			   (int64) offset,
> +			   iovcnt));
> +
> +	returnCode = FileAccess(file);
> +	if (returnCode < 0)
> +		return returnCode;
> +
> +	vfdP = &VfdCache[file];
> +
> +	pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);

FileStartReadV() and pgaio_io_prep_readv() advance the IO to PGAIO_HS_STAGED
w/ batch mode, PGAIO_HS_SUBMITTED w/o batch mode.  I didn't expect that from
functions so named.  The "start" verb sounds to me like unconditional
PGAIO_HS_SUBMITTED, and the "prep" verb sounds like PGAIO_HS_DEFINED.  I like
the "stage" verb, because it matches PGAIO_HS_STAGED, and the comment at
PGAIO_HS_STAGED succinctly covers what to expect.  Hence, I recommend names
FileStageReadV, pgaio_io_stage_readv, mdstagereadv, and smgrstageread.  How do
you see it?

> +/*
> + * AIO error reporting callback for mdstartreadv().
> + *
> + * Errors are encoded as follows:
> + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0

I recommend replacing "errno != 0" with either "that errno" or "errno ==
error_data".


> Subject: [PATCH v2.11 07/27] aio: Add README.md explaining higher level design

Ready for commit apart from some trivia:

> +if (ioret.result.status == PGAIO_RS_ERROR)
> +    pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);

I think ioret and aio_ret are supposed to be the same object.  If that's
right, change one of the names.  Likewise elsewhere in this file.

> +The central API piece for postgres' AIO abstraction are AIO handles. To
> +execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
> +then "defined", i.e. associate an IO operation with the handle.

s/"defined"/"define" it/ or similar

> +The "solution" to this the ability to associate multiple completion callbacks

s/this the/this is the/


> Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well

> @@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
>  		Assert(refcount > 0);
>  		if (refcount != 1)
>  			return false;
> +
> +		/*
> +		 * Check that the AIO subsystem doesn't have a pin. Likely not
> +		 * possible today, but better safe than sorry.
> +		 */
> +		bufHdr = GetLocalBufferDescriptor(-buffer - 1);
> +		buf_state = pg_atomic_read_u32(&bufHdr->state);
> +		refcount = BUF_STATE_GET_REFCOUNT(buf_state);
> +		Assert(refcount > 0);
> +		if (refcount != 1)
> +			return false;
> +

LockBufferForCleanup() should get code like this
ConditionalLockBufferForCleanup() code, either now or when "not possible
today" ends.  Currently, it just assumes all local buffers are
cleanup-lockable:

	/* Nobody else to wait for */
	if (BufferIsLocal(buffer))
		return;

> @@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
>  
>  	buf_state = pg_atomic_read_u32(&bufHdr->state);
>  
> -	if (check_unreferenced && LocalRefCount[bufid] != 0)
> +	/*
> +	 * We need to test not just LocalRefCount[bufid] but also the BufferDesc
> +	 * itself, as the latter is used to represent a pin by the AIO subsystem.
> +	 * This can happen if AIO is initiated and then the query errors out.
> +	 */
> +	if (check_unreferenced &&
> +		(LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
>  		elog(ERROR, "block %u of %s is still referenced (local %u)",

I didn't write a test to prove it, but I'm suspecting we'll reach the above
ERROR with this sequence:

  CREATE TEMP TABLE foo ...;
  [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
  DROP TABLE foo;

DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true).  I
think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
the particular rel) before InvalidateLocalBuffer().  Or use something like the
logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
corresponding bufmgr code.  I think that bufmgr ERROR is unreachable, since
only a private refcnt triggers that bufmgr ERROR.  Is there something
preventing the localbuf error from being a problem?  (This wouldn't require
changes to the current patch; responsibility would fall in a bufmgr AIO
patch.)


> Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support

(Still reviewing this and later patches, but incidental observations follow.)

> +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> +						  bool failed, bool is_temp)
> +{
...
> +	PgAioResult result;
...
> +	result.status = PGAIO_RS_OK;
...
> +	return result;

gcc 14.2.0 -Werror gives me:

  bufmgr.c:7297:16: error: ‘result’ may be used uninitialized [-Werror=maybe-uninitialized]

Zeroing the unset fields silenced it:

--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -7221,3 +7221,3 @@ buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
 	char	   *bufdata = BufferGetBlock(buffer);
-	PgAioResult result;
+	PgAioResult result = { .status = PGAIO_RS_OK };
 	uint32		set_flag_bits;
@@ -7238,4 +7238,2 @@ buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
 
-	result.status = PGAIO_RS_OK;
-
 	/* check for garbage data */


> Subject: [PATCH v2.11 13/27] aio: Basic read_stream adjustments for real AIO

> @@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
>  static void
>  read_stream_look_ahead(ReadStream *stream)
>  {
> +	/*
> +	 * Allow amortizing the cost of submitting IO over multiple IOs. This
> +	 * requires that we don't do any operations that could lead to a deadlock
> +	 * with staged-but-unsubmitted IO.
> +	 */
> +	pgaio_enter_batchmode();

We call read_stream_get_block() while in batchmode, so the stream callback
needs to be ready for that.  A complicated case is
collect_corrupt_items_read_stream_next_block(), which may do its own buffer
I/O to read in a vmbuffer for VM_ALL_FROZEN().  That's feeling to me like a
recipe for corner cases reaching ERROR "starting batch while batch already in
progress".  Are there mitigating factors?


> Subject: [PATCH v2.11 17/27] aio: Add test_aio module

> +	# verify that page verification errors are detected even as part of a
> +	# shortened multi-block read (tbl_corr, block 1 is tbl_corred)

Is "tbl_corred" a typo of something?

> --- /dev/null
> +++ b/src/test/modules/test_aio/test_aio.c
> @@ -0,0 +1,657 @@
> +/*-------------------------------------------------------------------------
> + *
> + * delay_execution.c
> + *		Test module to allow delay between parsing and execution of a query.
> + *
> + * The delay is implemented by taking and immediately releasing a specified
> + * advisory lock.  If another process has previously taken that lock, the
> + * current process will be blocked until the lock is released; otherwise,
> + * there's no effect.  This allows an isolationtester script to reliably
> + * test behaviors where some specified action happens in another backend
> + * between parsing and execution of any desired query.
> + *
> + * Copyright (c) 2020-2025, PostgreSQL Global Development Group
> + *
> + * IDENTIFICATION
> + *	  src/test/modules/delay_execution/delay_execution.c

Header comment is surviving from copy-paste of delay_execution.c.

> +	 * Tor tests we don't want the resowner release preventing us from

s/Tor/For/

  109. Re: AIO v2.5

    Thomas Munro <thomas.munro@gmail.com> — 2025-03-24T01:11:24Z

    On Mon, Mar 24, 2025 at 5:59 AM Andres Freund <andres@anarazel.de> wrote:
> On 2025-03-23 08:55:29 -0700, Noah Misch wrote:
> > An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
> > PGAIO_HS_COMPLETED_IO clearly doesn't block that close.  For io_method=worker,
> > closing in PGAIO_HS_SUBMITTED is okay.  For io_method=io_uring, is there a
> > reference about it being okay to close during PGAIO_HS_SUBMITTED?  I looked
> > awhile for an authoritative view on that, but I didn't find one.  If we can
> > rely on io_uring_submit() returning only after the kernel has given the
> > io_uring its own reference to all applicable file descriptors, I expect it's
> > okay to close the process's FD.  If the io_uring acquires its reference later
> > than that, I expect we shouldn't close before that later time.
>
> I'm fairly sure io_uring has its own reference for the file descriptor by the
> time io_uring_enter() returns [1].  What io_uring does *not* reliably tolerate
> is the issuing process *exiting* before the IO completes, even if there are
> other processes attached to the same io_uring instance.

It is a bit strange that the documentation doesn't say that
explicitly.  You can sorta-maybe-kinda infer it from the fact that
io_uring didn't originally support cancelling requests at all, maybe a
small clue that it also didn't cancel them when you closed the fd :-)
The only sane alternative would seem to be that they keep running and
have their own reference to the *file* (not the fd), which is the
actual case, and might also be inferrable at a stretch from the
io_uring_register() documentation that says it reduces overheads with
a "long term reference" reducing "per-I/O overhead".  (The distant
third option/non-option is a sort of late/async binding fd as seen in
the Glibc user space POSIX AIO implementation, but that sort of
madness doesn't seem to be the sort of thing anyone working in the
kernel would entertain for a nanosecond...)  Anyway, there are also
public discussions involving Mr Axboe that discuss the fact that async
operations continue to run when the associated fd is closed, eg from
people who were surprised by that when porting stuff from other
systems, which might help fill in the documentation gap a teensy bit
if people want to see something outside the source code:

https://github.com/axboe/liburing/issues/568

> AIO v1 had a posix_aio backend, which, on several platforms, did *not*
> tolerate the FD being closed before the IO completes. Because of that
> IoMethodOps had a closing_fd callback, which posix_aio used to wait for the
> IO's completion [2].

Just for the record while remembering this stuff: Windows is another
system that took the cancel-on-close approach, so the Windows IOCP
proof-of-concept patches also used that AIO v1 callback and we'll have
to think about that again if/when we want to get that stuff
going on AIO v2.  I recall also speculating that it might be better to
teach the vfd system to pick another victim to close instead if an fd
was currently tied up with an asynchronous I/O for the benefit of
those cancel-on-close systems, hopefully without any happy-path
book-keeping.  But just submitting staged I/O is a nice and cheap
solution for now, without them in the picture.

  110. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-24T15:43:47Z

    Hi,

On 2025-03-23 17:29:39 -0700, Noah Misch wrote:
> commit 247ce06b wrote:
> > +			pgaio_io_reopen(ioh);
> > +
> > +			/*
> > +			 * To be able to exercise the reopen-fails path, allow injection
> > +			 * points to trigger a failure at this point.
> > +			 */
> > +			pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
> > +
> > +			error_errno = 0;
> > +			error_ioh = NULL;
> > +
> > +			/*
> > +			 * We don't expect this to ever fail with ERROR or FATAL, no need
> > +			 * to keep error_ioh set to the IO.
> > +			 * pgaio_io_perform_synchronously() contains a critical section to
> > +			 * ensure we don't accidentally fail.
> > +			 */
> > +			pgaio_io_perform_synchronously(ioh);
>
> A CHECK_FOR_INTERRUPTS() could close() the FD that pgaio_io_reopen() callee
> smgr_aio_reopen() stores.  Hence, I think smgrfd() should assert that
> interrupts are held instead of doing its own HOLD_INTERRUPTS(), and a
> HOLD_INTERRUPTS() should surround the above region of code.  It's likely hard
> to reproduce a problem, because pgaio_io_call_inj() does nothing in many
> builds, and pgaio_io_perform_synchronously() starts by entering a critical
> section.

Hm, I guess you're right - it would be pretty bonkers for the injection to
process interrupts, but its much better to clarify the code to make that not
an option.  Once doing that it seemed we should also have a similar assertion
in pgaio_io_before_prep() would be appropriate.



> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Attached v2.11
>
> > Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
>
> > +int
> > +FileStartReadV(PgAioHandle *ioh, File file,
> > +			   int iovcnt, off_t offset,
> > +			   uint32 wait_event_info)
> > +{
> > +	int			returnCode;
> > +	Vfd		   *vfdP;
> > +
> > +	Assert(FileIsValid(file));
> > +
> > +	DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
> > +			   file, VfdCache[file].fileName,
> > +			   (int64) offset,
> > +			   iovcnt));
> > +
> > +	returnCode = FileAccess(file);
> > +	if (returnCode < 0)
> > +		return returnCode;
> > +
> > +	vfdP = &VfdCache[file];
> > +
> > +	pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
>
> FileStartReadV() and pgaio_io_prep_readv() advance the IO to PGAIO_HS_STAGED
> w/ batch mode, PGAIO_HS_SUBMITTED w/o batch mode.  I didn't expect that from
> functions so named.  The "start" verb sounds to me like unconditional
> PGAIO_HS_SUBMITTED, and the "prep" verb sounds like PGAIO_HS_DEFINED.  I like
> the "stage" verb, because it matches PGAIO_HS_STAGED, and the comment at
> PGAIO_HS_STAGED succinctly covers what to expect.  Hence, I recommend names
> FileStageReadV, pgaio_io_stage_readv, mdstagereadv, and smgrstageread.  How do
> you see it?

I have a surprisingly strong negative reaction to that proposed naming. To me
the staging is a distinct step that happens *after* the IO is fully
defined. Making all the layered calls that lead up to that named that way
would IMO be a bad idea.

I however don't particularly like the *start* or *prep* names, I've gone back
and forth on those a couple times. I could see "begin" work uniformly across
those.


> > +/*
> > + * AIO error reporting callback for mdstartreadv().
> > + *
> > + * Errors are encoded as follows:
> > + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
>
> I recommend replacing "errno != 0" with either "that errno" or "errno ==
> error_data".

Done.


> > Subject: [PATCH v2.11 07/27] aio: Add README.md explaining higher level design
>
> Ready for commit apart from some trivia:

Great.


> > +if (ioret.result.status == PGAIO_RS_ERROR)
> > +    pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
>
> I think ioret and aio_ret are supposed to be the same object.  If that's
> right, change one of the names.  Likewise elsewhere in this file.

You're right.


> > +The central API piece for postgres' AIO abstraction are AIO handles. To
> > +execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
> > +then "defined", i.e. associate an IO operation with the handle.
>
> s/"defined"/"define" it/ or similar
>
> > +The "solution" to this the ability to associate multiple completion callbacks
>
> s/this the/this is the/

Applied.


> > Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
>
> > @@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
> >  		Assert(refcount > 0);
> >  		if (refcount != 1)
> >  			return false;
> > +
> > +		/*
> > +		 * Check that the AIO subsystem doesn't have a pin. Likely not
> > +		 * possible today, but better safe than sorry.
> > +		 */
> > +		bufHdr = GetLocalBufferDescriptor(-buffer - 1);
> > +		buf_state = pg_atomic_read_u32(&bufHdr->state);
> > +		refcount = BUF_STATE_GET_REFCOUNT(buf_state);
> > +		Assert(refcount > 0);
> > +		if (refcount != 1)
> > +			return false;
> > +
>
> LockBufferForCleanup() should get code like this
> ConditionalLockBufferForCleanup() code, either now or when "not possible
> today" ends.  Currently, it just assumes all local buffers are
> cleanup-lockable:
>
> 	/* Nobody else to wait for */
> 	if (BufferIsLocal(buffer))
> 		return;

Kinda, yes, kinda no?  LockBufferForCleanup() assumes, even for shared
buffers, that the current backend can't be doing anything that conflicts with
acquiring a buffer pin - note that it doesn't check the backend local pincount
for shared buffers either.


LockBufferForCleanup() kind of has to make that assumption, because there's no
way to wait for yourself to release another pin, because obviously waiting in
LockBufferForCleanup() would prevent that from ever happening.

It's somewhat disheartening that the comments for LockBufferForCleanup() don't
mention that somehow the caller needs to be sure not to be called with other
pins on a relation. Nor does LockBufferForCleanup() have any asserts checking
how many backend-local pins exist.


Leaving documentation / asserts aside, I think this is largely a safe
assumption given current callers. With one exception, it's all vacuum or
recovery related code - as vacuum can't run in a transaction, we can't
conflict with another pin by the same backend.

The one exception is heap_surgery.c - it doesn't quite seem safe, the
surrounding (or another query with a cursor) could have a pin on the target
block. The most obvious fix would be to use CheckTableNotInUse(), but that
might break some reasonable uses. Or maybe it should just not use a cleanup
lock, it's not obvious to me why it uses one.  But tbh, I don't care too much,
given what heap_surgery is.


> > @@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
> >
> >  	buf_state = pg_atomic_read_u32(&bufHdr->state);
> >
> > -	if (check_unreferenced && LocalRefCount[bufid] != 0)
> > +	/*
> > +	 * We need to test not just LocalRefCount[bufid] but also the BufferDesc
> > +	 * itself, as the latter is used to represent a pin by the AIO subsystem.
> > +	 * This can happen if AIO is initiated and then the query errors out.
> > +	 */
> > +	if (check_unreferenced &&
> > +		(LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
> >  		elog(ERROR, "block %u of %s is still referenced (local %u)",
>
> I didn't write a test to prove it, but I'm suspecting we'll reach the above
> ERROR with this sequence:
>
>   CREATE TEMP TABLE foo ...;
>   [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
>   DROP TABLE foo;

That seems plausible.  I'll try to write a test after this email.


> DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true).  I
> think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
> the particular rel) before InvalidateLocalBuffer().  Or use something like the
> logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
> corresponding bufmgr code.

Just waiting for the IO in InvalidateBuffer() does seem like the best bet to
me. It's going to be pretty rarely reached, waiting for all concurrent IO
seems unnecessarily heavyweight. I don't think it matters much today, but once
we do things like asynchronously writing back buffers or WAL, the situation
will be different.

I think this points to the comment above the WaitIO() in InvalidateBuffer()
needing a bit of adapting - an in-progress read can trigger the WaitIO as
well. Something like:

	/*
	 * We assume the reason for it to be pinned is that either we were
	 * asynchronously reading the page in before erroring out or someone else
	 * is flushing the page out.  Wait for the IO to finish.  (This could be
	 * an infinite loop if the refcount is messed up... it would be nice to
	 * time out after awhile, but there seems no way to be sure how many loops
	 * may be needed.  Note that if the other guy has pinned the buffer but
	 * not yet done StartBufferIO, WaitIO will fall through and we'll
	 * effectively be busy-looping here.)
	 */



> > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > +						  bool failed, bool is_temp)
> > +{
> ...
> > +	PgAioResult result;
> ...
> > +	result.status = PGAIO_RS_OK;
> ...
> > +	return result;
>
> gcc 14.2.0 -Werror gives me:
>
>   bufmgr.c:7297:16: error: ‘result’ may be used uninitialized [-Werror=maybe-uninitialized]

Gngngng.  Since when is it a bug for some fields of a struct to be
uninitialized, as long they're not used?

Interestingly I don't see that warning, despite also using gcc 14.2.0.

I'll just move to your solution, but it seems odd.


> > Subject: [PATCH v2.11 13/27] aio: Basic read_stream adjustments for real AIO
>
> > @@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
> >  static void
> >  read_stream_look_ahead(ReadStream *stream)
> >  {
> > +	/*
> > +	 * Allow amortizing the cost of submitting IO over multiple IOs. This
> > +	 * requires that we don't do any operations that could lead to a deadlock
> > +	 * with staged-but-unsubmitted IO.
> > +	 */
> > +	pgaio_enter_batchmode();
>
> We call read_stream_get_block() while in batchmode, so the stream callback
> needs to be ready for that.  A complicated case is
> collect_corrupt_items_read_stream_next_block(), which may do its own buffer
> I/O to read in a vmbuffer for VM_ALL_FROZEN().  That's feeling to me like a
> recipe for corner cases reaching ERROR "starting batch while batch already in
> progress".  Are there mitigating factors?

Ugh, yes, you're right.  heap_vac_scan_next_block() is also affected.

I don't think "starting batch while batch already in progress" is the real
issue though - it seems easy enough to avoid starting another batch inside,
partially because current cases seem unlikely to need to do batchable IO
inside. What worries me more is that code might block while there's
unsubmitted IO - which seems entirely plausible.


I can see a few approaches:

1) Declare that all read stream callbacks have to be careful and cope with
   batch mode

   I'm not sure how viable that is, not starting batches seems ok, but
   ensuring that the code doesn't block is a different story.


2) Have read stream users opt-in to batching

   Presumably via a flag like READ_STREAM_USE_BATCHING. That'd be easy enough
   to implement and to add to the callsites where that's fine.


3) Teach read stream to "look ahead" far enough to determine all the blocks
   that could be issued in a batch outside of batchmode

   I think that's probably not a great idea, it'd lead us to looking further
   ahead than we really need to, which could increase "unfairness" in
   e.g. parallel sequential scan.


4) Just defer using batch mode for now

   It's a nice win with io_uring for random IO, e.g. from bitmap heap scans ,
   but there's no need to immediately solve this.


I think regardless of what we go for, it's worth splitting
  "aio: Basic read_stream adjustments for real AIO"
into the actually basic parts (i.e. introducing sync_mode) from the not
actually so basic parts (i.e. batching).


I suspect that 2) would be the best approach. Only the read stream user knows
what it needs to do in the callback.


>
> > Subject: [PATCH v2.11 17/27] aio: Add test_aio module
>
> > +	# verify that page verification errors are detected even as part of a
> > +	# shortened multi-block read (tbl_corr, block 1 is tbl_corred)
>
> Is "tbl_corred" a typo of something?

I think that was a search&replace of the table name gone wrong. It was just
supposed to be "corrupted".


> > + *
> > + * IDENTIFICATION
> > + *	  src/test/modules/delay_execution/delay_execution.c
>
> Header comment is surviving from copy-paste of delay_execution.c.

Oh, how I hate these pointless comments. Fixed.

Greetings,

Andres Freund

  111. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-24T22:55:22Z

    Hi,

On 2025-03-24 11:43:47 -0400, Andres Freund wrote:
> > I didn't write a test to prove it, but I'm suspecting we'll reach the above
> > ERROR with this sequence:
> >
> >   CREATE TEMP TABLE foo ...;
> >   [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
> >   DROP TABLE foo;
> 
> That seems plausible.  I'll try to write a test after this email.

FWIW, a test did indeed confirm that.  Luckily:

> > DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true).  I
> > think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
> > the particular rel) before InvalidateLocalBuffer().  Or use something like the
> > logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
> > corresponding bufmgr code.
> 
> Just waiting for the IO in InvalidateBuffer() does seem like the best bet to
> me.

This did indeed resolve the issue.

I've extended the testsuite to test for that and a bunch more things. Working
on sending out a new version...


> > We call read_stream_get_block() while in batchmode, so the stream callback
> > needs to be ready for that.  A complicated case is
> > collect_corrupt_items_read_stream_next_block(), which may do its own buffer
> > I/O to read in a vmbuffer for VM_ALL_FROZEN().  That's feeling to me like a
> > recipe for corner cases reaching ERROR "starting batch while batch already in
> > progress".  Are there mitigating factors?
> 
> Ugh, yes, you're right.  heap_vac_scan_next_block() is also affected.
> 
> I don't think "starting batch while batch already in progress" is the real
> issue though - it seems easy enough to avoid starting another batch inside,
> partially because current cases seem unlikely to need to do batchable IO
> inside. What worries me more is that code might block while there's
> unsubmitted IO - which seems entirely plausible.
> 
> 
> I can see a few approaches:
> 
> 1) Declare that all read stream callbacks have to be careful and cope with
>    batch mode
> 
>    I'm not sure how viable that is, not starting batches seems ok, but
>    ensuring that the code doesn't block is a different story.
> 
> 
> 2) Have read stream users opt-in to batching
> 
>    Presumably via a flag like READ_STREAM_USE_BATCHING. That'd be easy enough
>    to implement and to add to the callsites where that's fine.
> 
> 
> 3) Teach read stream to "look ahead" far enough to determine all the blocks
>    that could be issued in a batch outside of batchmode
> 
>    I think that's probably not a great idea, it'd lead us to looking further
>    ahead than we really need to, which could increase "unfairness" in
>    e.g. parallel sequential scan.
> 
> 
> 4) Just defer using batch mode for now
> 
>    It's a nice win with io_uring for random IO, e.g. from bitmap heap scans ,
>    but there's no need to immediately solve this.
> 
> 
> I think regardless of what we go for, it's worth splitting
>   "aio: Basic read_stream adjustments for real AIO"
> into the actually basic parts (i.e. introducing sync_mode) from the not
> actually so basic parts (i.e. batching).
> 
> 
> I suspect that 2) would be the best approach. Only the read stream user knows
> what it needs to do in the callback.

I still think 2) would be the best option.

Writing a patch for that.

If a callback may sometimes need to block, it can still opt into
READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.

The hardest part is to explain the flag. Here's my current attempt:

/* ---
 * Opt-in to using AIO batchmode.
 *
 * Submitting IO in larger batches can be more efficient than doing so
 * one-by-one, particularly for many small reads. It does, however, require
 * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
 * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
 * a) block without first calling pgaio_submit_staged(), unless a
 *    to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
 *    never acquired in a nested fashion
 * b) directly or indirectly start another batch pgaio_enter_batchmode()
 *
 * As this requires care and is nontrivial in some cases, batching is only
 * used with explicit opt-in.
 * ---
 */
#define READ_STREAM_USE_BATCHING 0x08


Greetings,

Andres Freund

  112. Re: AIO v2.5

    Thomas Munro <thomas.munro@gmail.com> — 2025-03-25T00:07:49Z

    On Tue, Mar 25, 2025 at 11:55 AM Andres Freund <andres@anarazel.de> wrote:
> If a callback may sometimes need to block, it can still opt into
> READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.
>
> The hardest part is to explain the flag. Here's my current attempt:
>
> /* ---
>  * Opt-in to using AIO batchmode.
>  *
>  * Submitting IO in larger batches can be more efficient than doing so
>  * one-by-one, particularly for many small reads. It does, however, require
>  * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
>  * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
>  * a) block without first calling pgaio_submit_staged(), unless a
>  *    to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
>  *    never acquired in a nested fashion
>  * b) directly or indirectly start another batch pgaio_enter_batchmode()
>  *
>  * As this requires care and is nontrivial in some cases, batching is only
>  * used with explicit opt-in.
>  * ---
>  */
> #define READ_STREAM_USE_BATCHING 0x08

+1

I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
would be better, to highlight that you are making a declaration about
a property of your callback, not just turning on an independent
go-fast feature... I fished those words out of the main (?)
description of this topic atop pgaio_enter_batchmode().  Just a
thought, IDK.

  113. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T00:18:51Z

    Hi,

On 2025-03-25 13:07:49 +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 11:55 AM Andres Freund <andres@anarazel.de> wrote:
> > #define READ_STREAM_USE_BATCHING 0x08
> 
> +1
> 
> I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> would be better, to highlight that you are making a declaration about
> a property of your callback, not just turning on an independent
> go-fast feature... I fished those words out of the main (?)
> description of this topic atop pgaio_enter_batchmode().  Just a
> thought, IDK.

The relevant lines are already very deeply indented, so I'm a bit wary of such
a long name.  I think we'd basically have to use a separate flags variable
everywhere and that is annoying due to us following C89 variable declaration
positions...

Greetings,

Andres Freund

  114. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T00:45:37Z

    On Mon, Mar 24, 2025 at 11:43:47AM -0400, Andres Freund wrote:
> On 2025-03-23 17:29:39 -0700, Noah Misch wrote:
> > commit 247ce06b wrote:
> > > +			pgaio_io_reopen(ioh);
> > > +
> > > +			/*
> > > +			 * To be able to exercise the reopen-fails path, allow injection
> > > +			 * points to trigger a failure at this point.
> > > +			 */
> > > +			pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
> > > +
> > > +			error_errno = 0;
> > > +			error_ioh = NULL;
> > > +
> > > +			/*
> > > +			 * We don't expect this to ever fail with ERROR or FATAL, no need
> > > +			 * to keep error_ioh set to the IO.
> > > +			 * pgaio_io_perform_synchronously() contains a critical section to
> > > +			 * ensure we don't accidentally fail.
> > > +			 */
> > > +			pgaio_io_perform_synchronously(ioh);
> >
> > A CHECK_FOR_INTERRUPTS() could close() the FD that pgaio_io_reopen() callee
> > smgr_aio_reopen() stores.  Hence, I think smgrfd() should assert that
> > interrupts are held instead of doing its own HOLD_INTERRUPTS(), and a
> > HOLD_INTERRUPTS() should surround the above region of code.  It's likely hard
> > to reproduce a problem, because pgaio_io_call_inj() does nothing in many
> > builds, and pgaio_io_perform_synchronously() starts by entering a critical
> > section.
> 
> Hm, I guess you're right - it would be pretty bonkers for the injection to
> process interrupts, but its much better to clarify the code to make that not
> an option.  Once doing that it seemed we should also have a similar assertion
> in pgaio_io_before_prep() would be appropriate.

Agreed.  Following that line of thinking, the io_uring case needs to
HOLD_INTERRUPTS() (or hold smgrrelease() specifically) all the way from
pgaio_io_before_prep() to PGAIO_HS_SUBMITTED.  The fd has to stay valid until
io_uring_submit().

(We may be due for a test mode that does smgrreleaseall() at every
CHECK_FOR_INTERRUPTS()?)

> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
> >
> > > +int
> > > +FileStartReadV(PgAioHandle *ioh, File file,
> > > +			   int iovcnt, off_t offset,
> > > +			   uint32 wait_event_info)
> > > +{
> > > +	int			returnCode;
> > > +	Vfd		   *vfdP;
> > > +
> > > +	Assert(FileIsValid(file));
> > > +
> > > +	DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
> > > +			   file, VfdCache[file].fileName,
> > > +			   (int64) offset,
> > > +			   iovcnt));
> > > +
> > > +	returnCode = FileAccess(file);
> > > +	if (returnCode < 0)
> > > +		return returnCode;
> > > +
> > > +	vfdP = &VfdCache[file];
> > > +
> > > +	pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
> >
> > FileStartReadV() and pgaio_io_prep_readv() advance the IO to PGAIO_HS_STAGED
> > w/ batch mode, PGAIO_HS_SUBMITTED w/o batch mode.  I didn't expect that from
> > functions so named.  The "start" verb sounds to me like unconditional
> > PGAIO_HS_SUBMITTED, and the "prep" verb sounds like PGAIO_HS_DEFINED.  I like
> > the "stage" verb, because it matches PGAIO_HS_STAGED, and the comment at
> > PGAIO_HS_STAGED succinctly covers what to expect.  Hence, I recommend names
> > FileStageReadV, pgaio_io_stage_readv, mdstagereadv, and smgrstageread.  How do
> > you see it?
> 
> I have a surprisingly strong negative reaction to that proposed naming. To me
> the staging is a distinct step that happens *after* the IO is fully
> defined. Making all the layered calls that lead up to that named that way
> would IMO be a bad idea.

As a general naming principle, I think the name of a function that advances
through multiple named steps should mention the last step.  Naming the
function after just a non-last step feels weird to me.  For example, serving a
meal consists of steps menu_define, mix_ingredients, and plate_food.  It would
be weird to me if a function called meal_menu_define() mixed ingredients or
plated food, but it's fine if meal_plate_food() does all three steps.  A
second strategy is to name both the first and last steps:
meal_define_menu_thru_plate_food() is fine apart from being long.  A third
strategy is to have meal_plate_food() assert than meal_mix_ingredients() has
been called.

I wouldn't mind "staging" as a distinct step, but I think today's API
boundaries hide the distinction.  PGAIO_HS_DEFINED is a temporary state during
a pgaio_io_stage() call, so the process that defines and stages the IO can
observe PGAIO_HS_DEFINED only while pgaio_io_stage() is on the stack.

The aforementioned "third strategy" could map to having distinct
smgrdefinereadv() and smgrstagereadv().  I don't know how well that would work
out overall.  I wouldn't be optimistic about that winning, but I mention it
for completeness.

> I however don't particularly like the *start* or *prep* names, I've gone back
> and forth on those a couple times. I could see "begin" work uniformly across
> those.

For ease of new readers understanding things, I think it helps for the
functions that advance PgAioHandleState to have names that use words from
PgAioHandleState.  It's one less mapping to get into the reader's head.
"Begin", "Start" and "prep" are all outside that taxonomy, making the reader
learn how to map them to the taxonomy.  What reward does the reader get at the
end of that exercise?  I'm not seeing one, but please do tell me what I'm
missing here.

> > > Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
> >
> > > @@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
> > >  		Assert(refcount > 0);
> > >  		if (refcount != 1)
> > >  			return false;
> > > +
> > > +		/*
> > > +		 * Check that the AIO subsystem doesn't have a pin. Likely not
> > > +		 * possible today, but better safe than sorry.
> > > +		 */
> > > +		bufHdr = GetLocalBufferDescriptor(-buffer - 1);
> > > +		buf_state = pg_atomic_read_u32(&bufHdr->state);
> > > +		refcount = BUF_STATE_GET_REFCOUNT(buf_state);
> > > +		Assert(refcount > 0);
> > > +		if (refcount != 1)
> > > +			return false;
> > > +
> >
> > LockBufferForCleanup() should get code like this
> > ConditionalLockBufferForCleanup() code, either now or when "not possible
> > today" ends.  Currently, it just assumes all local buffers are
> > cleanup-lockable:
> >
> > 	/* Nobody else to wait for */
> > 	if (BufferIsLocal(buffer))
> > 		return;
> 
> Kinda, yes, kinda no?  LockBufferForCleanup() assumes, even for shared
> buffers, that the current backend can't be doing anything that conflicts with
> acquiring a buffer pin - note that it doesn't check the backend local pincount
> for shared buffers either.

It checks the local pincount via callee CheckBufferIsPinnedOnce().

As the patch stands, LockBufferForCleanup() can succeed when
ConditionalLockBufferForCleanup() would have returned false.  I'm not seeking
to raise the overall standard of *Cleanup() family of functions, but I am
trying to keep members of that family agreeing on the standard.

Like the comment, I expect it's academic today.  I expect it will stay
academic.  Anything that does a cleanup will start by reading the buffer,
which will resolve any refcnt the AIO subsystems holds for a read.  If there's
an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
that.  How about just removing the ConditionalLockBufferForCleanup() changes
or replacing them with a comment (like the present paragraph)?

> I think this points to the comment above the WaitIO() in InvalidateBuffer()
> needing a bit of adapting - an in-progress read can trigger the WaitIO as
> well. Something like:
> 
> 	/*
> 	 * We assume the reason for it to be pinned is that either we were
> 	 * asynchronously reading the page in before erroring out or someone else
> 	 * is flushing the page out.  Wait for the IO to finish.  (This could be
> 	 * an infinite loop if the refcount is messed up... it would be nice to
> 	 * time out after awhile, but there seems no way to be sure how many loops
> 	 * may be needed.  Note that if the other guy has pinned the buffer but
> 	 * not yet done StartBufferIO, WaitIO will fall through and we'll
> 	 * effectively be busy-looping here.)
> 	 */

Agreed.

> > > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > > +						  bool failed, bool is_temp)
> > > +{
> > ...
> > > +	PgAioResult result;
> > ...
> > > +	result.status = PGAIO_RS_OK;
> > ...
> > > +	return result;
> >
> > gcc 14.2.0 -Werror gives me:
> >
> >   bufmgr.c:7297:16: error: ‘result’ may be used uninitialized [-Werror=maybe-uninitialized]
> 
> Gngngng.  Since when is it a bug for some fields of a struct to be
> uninitialized, as long they're not used?
> 
> Interestingly I don't see that warning, despite also using gcc 14.2.0.

I badly neglected to mention my non-default flags:

CFLAGS='-O2 -fno-sanitize-recover=all -fsanitize=address,alignment,undefined --param=max-vartrack-size=150000000 -ftrivial-auto-var-init=pattern'
COPT=-Werror -Wno-error=array-bounds

Final CFLAGS, including the ones "configure" elects on its own:

configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Werror=unguarded-availability-new -Wendif-labels -Wmissing-format-attribute -Wcast-function-type -Wformat-security -Wmissing-variable-declarations -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -Wno-format-truncation -Wno-cast-function-type-strict -g -O2 -fno-sanitize-recover=all -fsanitize=address,alignment,undefined --param=max-vartrack-size=150000000 -ftrivial-auto-var-init=pattern

(I use -Wno-error=array-bounds because the sanitizer options elicit a lot of
those warnings.  Today's master is free from maybe-uninitialized warnings in
this configuration, though.)

> I'll just move to your solution, but it seems odd.

Got it.

> I think regardless of what we go for, it's worth splitting
>   "aio: Basic read_stream adjustments for real AIO"
> into the actually basic parts (i.e. introducing sync_mode) from the not
> actually so basic parts (i.e. batching).

Fair.

On Mon, Mar 24, 2025 at 06:55:22PM -0400, Andres Freund wrote:
> Hi,
> 
> On 2025-03-24 11:43:47 -0400, Andres Freund wrote:
> > > I didn't write a test to prove it, but I'm suspecting we'll reach the above
> > > ERROR with this sequence:
> > >
> > >   CREATE TEMP TABLE foo ...;
> > >   [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
> > >   DROP TABLE foo;
> > 
> > That seems plausible.  I'll try to write a test after this email.
> 
> FWIW, a test did indeed confirm that.  Luckily:
> 
> > > DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true).  I
> > > think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
> > > the particular rel) before InvalidateLocalBuffer().  Or use something like the
> > > logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
> > > corresponding bufmgr code.
> > 
> > Just waiting for the IO in InvalidateBuffer() does seem like the best bet to
> > me.
> 
> This did indeed resolve the issue.

I'm happy with that approach.

On Tue, Mar 25, 2025 at 01:07:49PM +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 11:55 AM Andres Freund <andres@anarazel.de> wrote:
> > If a callback may sometimes need to block, it can still opt into
> > READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.
> >
> > The hardest part is to explain the flag. Here's my current attempt:
> >
> > /* ---
> >  * Opt-in to using AIO batchmode.
> >  *
> >  * Submitting IO in larger batches can be more efficient than doing so
> >  * one-by-one, particularly for many small reads. It does, however, require
> >  * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> >  * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> >  * a) block without first calling pgaio_submit_staged(), unless a
> >  *    to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> >  *    never acquired in a nested fashion
> >  * b) directly or indirectly start another batch pgaio_enter_batchmode()

I think a callback could still do:

  pgaio_exit_batchmode()
  ... arbitrary code that might reach pgaio_enter_batchmode() ...
  pgaio_enter_batchmode()

> >  *
> >  * As this requires care and is nontrivial in some cases, batching is only
> >  * used with explicit opt-in.
> >  * ---
> >  */
> > #define READ_STREAM_USE_BATCHING 0x08
> 
> +1

Agreed.  It's simple, and there's no loss of generality.

> I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> would be better, to highlight that you are making a declaration about
> a property of your callback, not just turning on an independent
> go-fast feature... I fished those words out of the main (?)
> description of this topic atop pgaio_enter_batchmode().  Just a
> thought, IDK.

Good points.  I lean toward your renaming suggestion, or shortening to
READ_STREAM_BATCHMODE_AWARE or READ_STREAM_BATCH_OK.  I'm also fine with the
original name, though.

Thanks,
nm

  115. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T01:18:06Z

    Hi,

Attached v2.12, with the following changes:

- Pushed the max_files_per_process change

  I plan to look at what parts of Jelte's change is worth doing ontop.

  Thanks for the review Noah.


- Rebased over Thomas' commit of the remaining read stream changes

  Yay!


- Addressed Noah's review comments


- Added another test to test_aio/, to test that changing io_workers while
  running works, and that workers are restarted if terminated

  Written by Bilal


- Made InvalidateLocalBuffer wait for IO if necessary

  As reported / suggested by Noah


- Added tests for dropping tables with ongoing IO

  This failed, as Noah predicted, without the InvalidateLocalBuffer() change.


- Added a commit to explicitly hold interrupts in workers after
  pgaio_io_reopen()

  As suggested by Noah.


- Added a commit to fix a logic error around what gets passed to
  ioh->report_return - this lead to temporary buffer validation errors not
  being reported

  Discovered while extending the tests, as noted in the next point.

  I could see a few different "formulations" of this change (e.g. the
  report_return stuff could be populated by pgaio_io_call_complete_local()
  instead), but I don't think it matters much.


- Add temporary table coverage to test_aio

  This required hanged test_aio.c to cope with temporary tables as well.


- io_uring tests don't run anymore when built with EXEC_BACKEND and liburing
  enabled


- Split the read stream patch into two

  Noah, quite rightly, pointed out that it's not safe to use batching if the
  next-block callback may block (or start its own batch). The best idea seems
  to be to make users of read stream opt-in to batching.  I've done that in a
  patch that uses where it seems safe without doing extra work. See also the
  commit message.


- Added a commit to add I/O, Asynchronous I/O glossary and acronym entries


- Docs for pg_aios


- Renamed pg_aios.offset to off, to avoid use of a keyword


- Updated the io_uring wait event name while waiting for IOs to complete to
  AIO_IO_URING_COMPLETION and updated the description of AIO_IO_COMPLETION to
  "Waiting for another process to complete IO."

  I think this is a mix of different suggestions by Noah.


TODO:


- There are more tests in test_aio that should be expanded to run for temp
  tables as well, not just normal tables


- Add an explicit test for the checksum verification in the completion callback

  There is an existing test for testing an invalid page due to page header
  verification in test_aio, but not for checksum failures.

  I think it's indirectly covered (e.g. in amcheck), but seems better to test
  it explicitly.

  Wonder if it's worth adding some coverage for when checksums are disabled?
  Probably not necessary?


Greetings,

Andres Freund

  116. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T01:40:47Z

    Hi,

On 2025-03-23 09:32:48 -0700, Noah Misch wrote:
> Another candidate description string:
> 
> AIO_COMPLETED_SHARED	"Waiting for another process to complete IO."

I liked that one and adopted it.


> > A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
> > "execution" or something like that, to hint at a separation between the raw IO
> > being completed and the IO, including the callbacks completing.
> 
> Yes, that would work for me.

I updated both the name and the description of this one to EXECUTION, but I'm
not sure I like it for the name...


> > > > --- a/doc/src/sgml/config.sgml
> > > > +++ b/doc/src/sgml/config.sgml
> > > > @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> > > >              <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > > >             </para>
> > > >            </listitem>
> > > > +          <listitem>
> > > > +           <para>
> > > > +            <literal>io_uring</literal> (execute asynchronous I/O using
> > > > +            io_uring, if available)
> > >
> > > I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
> > > Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
> > > there to reduce surprise on other platforms.
> > 
> > You're right, the if available can be misunderstood. But not mentioning that
> > it's an optional dependency seems odd too. What about something like
> > 
> >            <para>
> >             <literal>io_uring</literal> (execute asynchronous I/O using
> >             io_uring, requires postgres to have been built with
> >             <link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
> >             <link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
> >            </para>
> 
> I'd change s/postgres to have been built/a build with/ since the SGML docs
> don't use the term "postgres" that way.  Otherwise, that works for me.

Went with that.

Greetings,

Andres Freund

  117. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T02:20:37Z

    On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support

[I checked that v2.12 doesn't invalidate these review comments, but I didn't
technically rebase the review onto v2.12's line numbers.]

>  static void
>  TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
> -				  bool forget_owner)
> +				  bool forget_owner, bool syncio)
>  {
>  	uint32		buf_state;
>  
> @@ -5586,6 +5636,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
>  	if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
>  		buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
>  
> +	if (!syncio)
> +	{
> +		/* release ownership by the AIO subsystem */
> +		Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
> +		buf_state -= BUF_REFCOUNT_ONE;
> +		pgaio_wref_clear(&buf->io_wref);
> +	}

Looking at the callers:

ZeroAndLockBuffer[1083]        TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
ExtendBufferedRelShared[2869]  TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
FlushBuffer[4827]              TerminateBufferIO(buf, true, 0, true, true);
AbortBufferIO[6637]            TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
buffer_readv_complete_one[7279] TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
buffer_writev_complete_one[7427] TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);

I think we can improve on the "syncio" arg name.  The first two aren't doing
IO, and AbortBufferIO() may be cleaning up what would have been an AIO if it
hadn't failed early.  Perhaps name the arg "release_aio" and pass
release_aio=true instead of syncio=false (release_aio = !syncio).

> +		 * about which buffers are target by IO can be hard to debug, making

s/target/targeted/

> +static pg_attribute_always_inline PgAioResult
> +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> +						  bool failed, bool is_temp)
> +{
...
> +		if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
> +		{
> +			ereport(WARNING,
> +					(errcode(ERRCODE_DATA_CORRUPTED),
> +					 errmsg("invalid page in block %u of relation %s; zeroing out page",

My earlier review requested s/LOG/WARNING/, but I wasn't thinking about this
in full depth.  In the !is_temp case, this runs in a complete_shared callback.
A process unrelated to the original IO may run this callback.  That's
unfortunate in two ways.  First, that other process's client gets an
unexpected WARNING.  The process getting the WARNING may not even have
zero_damaged_pages enabled.  Second, the client of the process that staged the
IO gets no message.

AIO ERROR-level messages handle this optimally.  We emit a LOG-level message
in the process that runs the complete_shared callback, and we arrange for the
ERROR-level message in the stager.  That would be ideal here: LOG in the
complete_shared runner, WARNING in the stager.

One could simplify things by forcing io_method=sync under ZERO_ON_ERROR ||
zero_damaged_pages, perhaps as a short-term approach.

Thoughts?

  118. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T02:30:27Z

    Hi,

On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> (We may be due for a test mode that does smgrreleaseall() at every
> CHECK_FOR_INTERRUPTS()?)

I suspect we are. I'm a bit afraid of even trying...

...

It's extremely slow - but at least the main regression as well as the aio tests pass!


> > I however don't particularly like the *start* or *prep* names, I've gone back
> > and forth on those a couple times. I could see "begin" work uniformly across
> > those.
> 
> For ease of new readers understanding things, I think it helps for the
> functions that advance PgAioHandleState to have names that use words from
> PgAioHandleState.  It's one less mapping to get into the reader's head.

Unfortunately for me it's kind of the opposite in this case, see below.


> "Begin", "Start" and "prep" are all outside that taxonomy, making the reader
> learn how to map them to the taxonomy.  What reward does the reader get at the
> end of that exercise?  I'm not seeing one, but please do tell me what I'm
> missing here.

Because the end state varies, depending on the number of previously staged
IOs, the IO method and whether batchmode is enabled, I think it's better if
the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
*not* aligned with an internal state name.  It will just mislead readers to
think that there's a deterministic mapping when that does not exist.

That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
naming that I just stopped seeing.

I'll try to think more about this, perhaps I can make myself see your POV
more.



> > > > Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
> > > LockBufferForCleanup() should get code like this
> > > ConditionalLockBufferForCleanup() code, either now or when "not possible
> > > today" ends.  Currently, it just assumes all local buffers are
> > > cleanup-lockable:
> > >
> > > 	/* Nobody else to wait for */
> > > 	if (BufferIsLocal(buffer))
> > > 		return;
> > 
> > Kinda, yes, kinda no?  LockBufferForCleanup() assumes, even for shared
> > buffers, that the current backend can't be doing anything that conflicts with
> > acquiring a buffer pin - note that it doesn't check the backend local pincount
> > for shared buffers either.
> 
> It checks the local pincount via callee CheckBufferIsPinnedOnce().

In exactly one of the callers :/


> As the patch stands, LockBufferForCleanup() can succeed when
> ConditionalLockBufferForCleanup() would have returned false.

That's already true today, right? In master ConditionalLockBufferForCleanup()
for temp buffers checks LocalRefCount, whereas LockBufferForCleanup() doesn't.
I think I agree with your suggestion further below, but independent of that, I
don't see how the current modification in the patch makes the worse.

Historically this behaviour of LockBufferForCleanup() kinda somewhat makes
sense - the only place we use LockBufferForCleanup() is in a non-transactional
command i.e. vacuum / index vacuum. So LockBufferForCleanup() turns out to
only be safe in that context.


> Like the comment, I expect it's academic today.  I expect it will stay
> academic.  Anything that does a cleanup will start by reading the buffer,
> which will resolve any refcnt the AIO subsystems holds for a read.  If there's
> an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> that.  How about just removing the ConditionalLockBufferForCleanup() changes
> or replacing them with a comment (like the present paragraph)?

I think we'll need an expanded version of what I suggest once we have writes -
but as you say, it shouldn't matter as long as we only have reads. So I think
moving the relevant changes, with adjusted caveats, to the bufmgr: write
change makes sense.



> > > /* ---
> > >  * Opt-in to using AIO batchmode.
> > >  *
> > >  * Submitting IO in larger batches can be more efficient than doing so
> > >  * one-by-one, particularly for many small reads. It does, however, require
> > >  * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> > >  * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> > >  * a) block without first calling pgaio_submit_staged(), unless a
> > >  *    to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> > >  *    never acquired in a nested fashion
> > >  * b) directly or indirectly start another batch pgaio_enter_batchmode()
> 
> I think a callback could still do:
> 
>   pgaio_exit_batchmode()
>   ... arbitrary code that might reach pgaio_enter_batchmode() ...
>   pgaio_enter_batchmode()

Yea - but I somehow doubt there are many cases where it makes sense to
deep-queue IOs within the callback. The cases I can think of are things like
ensuring the right VM buffer is in s_b.  But if it turns out to be necessary,
what you seuggest would be an out.

Do you think it's worth mentioning the above workaround? I'm mildly inclined
that not.

If it turns out to be actually useful to do nested batching, we can change it
so that nested batching *is* allowed, that'd not be hard.



> > >  *
> > >  * As this requires care and is nontrivial in some cases, batching is only
> > >  * used with explicit opt-in.
> > >  * ---
> > >  */
> > > #define READ_STREAM_USE_BATCHING 0x08
> > 
> > +1
> 
> Agreed.  It's simple, and there's no loss of generality.
> 
> > I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> > would be better, to highlight that you are making a declaration about
> > a property of your callback, not just turning on an independent
> > go-fast feature... I fished those words out of the main (?)
> > description of this topic atop pgaio_enter_batchmode().  Just a
> > thought, IDK.
> 
> Good points.  I lean toward your renaming suggestion, or shortening to
> READ_STREAM_BATCHMODE_AWARE or READ_STREAM_BATCH_OK.  I'm also fine with the
> original name, though.

I'm ok with all of these. In order of preference:

1) READ_STREAM_USE_BATCHING or READ_STREAM_BATCH_OK
2) READ_STREAM_BATCHMODE_AWARE
3) READ_STREAM_CALLBACK_BATCHMODE_AWARE

Greetings,

Andres Freund

  119. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T02:52:19Z

    Hi,

On 2025-03-24 19:20:37 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> >  static void
> >  TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
> > -				  bool forget_owner)
> > +				  bool forget_owner, bool syncio)
> > ...
> Looking at the callers:
> 
> ZeroAndLockBuffer[1083]        TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
> ExtendBufferedRelShared[2869]  TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
> FlushBuffer[4827]              TerminateBufferIO(buf, true, 0, true, true);
> AbortBufferIO[6637]            TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
> buffer_readv_complete_one[7279] TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
> buffer_writev_complete_one[7427] TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
> 
> I think we can improve on the "syncio" arg name.  The first two aren't doing
> IO, and AbortBufferIO() may be cleaning up what would have been an AIO if it
> hadn't failed early.  Perhaps name the arg "release_aio" and pass
> release_aio=true instead of syncio=false (release_aio = !syncio).

Yes, I think that makes sense.  Will do that tomorrow.


> > +static pg_attribute_always_inline PgAioResult
> > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > +						  bool failed, bool is_temp)
> > +{
> ...
> > +		if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
> > +		{
> > +			ereport(WARNING,
> > +					(errcode(ERRCODE_DATA_CORRUPTED),
> > +					 errmsg("invalid page in block %u of relation %s; zeroing out page",
> 
> My earlier review requested s/LOG/WARNING/, but I wasn't thinking about this
> in full depth. In the !is_temp case, this runs in a complete_shared
> callback.  A process unrelated to the original IO may run this callback.
> That's unfortunate in two ways.  First, that other process's client gets an
> unexpected WARNING.  The process getting the WARNING may not even have
> zero_damaged_pages enabled.  Second, the client of the process that staged
> the IO gets no message.

Ah, right. That could be why I had flipped it. If so, shame on me for not
adding a comment...


> AIO ERROR-level messages handle this optimally.  We emit a LOG-level message
> in the process that runs the complete_shared callback, and we arrange for the
> ERROR-level message in the stager.  That would be ideal here: LOG in the
> complete_shared runner, WARNING in the stager.

We could obviously downgrade (crossgrade? A LOG is more severe than a LOG in
some ways, but not others) the message when run in a different backend fairly
easily.  Still emitting a WARNING in the stager however is a bit more tricky.

Before thinking more deeply about how we could emit WARNING in the stage:

Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that could
trigger a rather massive flood of messages to the client in a *normal*
situation. I'm thinking of something like an insert extending a relation some
time after an immediate restart and encountering a lot of FSM corruption (due
to its non-crash-safe-ness) during the search for free space and the
subsequent FSM vacuum.  It might be ok to LOG that, but sending a lot of
WARNINGs to the client seems not quite right.


If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
then could tell the stager to issue the WARNING. It would add a bit of
distributed cost, both to callbacks and users of AIO, but it might not be too
bad.


> One could simplify things by forcing io_method=sync under ZERO_ON_ERROR ||
> zero_damaged_pages, perhaps as a short-term approach.

Yea, that could work.  Perhaps even just for zero_damaged_pages, after
changing it so that ZERO_ON_ERROR always just LOGs.

Hm, it seems somewhat nasty to have rather different performance
characteristics when forced to use zero_damaged_pages to recover from a
problem. Imagine an instance that's configured to use DIO and then needs to
use zero_damaged_pages to recove from corruption...


/me adds writing a test for both ZERO_ON_ERROR and zero_damaged_pages to the
TODO.

Greetings,

Andres Freund

  120. Re: AIO v2.5

    Thomas Munro <thomas.munro@gmail.com> — 2025-03-25T04:10:19Z

    On Tue, Mar 25, 2025 at 2:18 PM Andres Freund <andres@anarazel.de> wrote:
> Attached v2.12, with the following changes:

Here's a tiny fixup to make io_concurrency=0 turn on
READ_BUFFERS_SYNCHRONOUSLY as mooted in a FIXME.  Without this, AIO
will still run at level 1 even if you asked for 0.  Feel free to
squash, or ignore and I'll push it later, whatever suits... (tested on
the tip of your public aio-2 branch).

  121. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T13:33:21Z

    On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > (We may be due for a test mode that does smgrreleaseall() at every
> > CHECK_FOR_INTERRUPTS()?)
> 
> I suspect we are. I'm a bit afraid of even trying...
> 
> ...
> 
> It's extremely slow - but at least the main regression as well as the aio tests pass!

One less thing!

> > > I however don't particularly like the *start* or *prep* names, I've gone back
> > > and forth on those a couple times. I could see "begin" work uniformly across
> > > those.
> > 
> > For ease of new readers understanding things, I think it helps for the
> > functions that advance PgAioHandleState to have names that use words from
> > PgAioHandleState.  It's one less mapping to get into the reader's head.
> 
> Unfortunately for me it's kind of the opposite in this case, see below.
> 
> 
> > "Begin", "Start" and "prep" are all outside that taxonomy, making the reader
> > learn how to map them to the taxonomy.  What reward does the reader get at the
> > end of that exercise?  I'm not seeing one, but please do tell me what I'm
> > missing here.
> 
> Because the end state varies, depending on the number of previously staged
> IOs, the IO method and whether batchmode is enabled, I think it's better if
> the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
> *not* aligned with an internal state name.  It will just mislead readers to
> think that there's a deterministic mapping when that does not exist.

That's fair.  Could we provide the mapping in a comment, something like the
following?

--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -34,5 +34,10 @@
  * linearly through all states.
  *
- * State changes should all go through pgaio_io_update_state().
+ * State changes should all go through pgaio_io_update_state().  Its callers
+ * use these naming conventions:
+ *
+ * - A "start" function (e.g. FileStartReadV()) moves an IO from
+ *   PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
+ *   PGAIO_HS_COMPLETED_LOCAL.
  */
 typedef enum PgAioHandleState

> That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
> naming that I just stopped seeing.
> 
> I'll try to think more about this, perhaps I can make myself see your POV
> more.

> > As the patch stands, LockBufferForCleanup() can succeed when
> > ConditionalLockBufferForCleanup() would have returned false.
> 
> That's already true today, right? In master ConditionalLockBufferForCleanup()
> for temp buffers checks LocalRefCount, whereas LockBufferForCleanup() doesn't.

I'm finding a LocalRefCount check under LockBufferForCleanup:

LockBufferForCleanup(Buffer buffer)
{
...
	CheckBufferIsPinnedOnce(buffer);

CheckBufferIsPinnedOnce(Buffer buffer)
{
	if (BufferIsLocal(buffer))
	{
		if (LocalRefCount[-buffer - 1] != 1)
			elog(ERROR, "incorrect local pin count: %d",
				 LocalRefCount[-buffer - 1]);
	}
	else
	{
		if (GetPrivateRefCount(buffer) != 1)
			elog(ERROR, "incorrect local pin count: %d",
				 GetPrivateRefCount(buffer));
	}
}

> > Like the comment, I expect it's academic today.  I expect it will stay
> > academic.  Anything that does a cleanup will start by reading the buffer,
> > which will resolve any refcnt the AIO subsystems holds for a read.  If there's
> > an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> > that.  How about just removing the ConditionalLockBufferForCleanup() changes
> > or replacing them with a comment (like the present paragraph)?
> 
> I think we'll need an expanded version of what I suggest once we have writes -
> but as you say, it shouldn't matter as long as we only have reads. So I think
> moving the relevant changes, with adjusted caveats, to the bufmgr: write
> change makes sense.

Moving those changes works for me.  I'm not currently seeing the need under
writes, but that may get clearer upon reaching those patches.

> > > > /* ---
> > > >  * Opt-in to using AIO batchmode.
> > > >  *
> > > >  * Submitting IO in larger batches can be more efficient than doing so
> > > >  * one-by-one, particularly for many small reads. It does, however, require
> > > >  * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> > > >  * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> > > >  * a) block without first calling pgaio_submit_staged(), unless a
> > > >  *    to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> > > >  *    never acquired in a nested fashion
> > > >  * b) directly or indirectly start another batch pgaio_enter_batchmode()
> > 
> > I think a callback could still do:
> > 
> >   pgaio_exit_batchmode()
> >   ... arbitrary code that might reach pgaio_enter_batchmode() ...
> >   pgaio_enter_batchmode()
> 
> Yea - but I somehow doubt there are many cases where it makes sense to
> deep-queue IOs within the callback. The cases I can think of are things like
> ensuring the right VM buffer is in s_b.  But if it turns out to be necessary,
> what you seuggest would be an out.

I don't foresee a callback specifically wanting to batch, but callbacks might
call into other infrastructure that can elect to batch.  The exit+reenter
pattern would be better than adding no-batch options to other infrastructure.

> Do you think it's worth mentioning the above workaround? I'm mildly inclined
> that not.

Perhaps not in that detail, but perhaps we can rephrase (b) to not imply
exit+reenter is banned.  Maybe "(b) start another batch (without first exiting
one)".  It's also fine as-is, though.

> If it turns out to be actually useful to do nested batching, we can change it
> so that nested batching *is* allowed, that'd not be hard.

Good point.

> I'm ok with all of these. In order of preference:
> 
> 1) READ_STREAM_USE_BATCHING or READ_STREAM_BATCH_OK
> 2) READ_STREAM_BATCHMODE_AWARE
> 3) READ_STREAM_CALLBACK_BATCHMODE_AWARE

Same for me.

  122. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T14:11:20Z

    On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> On 2025-03-24 19:20:37 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > +static pg_attribute_always_inline PgAioResult
> > > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > > +						  bool failed, bool is_temp)
> > > +{
> > ...
> > > +		if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
> > > +		{
> > > +			ereport(WARNING,
> > > +					(errcode(ERRCODE_DATA_CORRUPTED),
> > > +					 errmsg("invalid page in block %u of relation %s; zeroing out page",
> > 
> > My earlier review requested s/LOG/WARNING/, but I wasn't thinking about this
> > in full depth. In the !is_temp case, this runs in a complete_shared
> > callback.  A process unrelated to the original IO may run this callback.
> > That's unfortunate in two ways.  First, that other process's client gets an
> > unexpected WARNING.  The process getting the WARNING may not even have
> > zero_damaged_pages enabled.  Second, the client of the process that staged
> > the IO gets no message.
> 
> Ah, right. That could be why I had flipped it. If so, shame on me for not
> adding a comment...
> 
> 
> > AIO ERROR-level messages handle this optimally.  We emit a LOG-level message
> > in the process that runs the complete_shared callback, and we arrange for the
> > ERROR-level message in the stager.  That would be ideal here: LOG in the
> > complete_shared runner, WARNING in the stager.
> 
> We could obviously downgrade (crossgrade? A LOG is more severe than a LOG in
> some ways, but not others) the message when run in a different backend fairly
> easily.  Still emitting a WARNING in the stager however is a bit more tricky.
> 
> Before thinking more deeply about how we could emit WARNING in the stage:
> 
> Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that could
> trigger a rather massive flood of messages to the client in a *normal*
> situation. I'm thinking of something like an insert extending a relation some
> time after an immediate restart and encountering a lot of FSM corruption (due
> to its non-crash-safe-ness) during the search for free space and the
> subsequent FSM vacuum.  It might be ok to LOG that, but sending a lot of
> WARNINGs to the client seems not quite right.

Orthogonal to AIO, I do think LOG (or even DEBUG1?) is better for
ZERO_ON_ERROR.  The ZERO_ON_ERROR case also should not use
ERRCODE_DATA_CORRUPTED.  (That errcode shouldn't appear for business as usual.
It should signify wrong or irretrievable query results, essentially.)

For zero_damaged_pages, WARNING seems at least defensible, and
ERRCODE_DATA_CORRUPTED is right.  It wouldn't be the worst thing to change
zero_damaged_pages to LOG and let the complete_shared runner log it, as long
as we release-note that.  It's superuser-only, and the superuser can learn to
check the log.  One typically should use zero_damaged_pages in one session at
a time, so the logs won't be too confusing.

Another thought on complete_shared running on other backends: I wonder if we
should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
other process" or similar, so people wonder less about how "SELECT FROM a" led
to a log message about IO on table "b".

> If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
> then could tell the stager to issue the WARNING. It would add a bit of
> distributed cost, both to callbacks and users of AIO, but it might not be too
> bad.
> 
> 
> > One could simplify things by forcing io_method=sync under ZERO_ON_ERROR ||
> > zero_damaged_pages, perhaps as a short-term approach.
> 
> Yea, that could work.  Perhaps even just for zero_damaged_pages, after
> changing it so that ZERO_ON_ERROR always just LOGs.

Yes.

> Hm, it seems somewhat nasty to have rather different performance
> characteristics when forced to use zero_damaged_pages to recover from a
> problem. Imagine an instance that's configured to use DIO and then needs to
> use zero_damaged_pages to recove from corruption...

True.  I'd be willing to bet high-scale use of zero_damaged_pages is rare.  By
high scale, I mean something like reading a whole large table, as opposed to a
TID scan of the known-problematic range.  That said, people (including me)
expect the emergency tools to be good even if they're used rarely.  You're not
wrong to worry about it.

  123. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T15:26:14Z

    Hi,

On 2025-03-25 06:33:21 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> > On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > > (We may be due for a test mode that does smgrreleaseall() at every
> > > CHECK_FOR_INTERRUPTS()?)
> >
> > I suspect we are. I'm a bit afraid of even trying...
> >
> > ...
> >
> > It's extremely slow - but at least the main regression as well as the aio tests pass!
>
> One less thing!

Unfortunately I'm now doubting the thoroughness of my check - while I made
every CFI() execute smgrreleaseall(), I didn't trigger CFI() in cases where we
trigger it conditionally. E.g. elog(DEBUGN, ...) only executes a CFI if
log_min_messages <= DEBUGN...

I'll try that in a bit.


> > Because the end state varies, depending on the number of previously staged
> > IOs, the IO method and whether batchmode is enabled, I think it's better if
> > the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
> > *not* aligned with an internal state name.  It will just mislead readers to
> > think that there's a deterministic mapping when that does not exist.
>
> That's fair.  Could we provide the mapping in a comment, something like the
> following?

Yes!

I wonder if it should also be duplicated or referenced elsewhere, although I
am not sure where precisely.


> --- a/src/include/storage/aio_internal.h
> +++ b/src/include/storage/aio_internal.h
> @@ -34,5 +34,10 @@
>   * linearly through all states.
>   *
> - * State changes should all go through pgaio_io_update_state().
> + * State changes should all go through pgaio_io_update_state().  Its callers
> + * use these naming conventions:
> + *
> + * - A "start" function (e.g. FileStartReadV()) moves an IO from
> + *   PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
> + *   PGAIO_HS_COMPLETED_LOCAL.
>   */
>  typedef enum PgAioHandleState

One detail I'm not sure about: The above change is correct, but perhaps a bit
misleading, because we can actually go "back" to IDLE. Not sure how to best
phrase that though.


> > That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
> > naming that I just stopped seeing.

I assume you're on board with renaming _io_prep* to _io_start_*?


> > I'll try to think more about this, perhaps I can make myself see your POV
> > more.
>
> > > As the patch stands, LockBufferForCleanup() can succeed when
> > > ConditionalLockBufferForCleanup() would have returned false.
> >
> > That's already true today, right? In master ConditionalLockBufferForCleanup()
> > for temp buffers checks LocalRefCount, whereas LockBufferForCleanup() doesn't.
>
> I'm finding a LocalRefCount check under LockBufferForCleanup:

I guess I should have stopped looking at code / replying before my last email
last night... Not sure how I missed that.



> CheckBufferIsPinnedOnce(Buffer buffer)
> {
> 	if (BufferIsLocal(buffer))
> 	{
> 		if (LocalRefCount[-buffer - 1] != 1)
> 			elog(ERROR, "incorrect local pin count: %d",
> 				 LocalRefCount[-buffer - 1]);
> 	}
> 	else
> 	{
> 		if (GetPrivateRefCount(buffer) != 1)
> 			elog(ERROR, "incorrect local pin count: %d",
> 				 GetPrivateRefCount(buffer));
> 	}
> }

Pretty random orthogonal thought, that I was reminded of by the above code
snippet:

It sure seems we should at some point get rid of LocalRefCount[] and just use
the GetPrivateRefCount() infrastructure for both shared and local buffers.  I
don't think the GetPrivateRefCount() infrastructure cares about
local/non-local, leaving a few asserts aside.  If we do that, and start to use
BM_IO_IN_PROGRESS, combined with ResourceOwnerRememberBufferIO(), the set of
differences between shared and local buffers would be a lot smaller.


> > > Like the comment, I expect it's academic today.  I expect it will stay
> > > academic.  Anything that does a cleanup will start by reading the buffer,
> > > which will resolve any refcnt the AIO subsystems holds for a read.  If there's
> > > an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> > > that.  How about just removing the ConditionalLockBufferForCleanup() changes
> > > or replacing them with a comment (like the present paragraph)?
> >
> > I think we'll need an expanded version of what I suggest once we have writes -
> > but as you say, it shouldn't matter as long as we only have reads. So I think
> > moving the relevant changes, with adjusted caveats, to the bufmgr: write
> > change makes sense.
>
> Moving those changes works for me.  I'm not currently seeing the need under
> writes, but that may get clearer upon reaching those patches.

FWIW, I don't think it's currently worth looking at the write side in detail,
there's enough required changes to make that not necessarily the best use of
your time at this point. At least:


- Write logic needs to be rebased ontop of the patch series to not hit bit
  dirty buffers while IO is going on

  The performance impact of doing the memory copies is rather substantial, as
  on intel memory bandwidth is *the* IO bottleneck even just for the checksum
  computation, without a copy. That makes the memory copy for something like
  bounce buffers hurt really badly.

  And the memory usage of bounce buffers is also really concerning.

  And even without checksums, several filesystems *really* don't like buffers
  getting modified during DIO writes. Which I think would mean we ought to use
  bounce buffers for *all* writes, which would impose a *very* substantial
  overhead (basically removing the benefit of DMA happening off-cpu).


- Right now the sync.c integration with smgr.c/md.c isn't properly safe to use
  in a critical section

  The only reason it doesn't immediately fail is that it's reasonably rare
  that RegisterSyncRequest() fails *and* either:

  - smgropen()->hash_search(HASH_ENTER) decides to resize the hash table, even
    though the lookup is guaranteed to suceed for io_method=worker.

  - an io_method=uring completion is run in a different backend and smgropen()
    needs to build a new entry and thus needs to allocate memory

  For a bit I thought this could be worked around easily enough by not doing
  an smgropen() in mdsyncfiletag(), or adding a "fallible" smgropen() and
  instead just opening the file directly. That actually does kinda solve the
  problem, but only because the memory allocation in PathNameOpenFile()
  uses malloc(), not palloc() and thus doesn't trigger


- I think it requires new lwlock.c infrastructure (as v1 of aio had), to make
  LockBuffer(BUFFER_LOCK_EXCLUSIVE) etc wait in a concurrency safe manner for
  in-progress writes

  I can think of ways to solve this purely in bufmgr.c, but only in ways that
  would cause other problems (e.g. setting BM_IO_IN_PROGRESS before waiting
  for an exclusive lock) and/or expensive.


- My current set of patches doesn't implement bgwriter_flush_after,
  checkpointer_flush_after

  I think that's not too hard to do, it's mainly round tuits.


- temp_file_limit is not respected by aio writes

  I guess that could be ok if AIO writes are only used by checkpointer /
  bgwriter, but we need to figure out a way to deal with that. Perhaps by
  redesigning temp_file_limit, the current implementation seems like rather
  substantial layering violation.


- Too much duplicated code, as there's the aio and non-aio write paths.  That
  might be ok for a bit.


I updated the commit messages of the relevant commits with the above, there
were abbreviated versions of most of the above, but not in enough detail for
anybody but me (and maybe not even that).



> > Do you think it's worth mentioning the above workaround? I'm mildly inclined
> > that not.
>
> Perhaps not in that detail, but perhaps we can rephrase (b) to not imply
> exit+reenter is banned.  Maybe "(b) start another batch (without first exiting
> one)".  It's also fine as-is, though.

I updated it to:

 * b) start another batch (without first exiting batchmode and re-entering
 *    before returning)


> > I'm ok with all of these. In order of preference:
> >
> > 1) READ_STREAM_USE_BATCHING or READ_STREAM_BATCH_OK
> > 2) READ_STREAM_BATCHMODE_AWARE
> > 3) READ_STREAM_CALLBACK_BATCHMODE_AWARE
>
> Same for me.

For now I'll leave it at READ_STREAM_USE_BATCHING, but if Thomas has a
preference I'll go for whatever we have a majority for.

Greetings,

Andres Freund

  124. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T15:33:33Z

    Hi,

On 2025-03-25 17:10:19 +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 2:18 PM Andres Freund <andres@anarazel.de> wrote:
> > Attached v2.12, with the following changes:
> 
> Here's a tiny fixup to make io_concurrency=0 turn on
> READ_BUFFERS_SYNCHRONOUSLY as mooted in a FIXME.  Without this, AIO
> will still run at level 1 even if you asked for 0.  Feel free to
> squash, or ignore and I'll push it later, whatever suits... (tested on
> the tip of your public aio-2 branch).

Thanks! I squashed it into "aio: Basic read_stream adjustments for real AIO"
and updated the commit message to account for that.

Greetings,

Andres Freund

  125. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T15:57:58Z

    Hi,

On 2025-03-25 07:11:20 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> > Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that could
> > trigger a rather massive flood of messages to the client in a *normal*
> > situation. I'm thinking of something like an insert extending a relation some
> > time after an immediate restart and encountering a lot of FSM corruption (due
> > to its non-crash-safe-ness) during the search for free space and the
> > subsequent FSM vacuum.  It might be ok to LOG that, but sending a lot of
> > WARNINGs to the client seems not quite right.
> 
> Orthogonal to AIO, I do think LOG (or even DEBUG1?) is better for
> ZERO_ON_ERROR.  The ZERO_ON_ERROR case also should not use
> ERRCODE_DATA_CORRUPTED.  (That errcode shouldn't appear for business as usual.
> It should signify wrong or irretrievable query results, essentially.)

I strongly agree on the errcode - basically makes it much harder to use the
errcode to trigger alerting. And we don't have any other way to do that...

I'm, obviously, positive on not using WARNING for ZERO_ON_ERROR. I'm neutral
on LOG vs DEBUG1, I can see arguments for either.


> For zero_damaged_pages, WARNING seems at least defensible, and
> ERRCODE_DATA_CORRUPTED is right.  It wouldn't be the worst thing to change
> zero_damaged_pages to LOG and let the complete_shared runner log it, as long
> as we release-note that.  It's superuser-only, and the superuser can learn to
> check the log.  One typically should use zero_damaged_pages in one session at
> a time, so the logs won't be too confusing.

It's obviously tempting to go for that, I'm somewhat undecided what the best
way is right now.  There might be a compromise, see below:


> > If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
> > then could tell the stager to issue the WARNING. It would add a bit of
> > distributed cost, both to callbacks and users of AIO, but it might not be too
> > bad.

FWIW, I prototyped this, it's not hard.

But it can't replace the current WARNING with 100% fidelity: If we read 60
blocks in a single smgrreadv, we today would would emit 60 WARNINGs.  But we
can't encode that many block offset in single PgAioResult, there's not enough
space, and enlarging it far enough doesn't seem to make sense either.


What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
with that warning saying that there were N zeroed blocks in a read from block
N to block Y and a HINT saying that there are more details in the server log.



> Another thought on complete_shared running on other backends: I wonder if we
> should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> other process" or similar, so people wonder less about how "SELECT FROM a" led
> to a log message about IO on table "b".

I've been wondering about that as well, and yes, we probably should.

I'd add the pid of the backend that started the IO to the message - although
need to check whether we're trying to keep PIDs of other processes from
unprivileged users.

I think we probably should add a similar, but not equivalent, context in io
workers. Maybe "I/O worker executing I/O on behalf of process %d".

Greetings,

Andres Freund

  126. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T15:58:08Z

    On Tue, Mar 25, 2025 at 11:26:14AM -0400, Andres Freund wrote:
> On 2025-03-25 06:33:21 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> > > On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > > > (We may be due for a test mode that does smgrreleaseall() at every
> > > > CHECK_FOR_INTERRUPTS()?)
> > >
> > > I suspect we are. I'm a bit afraid of even trying...
> > >
> > > ...
> > >
> > > It's extremely slow - but at least the main regression as well as the aio tests pass!
> >
> > One less thing!
> 
> Unfortunately I'm now doubting the thoroughness of my check - while I made
> every CFI() execute smgrreleaseall(), I didn't trigger CFI() in cases where we
> trigger it conditionally. E.g. elog(DEBUGN, ...) only executes a CFI if
> log_min_messages <= DEBUGN...
> 
> I'll try that in a bit.

While having nagging thoughts that we might be releasing FDs before io_uring
gets them into kernel custody, I tried this hack to maximize FD turnover:

static void
ReleaseLruFiles(void)
{
#if 0
	while (nfile + numAllocatedDescs + numExternalFDs >= max_safe_fds)
	{
		if (!ReleaseLruFile())
			break;
	}
#else
	while (ReleaseLruFile())
		;
#endif
}

"make check" with default settings (io_method=worker) passes, but
io_method=io_uring in the TEMP_CONFIG file got different diffs in each of two
runs.  s/#if 0/#if 1/ (restore normal FD turnover) removes the failures.
Here's the richer of the two diffs:

diff -U3 src/test/regress/expected/sanity_check.out src/test/regress/results/sanity_check.out
--- src/test/regress/expected/sanity_check.out  2024-10-24 12:43:25.741817594 -0700
+++ src/test/regress/results/sanity_check.out   2025-03-25 08:27:44.875151566 -0700
@@ -1,4 +1,7 @@
 VACUUM;
+ERROR:  index "pg_enum_oid_index" contains corrupted page at block 2
+HINT:  Please REINDEX it.
+CONTEXT:  while vacuuming index "pg_enum_oid_index" of relation "pg_catalog.pg_enum"
 --
 -- Sanity check: every system catalog that has OIDs should have
 -- a unique index on OID.  This ensures that the OIDs will be unique,
diff -U3 src/test/regress/expected/oidjoins.out src/test/regress/results/oidjoins.out
--- src/test/regress/expected/oidjoins.out      2023-07-06 19:58:07.686364439 -0700
+++ src/test/regress/results/oidjoins.out       2025-03-25 08:28:02.584335458 -0700
@@ -233,6 +233,8 @@
 NOTICE:  checking pg_policy {polrelid} => pg_class {oid}
 NOTICE:  checking pg_policy {polroles} => pg_authid {oid}
 NOTICE:  checking pg_default_acl {defaclrole} => pg_authid {oid}
+WARNING:  FK VIOLATION IN pg_default_acl({defaclrole}): ("(1,5)",0)
+WARNING:  FK VIOLATION IN pg_default_acl({defaclrole}): ("(1,7)",402654464)
 NOTICE:  checking pg_default_acl {defaclnamespace} => pg_namespace {oid}
 NOTICE:  checking pg_init_privs {classoid} => pg_class {oid}
 NOTICE:  checking pg_seclabel {classoid} => pg_class {oid}

> > > Because the end state varies, depending on the number of previously staged
> > > IOs, the IO method and whether batchmode is enabled, I think it's better if
> > > the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
> > > *not* aligned with an internal state name.  It will just mislead readers to
> > > think that there's a deterministic mapping when that does not exist.
> >
> > That's fair.  Could we provide the mapping in a comment, something like the
> > following?
> 
> Yes!
> 
> I wonder if it should also be duplicated or referenced elsewhere, although I
> am not sure where precisely.

I considered the README.md also, but adding that wasn't an obvious win.

> > --- a/src/include/storage/aio_internal.h
> > +++ b/src/include/storage/aio_internal.h
> > @@ -34,5 +34,10 @@
> >   * linearly through all states.
> >   *
> > - * State changes should all go through pgaio_io_update_state().
> > + * State changes should all go through pgaio_io_update_state().  Its callers
> > + * use these naming conventions:
> > + *
> > + * - A "start" function (e.g. FileStartReadV()) moves an IO from
> > + *   PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
> > + *   PGAIO_HS_COMPLETED_LOCAL.
> >   */
> >  typedef enum PgAioHandleState
> 
> One detail I'm not sure about: The above change is correct, but perhaps a bit
> misleading, because we can actually go "back" to IDLE. Not sure how to best
> phrase that though.

Not sure either.  Maybe the above could change to "to PGAIO_HS_STAGED or any
subsequent state" and the comment at PGAIO_HS_STAGED could say like "Once in
this state, concurrent activity could move the IO all the way to
PGAIO_HS_COMPLETED_LOCAL and recycle it back to IDLE."

> > > That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
> > > naming that I just stopped seeing.
> 
> I assume you're on board with renaming _io_prep* to _io_start_*?

Yes.

> > > I'll try to think more about this, perhaps I can make myself see your POV
> > > more.

> > CheckBufferIsPinnedOnce(Buffer buffer)
> > {
> > 	if (BufferIsLocal(buffer))
> > 	{
> > 		if (LocalRefCount[-buffer - 1] != 1)
> > 			elog(ERROR, "incorrect local pin count: %d",
> > 				 LocalRefCount[-buffer - 1]);
> > 	}
> > 	else
> > 	{
> > 		if (GetPrivateRefCount(buffer) != 1)
> > 			elog(ERROR, "incorrect local pin count: %d",
> > 				 GetPrivateRefCount(buffer));
> > 	}
> > }
> 
> Pretty random orthogonal thought, that I was reminded of by the above code
> snippet:
> 
> It sure seems we should at some point get rid of LocalRefCount[] and just use
> the GetPrivateRefCount() infrastructure for both shared and local buffers.  I
> don't think the GetPrivateRefCount() infrastructure cares about
> local/non-local, leaving a few asserts aside.  If we do that, and start to use
> BM_IO_IN_PROGRESS, combined with ResourceOwnerRememberBufferIO(), the set of
> differences between shared and local buffers would be a lot smaller.

That sounds promising.

> > > > Like the comment, I expect it's academic today.  I expect it will stay
> > > > academic.  Anything that does a cleanup will start by reading the buffer,
> > > > which will resolve any refcnt the AIO subsystems holds for a read.  If there's
> > > > an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> > > > that.  How about just removing the ConditionalLockBufferForCleanup() changes
> > > > or replacing them with a comment (like the present paragraph)?
> > >
> > > I think we'll need an expanded version of what I suggest once we have writes -
> > > but as you say, it shouldn't matter as long as we only have reads. So I think
> > > moving the relevant changes, with adjusted caveats, to the bufmgr: write
> > > change makes sense.
> >
> > Moving those changes works for me.  I'm not currently seeing the need under
> > writes, but that may get clearer upon reaching those patches.
> 
> FWIW, I don't think it's currently worth looking at the write side in detail,

Got it.  (I meant I didn't see a first-principles need, not that I had deduced
lack of need from a specific writes implementation.)

> > > Do you think it's worth mentioning the above workaround? I'm mildly inclined
> > > that not.
> >
> > Perhaps not in that detail, but perhaps we can rephrase (b) to not imply
> > exit+reenter is banned.  Maybe "(b) start another batch (without first exiting
> > one)".  It's also fine as-is, though.
> 
> I updated it to:
> 
>  * b) start another batch (without first exiting batchmode and re-entering
>  *    before returning)

That's good.

  127. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T16:15:43Z

    On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> On 2025-03-25 07:11:20 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> > > If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
> > > then could tell the stager to issue the WARNING. It would add a bit of
> > > distributed cost, both to callbacks and users of AIO, but it might not be too
> > > bad.
> 
> FWIW, I prototyped this, it's not hard.
> 
> But it can't replace the current WARNING with 100% fidelity: If we read 60
> blocks in a single smgrreadv, we today would would emit 60 WARNINGs.  But we
> can't encode that many block offset in single PgAioResult, there's not enough
> space, and enlarging it far enough doesn't seem to make sense either.
> 
> 
> What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
> with that warning saying that there were N zeroed blocks in a read from block
> N to block Y and a HINT saying that there are more details in the server log.

Sounds fine.

> > Another thought on complete_shared running on other backends: I wonder if we
> > should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> > other process" or similar, so people wonder less about how "SELECT FROM a" led
> > to a log message about IO on table "b".
> 
> I've been wondering about that as well, and yes, we probably should.
> 
> I'd add the pid of the backend that started the IO to the message - although
> need to check whether we're trying to keep PIDs of other processes from
> unprivileged users.

We don't.

> I think we probably should add a similar, but not equivalent, context in io
> workers. Maybe "I/O worker executing I/O on behalf of process %d".

Sounds good.

  128. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T18:58:37Z

    Hi,

On 2025-03-25 08:58:08 -0700, Noah Misch wrote:
> While having nagging thoughts that we might be releasing FDs before io_uring
> gets them into kernel custody, I tried this hack to maximize FD turnover:
> 
> static void
> ReleaseLruFiles(void)
> {
> #if 0
> 	while (nfile + numAllocatedDescs + numExternalFDs >= max_safe_fds)
> 	{
> 		if (!ReleaseLruFile())
> 			break;
> 	}
> #else
> 	while (ReleaseLruFile())
> 		;
> #endif
> }
> 
> "make check" with default settings (io_method=worker) passes, but
> io_method=io_uring in the TEMP_CONFIG file got different diffs in each of two
> runs.  s/#if 0/#if 1/ (restore normal FD turnover) removes the failures.
> Here's the richer of the two diffs:

Yikes. That's a very good catch.

I spent a bit of time debugging this. I think I see what's going on - it turns
out that the kernel does *not* open the FDs during io_uring_enter() if
IOSQE_ASYNC is specified [1].  Which we do add heuristically, in an attempt to
avoid a small but measurable slowdown for sequential scans that are fully
buffered (c.f. pgaio_uring_submit()).  If I disable that heuristic, your patch
above passes all tests here.


I don't know if that's an intentional or unintentional behavioral difference.

There are 2 1/2 ways around this:

1) Stop using IOSQE_ASYNC heuristic
2a) Wait for all in-flight IOs when any FD gets closed
2b) Wait for all in-flight IOs using FD when it gets closed

Given that we have clear evidence that io_uring doesn't completely support
closing FDs while IOs are in flight, be it a bug or intentional, it seems
clearly better to go for 2a or 2b.

Greetings,

Andres Freund


[1] Instead files are opened when the queue entry is being worked on
    instead. Interestingly that only happens when the IO is *explicitly*
    requested to be executed in the workqueue with IOSQE_ASYNC, not when it's
    put there because it couldn't be done in a non-blocking way.

  129. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T19:39:56Z

    On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> On 2025-03-25 08:58:08 -0700, Noah Misch wrote:
> > While having nagging thoughts that we might be releasing FDs before io_uring
> > gets them into kernel custody, I tried this hack to maximize FD turnover:
> > 
> > static void
> > ReleaseLruFiles(void)
> > {
> > #if 0
> > 	while (nfile + numAllocatedDescs + numExternalFDs >= max_safe_fds)
> > 	{
> > 		if (!ReleaseLruFile())
> > 			break;
> > 	}
> > #else
> > 	while (ReleaseLruFile())
> > 		;
> > #endif
> > }
> > 
> > "make check" with default settings (io_method=worker) passes, but
> > io_method=io_uring in the TEMP_CONFIG file got different diffs in each of two
> > runs.  s/#if 0/#if 1/ (restore normal FD turnover) removes the failures.
> > Here's the richer of the two diffs:
> 
> Yikes. That's a very good catch.
> 
> I spent a bit of time debugging this. I think I see what's going on - it turns
> out that the kernel does *not* open the FDs during io_uring_enter() if
> IOSQE_ASYNC is specified [1].  Which we do add heuristically, in an attempt to
> avoid a small but measurable slowdown for sequential scans that are fully
> buffered (c.f. pgaio_uring_submit()).  If I disable that heuristic, your patch
> above passes all tests here.

Same result here.  As an additional data point, I tried adding this so every
reopen gets a new FD number (leaks FDs wildly):

--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1304,5 +1304,5 @@ LruDelete(File file)
 	 * to leak the FD than to mess up our internal state.
 	 */
-	if (close(vfdP->fd) != 0)
+	if (dup2(2, vfdP->fd) != vfdP->fd)
 		elog(vfdP->fdstate & FD_TEMP_FILE_LIMIT ? LOG : data_sync_elevel(LOG),
 			 "could not close file \"%s\": %m", vfdP->fileName);

The same "make check" w/ TEMP_CONFIG io_method=io_uring passes with the
combination of that and the max-turnover change to ReleaseLruFiles().

> I don't know if that's an intentional or unintentional behavioral difference.
> 
> There are 2 1/2 ways around this:
> 
> 1) Stop using IOSQE_ASYNC heuristic
> 2a) Wait for all in-flight IOs when any FD gets closed
> 2b) Wait for all in-flight IOs using FD when it gets closed
> 
> Given that we have clear evidence that io_uring doesn't completely support
> closing FDs while IOs are in flight, be it a bug or intentional, it seems
> clearly better to go for 2a or 2b.

Agreed.  If a workload spends significant time on fd.c closing files, I
suspect that workload already won't have impressive benchmark numbers.
Performance-seeking workloads will already want to tune FD usage high enough
to keep FDs long-lived.  So (1) clearly loses, and neither (2a) nor (2b)
clearly beats the other.  I'd try (2b) first but, if complicated, quickly
abandon it in favor of (2a).  What other considerations could be important?

  130. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T20:07:35Z

    Hi,

On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > I don't know if that's an intentional or unintentional behavioral difference.
> > 
> > There are 2 1/2 ways around this:
> > 
> > 1) Stop using IOSQE_ASYNC heuristic
> > 2a) Wait for all in-flight IOs when any FD gets closed
> > 2b) Wait for all in-flight IOs using FD when it gets closed
> > 
> > Given that we have clear evidence that io_uring doesn't completely support
> > closing FDs while IOs are in flight, be it a bug or intentional, it seems
> > clearly better to go for 2a or 2b.
> 
> Agreed.  If a workload spends significant time on fd.c closing files, I
> suspect that workload already won't have impressive benchmark numbers.
> Performance-seeking workloads will already want to tune FD usage high enough
> to keep FDs long-lived.  So (1) clearly loses, and neither (2a) nor (2b)
> clearly beats the other.  I'd try (2b) first but, if complicated, quickly
> abandon it in favor of (2a).  What other considerations could be important?

The only other consideration I can think of is whether this should happen for
all io_methods or not.

I'm inclined to do it via a bool in IoMethodOps, but I guess one could argue
it's a bit weird to have a bool in a struct called *Ops.

Greetings,

Andres Freund

  131. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T20:18:50Z

    On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote:
> On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > > There are 2 1/2 ways around this:
> > > 
> > > 1) Stop using IOSQE_ASYNC heuristic
> > > 2a) Wait for all in-flight IOs when any FD gets closed
> > > 2b) Wait for all in-flight IOs using FD when it gets closed
> > > 
> > > Given that we have clear evidence that io_uring doesn't completely support
> > > closing FDs while IOs are in flight, be it a bug or intentional, it seems
> > > clearly better to go for 2a or 2b.
> > 
> > Agreed.  If a workload spends significant time on fd.c closing files, I
> > suspect that workload already won't have impressive benchmark numbers.
> > Performance-seeking workloads will already want to tune FD usage high enough
> > to keep FDs long-lived.  So (1) clearly loses, and neither (2a) nor (2b)
> > clearly beats the other.  I'd try (2b) first but, if complicated, quickly
> > abandon it in favor of (2a).  What other considerations could be important?
> 
> The only other consideration I can think of is whether this should happen for
> all io_methods or not.

Either way is fine, I think.

> I'm inclined to do it via a bool in IoMethodOps, but I guess one could argue
> it's a bit weird to have a bool in a struct called *Ops.

That wouldn't bother me.  IndexAmRoutine has many bools, and "Ops" is
basically a synonym of "Routine".

  132. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-25T20:56:53Z

    Hi,

On 2025-03-25 13:18:50 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote:
> > On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> > > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > > > There are 2 1/2 ways around this:
> > > > 
> > > > 1) Stop using IOSQE_ASYNC heuristic
> > > > 2a) Wait for all in-flight IOs when any FD gets closed
> > > > 2b) Wait for all in-flight IOs using FD when it gets closed
> > > > 
> > > > Given that we have clear evidence that io_uring doesn't completely support
> > > > closing FDs while IOs are in flight, be it a bug or intentional, it seems
> > > > clearly better to go for 2a or 2b.
> > > 
> > > Agreed.  If a workload spends significant time on fd.c closing files, I
> > > suspect that workload already won't have impressive benchmark numbers.
> > > Performance-seeking workloads will already want to tune FD usage high enough
> > > to keep FDs long-lived.  So (1) clearly loses, and neither (2a) nor (2b)
> > > clearly beats the other.  I'd try (2b) first but, if complicated, quickly
> > > abandon it in favor of (2a).  What other considerations could be important?
> > 
> > The only other consideration I can think of is whether this should happen for
> > all io_methods or not.
> 
> Either way is fine, I think.

Here's a draft incremental patch (attached as a .fixup to avoid triggering
cfbot) implementing 2b).


> > I'm inclined to do it via a bool in IoMethodOps, but I guess one could argue
> > it's a bit weird to have a bool in a struct called *Ops.
> 
> That wouldn't bother me.  IndexAmRoutine has many bools, and "Ops" is
> basically a synonym of "Routine".

Cool. Done that way.

The repeated-iteration approach taken in pgaio_closing_fd() isn't the
prettiest, but it's hard to to imagine that ever being a noticeable.


This survives a testrun where I use your torture patch and where I force all
IOs to use ASYNC. Previously that did not get very far.  I also did verify
that, if I allow a small number of FDs, we do not wrongly wait for all IOs.

Greetings,

Andres Freund

  133. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-25T22:44:40Z

    On Tue, Mar 25, 2025 at 04:56:53PM -0400, Andres Freund wrote:
> The repeated-iteration approach taken in pgaio_closing_fd() isn't the
> prettiest, but it's hard to to imagine that ever being a noticeable.

Yep.  I've reviewed the fixup code, and it looks all good.

> This survives a testrun where I use your torture patch and where I force all
> IOs to use ASYNC. Previously that did not get very far.  I also did verify
> that, if I allow a small number of FDs, we do not wrongly wait for all IOs.

I, too, see the test diffs gone.

  134. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-26T00:17:17Z

    Hi,

On 2025-03-25 09:15:43 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> > FWIW, I prototyped this, it's not hard.
> > 
> > But it can't replace the current WARNING with 100% fidelity: If we read 60
> > blocks in a single smgrreadv, we today would would emit 60 WARNINGs.  But we
> > can't encode that many block offset in single PgAioResult, there's not enough
> > space, and enlarging it far enough doesn't seem to make sense either.
> > 
> > 
> > What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
> > with that warning saying that there were N zeroed blocks in a read from block
> > N to block Y and a HINT saying that there are more details in the server log.

It should probably be DETAIL, not HINT...


> Sounds fine.

I got that working. To make it readable, it required changing division of
labor between buffer_readv_complete() and buffer_readv_complete_one() a bit,
but I think it's actually easier to understand now.

Still need to beef up the test infrastructure a bit to make the multi-block
cases more easily testable.


Could use some input on the framing of the message/detail. Right now it's:

ERROR:  invalid page in block 8 of relation base/5/16417
DETAIL: Read of 8 blocks, starting at block 7, 1 other pages in the same read are invalid.

But that doesn't seem great. Maybe:

DETAIL: Read of blocks 7..14, 1 other pages in the same read were also invalid.

But that still isn't really a sentence.

Greetings,

Andres Freund

  135. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-26T00:19:15Z

    On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> Attached v2.12, with the following changes:

> TODO:

>   Wonder if it's worth adding some coverage for when checksums are disabled?
>   Probably not necessary?

Probably not necessary, agreed.  Orthogonal to AIO, it's likely worth a CI
"SPECIAL" and/or buildfarm animal that runs all tests w/ checksums disabled.


> Subject: [PATCH v2.12 01/28] aio: Be more paranoid about interrupts

Ready for commit


> Subject: [PATCH v2.12 02/28] aio: Pass result of local callbacks to
>  ->report_return

Ready for commit w/ up to one cosmetic change:

> @@ -296,7 +299,9 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
>  
>  	/*
>  	 * Note that we don't save the result in ioh->distilled_result, the local
> -	 * callback's result should not ever matter to other waiters.
> +	 * callback's result should not ever matter to other waiters. However, the
> +	 * local backend does care, so we return the result as modified by local
> +	 * callbacks, which then can be passed to ioh->report_return->result.
>  	 */
>  	pgaio_debug_io(DEBUG3, ioh,
>  				   "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",

Should this debug message remove the word "distilled", since this commit
solidifies distilled_result as referring to the complete_shared result?


> Subject: [PATCH v2.12 03/28] aio: Add liburing dependency

Ready for commit


> Subject: [PATCH v2.12 04/28] aio: Add io_method=io_uring

Ready for commit w/ open_fd.fixup


> Subject: [PATCH v2.12 05/28] aio: Implement support for reads in smgr/md/fd

Ready for commit w/ up to two cosmetic changes:

> +/*
> + * AIO error reporting callback for mdstartreadv().
> + *
> + * Errors are encoded as follows:
> + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0

I recommend replacing "errno != 0" with either "that errno" or "errno ==
error_data".

Second, the aio_internal.h comment changes discussed in
postgr.es/m/20250325155808.f7.nmisch@google.com and earlier.


> Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design

Ready for commit

(This and the previous patch have three spots that would change with the
s/prep/start/ renames.  No opinion on whether to rename before or rename
after.)


> Subject: [PATCH v2.12 07/28] localbuf: Track pincount in BufferDesc as well

The plan here looks good:
postgr.es/m/dbeeaize47y7esifdrinpa2l7cqqb67k72exvuf3appyxywjnc@7bt76mozhcy2


> Subject: [PATCH v2.12 08/28] bufmgr: Implement AIO read support

See review here and later discussion:
postgr.es/m/20250325022037.91.nmisch@google.com


> Subject: [PATCH v2.12 09/28] bufmgr: Use AIO in StartReadBuffers()

Ready for commit after a batch of small things, all but one of which have no
implications beyond code cosmetics.  This is my first comprehensive review of
this patch.  I like the test coverage (by the end of the patch series).  For
anyone else following, I found "diff -w" helpful for the bufmgr.c changes.
That's because a key part is former WaitReadBuffers() code moving up an
indentation level to its home in new subroutine AsyncReadBuffers().

>  	Assert(*nblocks == 1 || allow_forwarding);
>  	Assert(*nblocks > 0);
>  	Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
> +	Assert(*nblocks == 1 || allow_forwarding);

Duplicates the assert three lines back.

> +		nblocks = aio_ret->result.result;
> +
> +		elog(DEBUG3, "partial read, will retry");
> +
> +	}
> +	else if (aio_ret->result.status == PGAIO_RS_ERROR)
> +	{
> +		pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
> +		nblocks = 0;			/* silence compiler */
> +	}
>  
>  	Assert(nblocks > 0);
>  	Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
>  
> +	operation->nblocks_done += nblocks;

I struggled somewhat from the variety of "nblocks" variables: this local
nblocks, operation->nblocks, actual_nblocks, and *nblocks in/out parameters of
some functions.  No one of them is clearly wrong to use the name, and some of
these names are preexisting.  That said, if you see opportunities to push in
the direction of more-specific names, I'd welcome it.

For example, this local variable could become add_to_nblocks_done instead.

> +		AsyncReadBuffers(operation, &nblocks);

I suggest renaming s/nblocks/ignored_nblocks_progress/ here.

> +	 * If we need to wait for IO before we can get a handle, submit already
> +	 * staged IO first, so that other backends don't need to wait. There

s/already staged/already-staged/.  Normally I'd skip this as nitpicking, but I
misread this particular sentence twice, as "submit" being the subject that
"staged" something.  (It's still nitpicking, alas.)

>  		/*
>  		 * How many neighboring-on-disk blocks can we scatter-read into other
>  		 * buffers at the same time?  In this case we don't wait if we see an
> -		 * I/O already in progress.  We already hold BM_IO_IN_PROGRESS for the
> +		 * I/O already in progress.  We already set BM_IO_IN_PROGRESS for the
>  		 * head block, so we should get on with that I/O as soon as possible.
> -		 * We'll come back to this block again, above.
> +		 *
> +		 * We'll come back to this block in the next call to
> +		 * StartReadBuffers() -> AsyncReadBuffers().

Did this mean to say "WaitReadBuffers() -> AsyncReadBuffers()"?  I'm guessing
so, since WaitReadBuffers() is the one that loops.  It might be referring to
read_stream_start_pending_read()'s next StartReadBuffers(), though.

I think this could just delete the last sentence.  The function header comment
already mentions the possibility of reading a subset of the request.  This
spot doesn't need to detail how the higher layers come back to here.

> +		smgrstartreadv(ioh, operation->smgr, forknum,
> +					   blocknum + nblocks_done,
> +					   io_pages, io_buffers_len);
> +		pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
> +								io_start, 1, *nblocks_progress * BLCKSZ);

We don't assign *nblocks_progress until lower in the function, so I think
"io_buffers_len" should replace "*nblocks_progress" here.  (This is my only
non-cosmetic comment on this patch.)


> Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO

(Still reviewing this and later patches, but incidental observations follow.)


> Subject: [PATCH v2.12 16/28] aio: Add test_aio module

> +use List::Util qw(sample);

sample() is new in 2020:
https://metacpan.org/release/PEVANS/Scalar-List-Utils-1.68/source/Changes#L100

Hence, I'd expect some buildfarm failures.  I'd try to use shuffle(), then
take the first N elements.

> +++ b/src/test/modules/test_aio/test_aio.c
> @@ -0,0 +1,712 @@
> +/*-------------------------------------------------------------------------
> + *
> + * delay_execution.c
> + *		Test module to allow delay between parsing and execution of a query.
> + *
> + * The delay is implemented by taking and immediately releasing a specified
> + * advisory lock.  If another process has previously taken that lock, the
> + * current process will be blocked until the lock is released; otherwise,
> + * there's no effect.  This allows an isolationtester script to reliably
> + * test behaviors where some specified action happens in another backend
> + * between parsing and execution of any desired query.
> + *
> + * Copyright (c) 2020-2025, PostgreSQL Global Development Group
> + *
> + * IDENTIFICATION
> + *	  src/test/modules/test_aio/test_aio.c

To elaborate on my last review, the entire header comment was a copy from
delay_execution.c.  v2.12 fixes the IDENTIFICATION, but the rest needs
updates.

Thanks,
nm

  136. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-26T00:43:28Z

    On Tue, Mar 25, 2025 at 08:17:17PM -0400, Andres Freund wrote:
> On 2025-03-25 09:15:43 -0700, Noah Misch wrote:
> > On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> > > FWIW, I prototyped this, it's not hard.
> > > 
> > > But it can't replace the current WARNING with 100% fidelity: If we read 60
> > > blocks in a single smgrreadv, we today would would emit 60 WARNINGs.  But we
> > > can't encode that many block offset in single PgAioResult, there's not enough
> > > space, and enlarging it far enough doesn't seem to make sense either.
> > > 
> > > 
> > > What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
> > > with that warning saying that there were N zeroed blocks in a read from block
> > > N to block Y and a HINT saying that there are more details in the server log.
> 
> It should probably be DETAIL, not HINT...

Either is fine with me.  I would go for HINT if referring to the server log,
given the precedent of errhint("See server log for query details.").  DETAIL
fits for block counts, though:

> Could use some input on the framing of the message/detail. Right now it's:
> 
> ERROR:  invalid page in block 8 of relation base/5/16417
> DETAIL: Read of 8 blocks, starting at block 7, 1 other pages in the same read are invalid.
> 
> But that doesn't seem great. Maybe:
> 
> DETAIL: Read of blocks 7..14, 1 other pages in the same read were also invalid.
> 
> But that still isn't really a sentence.

How about this for the multi-page case:

WARNING: zeroing out %u invalid pages among blocks %u..%u of relation %s
DETAIL:  Block %u held first invalid page.
HINT: See server log for the other %u invalid blocks.


For the one-page case, the old message can stay:

WARNING:  invalid page in block %u of relation %s; zeroing out page

  137. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-26T18:31:02Z

    I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr:
Comment fixes", the last patch before write support.
postgr.es/m/20250326001915.bc.nmisch@google.com covered patches 1-9, and this
email covers patches 10-17.  All remaining review comments are minor, so I've
marked the commitfest entry Ready for Committer.  If there's anything you'd
like re-reviewed before you commit it, feel free to bring it to my attention.
Thanks for getting the feature to this stage!

On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO

> @@ -631,6 +637,9 @@ read_stream_begin_impl(int flags,
>  	 * For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
>  	 * above.  If we had real asynchronous I/O we might need a slightly
>  	 * different definition.
> +	 *
> +	 * FIXME: Not sure what different definition we would need? I guess we
> +	 * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?

I think we don't need a different definition.  max_ios comes from
effective_io_concurrency and similar settings.  The above comment's definition
of max_ios=0 matches that GUC's documented behavior:

         The allowed range is
         <literal>1</literal> to <literal>1000</literal>, or
         <literal>0</literal> to disable issuance of asynchronous I/O requests.

I'll guess the comment meant that "advice disabled" is a no-op for AIO, so we
could reasonably argue to have effective_io_concurrency=0 distinguish itself
from effective_io_concurrency=1 in some different way for AIO.  Equally,
there's no hurry to use that freedom to distinguish them.


> Subject: [PATCH v2.12 11/28] read_stream: Introduce and use optional batchmode
>  support

> This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
> uses it where appropriate.

I'd also use the new flag on the read_stream_begin_smgr_relation() call in
RelationCopyStorageUsingBuffer().  It uses block_range_read_stream_cb, and
other streams of that callback rightly use the flag.

> + * b) directly or indirectly start another batch pgaio_enter_batchmode()

Needs new wording from end of postgr.es/m/20250325155808.f7.nmisch@google.com


> Subject: [PATCH v2.12 12/28] docs: Reframe track_io_timing related docs as
>  wait time


> Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems

Consider also updating this comment to stop focusing on prefetch; I think
changing that aligns with the patch's other changes:

/*
 * How many buffers PrefetchBuffer callers should try to stay ahead of their
 * ReadBuffer calls by.  Zero means "never prefetch".  This value is only used
 * for buffers not belonging to tablespaces that have their
 * effective_io_concurrency parameter set.
 */
int			effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;

> -#io_combine_limit = 128kB		# usually 1-128 blocks (depends on OS)
> +#io_combine_limit = 128kB		# usually 1-32 blocks (depends on OS)

I think "usually 1-128" remains right given:

GUC_UNIT_BLOCKS
#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
#define PG_IOV_MAX Min(IOV_MAX, 128)

> -         On systems without prefetch advice support, attempting to configure
> -         any value other than <literal>0</literal> will error out.
> +         On systems with prefetch advice support,
> +         <varname>effective_io_concurrency</varname> also controls the prefetch distance.

Wrap the last line.


> Subject: [PATCH v2.12 14/28] docs: Add acronym and glossary entries for I/O
>  and AIO

> These could use a lot more polish.

To me, it's fine as-is.

> I did not actually reference the new entries yet, because I don't really
> understand what our policy for that is.

I haven't seen much of a policy on that.


> Subject: [PATCH v2.12 15/28] aio: Add pg_aios view
> +retry:
> +
> +		/*
> +		 * There is no lock that could prevent the state of the IO to advance
> +		 * concurrently - and we don't want to introduce one, as that would
> +		 * introduce atomics into a very common path. Instead we
> +		 *
> +		 * 1) Determine the state + generation of the IO.
> +		 *
> +		 * 2) Copy the IO to local memory.
> +		 *
> +		 * 3) Check if state or generation of the IO changed. If the state
> +		 * changed, retry, if the generation changed don't display the IO.
> +		 */
> +
> +		/* 1) from above */
> +		start_generation = live_ioh->generation;
> +		pg_read_barrier();

Based on the "really started after this function was called" and "no risk of a
livelock here" comments below, I think "retry:"  should be here.  We don't
want to livelock in the form of chasing ever-growing start_generation numbers.

> +		/*
> +		 * The IO completed and a new one was started with the same ID. Don't
> +		 * display it - it really started after this function was called.
> +		 * There be a risk of a livelock if we just retried endlessly, if IOs
> +		 * complete very quickly.
> +		 */
> +		if (live_ioh->generation != start_generation)
> +			continue;
> +
> +		/*
> +		 * The IOs state changed while we were "rendering" it. Just start from

s/IOs/IO's/

> +		 * scratch. There's no risk of a livelock here, as an IO has a limited
> +		 * sets of states it can be in, and state changes go only in a single
> +		 * direction.
> +		 */
> +		if (live_ioh->state != start_state)
> +			goto retry;

> +      <entry role="catalog_table_entry"><para role="column_definition">
> +       <structfield>target</structfield> <type>text</type>
> +      </para>
> +      <para>
> +       What kind of object is the I/O targeting:
> +       <itemizedlist spacing="compact">
> +        <listitem>
> +         <para>
> +          <literal>smgr</literal>, I/O on postgres relations

s/postgres relations/relations/ since SGML docs don't use the term "postgres"
that way.


> Subject: [PATCH v2.12 16/28] aio: Add test_aio module

> --- a/src/test/modules/meson.build
> +++ b/src/test/modules/meson.build
> @@ -1,5 +1,6 @@
>  # Copyright (c) 2022-2025, PostgreSQL Global Development Group
>  
> +subdir('test_aio')
>  subdir('brin')

List is alphabetized; please preserve that.

> +++ b/src/test/modules/test_aio/Makefile
> @@ -0,0 +1,26 @@
> +# src/test/modules/delay_execution/Makefile

Update filename in comment.

> +++ b/src/test/modules/test_aio/meson.build
> @@ -0,0 +1,37 @@
> +# Copyright (c) 2022-2024, PostgreSQL Global Development Group

s/2024/2025/

> --- /dev/null
> +++ b/src/test/modules/test_aio/t/001_aio.pl

s/ {4}/\t/g on this file.  It's mostly \t now, with some exceptions.

> +	test_inject_worker('worker', $node_worker);

What do we expect to happen if autovacuum or checkpointer runs one of these
injection points?  I'm guessing it would at most make that process fail
without affecting the test outcome.  If so, that's fine.

> +		$waitfor,);

s/,//

> +	# normal handle use
> +	psql_like($io_method, $psql, "handle_get_release()",
> +		qq(SELECT handle_get_release()),
> +		qr/^$/, qr/^$/);
> +
> +	# should error out, API violation
> +	psql_like($io_method, $psql, "handle_get_twice()",
> +		qq(SELECT handle_get_release()),
> +		qr/^$/, qr/^$/);

Last two lines are a clone of the previous psql_like() call.  I guess this
wants to instead call handle_get_twice() and check for some stderr.

> +			"read_rel_block_ll() of $tblname page",

What does "_ll" stand for?

> +	# Issue IO without waiting for completion, then exit
> +	$psql_a->query_safe(
> +		qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
> +	$psql_a->reconnect_and_clear();
> +
> +	# Check that another backend can read the relevant block
> +	psql_like(
> +		$io_method,
> +		$psql_b,
> +		"completing read started by exited backend",

I think the exiting backend's pgaio_shutdown() completed it.

> +sub test_inject

This deserves a brief comment on the behaviors being tested, like the previous
functions have.  It seems to be about short reads and hard failures like EIO.


> Subject: [PATCH v2.12 17/28] aio, bufmgr: Comment fixes

  138. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-26T20:33:49Z

    Hi,

On 2025-03-25 17:19:15 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > @@ -296,7 +299,9 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
> >  
> >  	/*
> >  	 * Note that we don't save the result in ioh->distilled_result, the local
> > -	 * callback's result should not ever matter to other waiters.
> > +	 * callback's result should not ever matter to other waiters. However, the
> > +	 * local backend does care, so we return the result as modified by local
> > +	 * callbacks, which then can be passed to ioh->report_return->result.
> >  	 */
> >  	pgaio_debug_io(DEBUG3, ioh,
> >  				   "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
> 
> Should this debug message remove the word "distilled", since this commit
> solidifies distilled_result as referring to the complete_shared result?

Good point, updated.


> > Subject: [PATCH v2.12 01/28] aio: Be more paranoid about interrupts
> Ready for commit
> > Subject: [PATCH v2.12 02/28] aio: Pass result of local callbacks to
> >  ->report_return
> 
> Ready for commit w/ up to one cosmetic change:
> 

And pushed. Together with the s/pgaio_io_prep_/s/pgaio_io_start_/ renaming
we've been discussing. Btw, I figured out the origin of that, I was just
mirroring the liburing API...

Thanks again for the reviews.


> > Subject: [PATCH v2.12 03/28] aio: Add liburing dependency
> 
> Ready for commit
> 
> 
> > Subject: [PATCH v2.12 04/28] aio: Add io_method=io_uring
> 
> Ready for commit w/ open_fd.fixup

Yay.  Planning to push those soon.


> > Subject: [PATCH v2.12 05/28] aio: Implement support for reads in smgr/md/fd
> 
> Ready for commit w/ up to two cosmetic changes:

Cool.


> > +/*
> > + * AIO error reporting callback for mdstartreadv().
> > + *
> > + * Errors are encoded as follows:
> > + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
> 
> I recommend replacing "errno != 0" with either "that errno" or "errno ==
> error_data".

Applied.


> Second, the aio_internal.h comment changes discussed in
> postgr.es/m/20250325155808.f7.nmisch@google.com and earlier.

Here's my current version of that:

 * Note that the externally visible functions to start IO
 * (e.g. FileStartReadV(), via pgaio_io_start_readv()) move an IO from
 * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
 * PGAIO_HS_COMPLETED_LOCAL (at which point the handle will be reused).

Does that work?

I think I'll push that as part of the comment updates patch instead of
"Implement support for reads in smgr/md/fd", unless you see a reason to do so
differently. I'd have done it in the patch to s/prep/start/, but then it would
reference functions that don't exist yet...


> > Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design
> 
> Ready for commit

Cool.

Comments in it reference PGAIO_HCB_SHARED_BUFFER_READV, so I'm inclined to
reorder it until after "bufmgr: Implement AIO read support".

There's also a small change in a new patch in the series (not yet sent), due
to the changes related to emitting WARNINGs about checksum failures to the
client connection.  I think that part is fine, but...


> (This and the previous patch have three spots that would change with the
> s/prep/start/ renames.  No opinion on whether to rename before or rename
> after.)

I thought it'd be better to do the renaming first.



> > Subject: [PATCH v2.12 07/28] localbuf: Track pincount in BufferDesc as well
> 
> The plan here looks good:
> postgr.es/m/dbeeaize47y7esifdrinpa2l7cqqb67k72exvuf3appyxywjnc@7bt76mozhcy2

> > Subject: [PATCH v2.12 08/28] bufmgr: Implement AIO read support
> 
> See review here and later discussion:
> postgr.es/m/20250325022037.91.nmisch@google.com

I'm working on a version with those addressed.


> > Subject: [PATCH v2.12 09/28] bufmgr: Use AIO in StartReadBuffers()
> 
> Ready for commit after a batch of small things, all but one of which have no
> implications beyond code cosmetics.

Yay.


> I like the test coverage (by the end of the patch series).

I'm really shocked just how bad our test coverage for a lot of this is today
:(


> For anyone else following, I found "diff -w" helpful for the bufmgr.c
> changes.  That's because a key part is former WaitReadBuffers() code moving
> up an indentation level to its home in new subroutine AsyncReadBuffers().

For reviewing changes that move stuff around a lot I find this rather helpful:
git diff --color-moved --color-moved-ws=ignore-space-change

That highlights removed code differently from moved code, and due to
ignore-space-change considers code that changed just due to space changes, to
be moved.


> >  	Assert(*nblocks == 1 || allow_forwarding);
> >  	Assert(*nblocks > 0);
> >  	Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
> > +	Assert(*nblocks == 1 || allow_forwarding);
> 
> Duplicates the assert three lines back.

Ah, it was moved into ce1a75c4fea, which I didn't notice while rebasing...


> > +		nblocks = aio_ret->result.result;
> > +
> > +		elog(DEBUG3, "partial read, will retry");
> > +
> > +	}
> > +	else if (aio_ret->result.status == PGAIO_RS_ERROR)
> > +	{
> > +		pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
> > +		nblocks = 0;			/* silence compiler */
> > +	}
> >  
> >  	Assert(nblocks > 0);
> >  	Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
> >  
> > +	operation->nblocks_done += nblocks;
> 
> I struggled somewhat from the variety of "nblocks" variables: this local
> nblocks, operation->nblocks, actual_nblocks, and *nblocks in/out parameters of
> some functions.  No one of them is clearly wrong to use the name, and some of
> these names are preexisting.  That said, if you see opportunities to push in
> the direction of more-specific names, I'd welcome it.
> 
> For example, this local variable could become add_to_nblocks_done instead.

I named it "newly_read_blocks", hope that works?


> > +		AsyncReadBuffers(operation, &nblocks);
> 
> I suggest renaming s/nblocks/ignored_nblocks_progress/ here.

Adopted.


Unfortunately I didn't see a good way to reduce the amount of the other
nblocks variables, as they are all, I think, preexisting.


> > +	 * If we need to wait for IO before we can get a handle, submit already
> > +	 * staged IO first, so that other backends don't need to wait. There
> 
> s/already staged/already-staged/.  Normally I'd skip this as nitpicking, but I
> misread this particular sentence twice, as "submit" being the subject that
> "staged" something.  (It's still nitpicking, alas.)

Makes sense - it doesn't help that it was at a linebreak...


> >  		/*
> >  		 * How many neighboring-on-disk blocks can we scatter-read into other
> >  		 * buffers at the same time?  In this case we don't wait if we see an
> > -		 * I/O already in progress.  We already hold BM_IO_IN_PROGRESS for the
> > +		 * I/O already in progress.  We already set BM_IO_IN_PROGRESS for the
> >  		 * head block, so we should get on with that I/O as soon as possible.
> > -		 * We'll come back to this block again, above.
> > +		 *
> > +		 * We'll come back to this block in the next call to
> > +		 * StartReadBuffers() -> AsyncReadBuffers().
> 
> Did this mean to say "WaitReadBuffers() -> AsyncReadBuffers()"?  I'm guessing
> so, since WaitReadBuffers() is the one that loops.  It might be referring to
> read_stream_start_pending_read()'s next StartReadBuffers(), though.

I was referring to the latter, as that is the more common case (it's pretty
easy to hit if you e.g. have multiple sequential scans on the same table
going).


> I think this could just delete the last sentence.  The function header comment
> already mentions the possibility of reading a subset of the request.  This
> spot doesn't need to detail how the higher layers come back to here.

Agreed.


> > +		smgrstartreadv(ioh, operation->smgr, forknum,
> > +					   blocknum + nblocks_done,
> > +					   io_pages, io_buffers_len);
> > +		pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
> > +								io_start, 1, *nblocks_progress * BLCKSZ);
> 
> We don't assign *nblocks_progress until lower in the function, so I think
> "io_buffers_len" should replace "*nblocks_progress" here.  (This is my only
> non-cosmetic comment on this patch.)

Good catch!



> > Subject: [PATCH v2.12 16/28] aio: Add test_aio module
> 
> > +use List::Util qw(sample);
> 
> sample() is new in 2020:
> https://metacpan.org/release/PEVANS/Scalar-List-Utils-1.68/source/Changes#L100
> 
> Hence, I'd expect some buildfarm failures.  I'd try to use shuffle(), then
> take the first N elements.

Hah. Bilal's patch was using shuffe(). I wanted to reduce the number of
iterations and first did as you suggest and then saw that there's a nicer
way...

Done that way again...


> > +++ b/src/test/modules/test_aio/test_aio.c
> > @@ -0,0 +1,712 @@
> > +/*-------------------------------------------------------------------------
> > + *
> > + * delay_execution.c
> > + *		Test module to allow delay between parsing and execution of a query.
> > + *
> > + * The delay is implemented by taking and immediately releasing a specified
> > + * advisory lock.  If another process has previously taken that lock, the
> > + * current process will be blocked until the lock is released; otherwise,
> > + * there's no effect.  This allows an isolationtester script to reliably
> > + * test behaviors where some specified action happens in another backend
> > + * between parsing and execution of any desired query.
> > + *
> > + * Copyright (c) 2020-2025, PostgreSQL Global Development Group
> > + *
> > + * IDENTIFICATION
> > + *	  src/test/modules/test_aio/test_aio.c
> 
> To elaborate on my last review, the entire header comment was a copy from
> delay_execution.c.  v2.12 fixes the IDENTIFICATION, but the rest needs
> updates.

I was really too tired that day... Embarassing.

Greetings,

Andres Freund

  139. Re: AIO v2.5

    Thom Brown <thom@linux.com> — 2025-03-26T21:20:47Z

    On Tue, 25 Mar 2025 at 01:18, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> Attached v2.12, with the following changes:

I took a quick gander through this just out of curiosity (yes, I know
I'm late), and found these show-stoppers:

v2.12-0015-aio-Add-pg_aios-view.patch:

+          <literal>ERROR</literal> mean the I/O failed with an error.

s/mean/means/


v2.12-0021-bufmgr-Implement-AIO-write-support.patch

+shared buffer lock still allows some modification, e.g., for hint bits(see

s/bits\(see/bits \(see)

+buffers that can be used as the source / target for IO. A bounce buffer be

s/be/can be/

Regards

Thom

  140. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-26T21:21:15Z

    On Wed, Mar 26, 2025 at 04:33:49PM -0400, Andres Freund wrote:
> On 2025-03-25 17:19:15 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:

> > Second, the aio_internal.h comment changes discussed in
> > postgr.es/m/20250325155808.f7.nmisch@google.com and earlier.
> 
> Here's my current version of that:
> 
>  * Note that the externally visible functions to start IO
>  * (e.g. FileStartReadV(), via pgaio_io_start_readv()) move an IO from
>  * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
>  * PGAIO_HS_COMPLETED_LOCAL (at which point the handle will be reused).
> 
> Does that work?

Yes.

> I think I'll push that as part of the comment updates patch instead of
> "Implement support for reads in smgr/md/fd", unless you see a reason to do so
> differently. I'd have done it in the patch to s/prep/start/, but then it would
> reference functions that don't exist yet...

Agreed.

> > > Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design
> > 
> > Ready for commit
> 
> Cool.
> 
> Comments in it reference PGAIO_HCB_SHARED_BUFFER_READV, so I'm inclined to
> reorder it until after "bufmgr: Implement AIO read support".

Agreed.

> > For example, this local variable could become add_to_nblocks_done instead.
> 
> I named it "newly_read_blocks", hope that works?

Yes.

  141. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-26T21:41:42Z

    Hi,

On 2025-03-26 11:31:02 -0700, Noah Misch wrote:
> I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr:
> Comment fixes", the last patch before write support.

Thanks!


> postgr.es/m/20250326001915.bc.nmisch@google.com covered patches 1-9, and this
> email covers patches 10-17.  All remaining review comments are minor, so I've
> marked the commitfest entry Ready for Committer.  If there's anything you'd
> like re-reviewed before you commit it, feel free to bring it to my attention.
> Thanks for getting the feature to this stage!

As part of our discussion around the WARNING stuff I did make some changes,
it'd be good if you could look at those once I send them.  While I squashed
the rest of the changes (addressing review comments) into their base commits,
I left the error-reporting related bits and pieces in fixup commits, to make
that easier.


> On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO
> 
> > @@ -631,6 +637,9 @@ read_stream_begin_impl(int flags,
> >  	 * For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
> >  	 * above.  If we had real asynchronous I/O we might need a slightly
> >  	 * different definition.
> > +	 *
> > +	 * FIXME: Not sure what different definition we would need? I guess we
> > +	 * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
> 
> I think we don't need a different definition.  max_ios comes from
> effective_io_concurrency and similar settings.  The above comment's definition
> of max_ios=0 matches that GUC's documented behavior:
> 
>          The allowed range is
>          <literal>1</literal> to <literal>1000</literal>, or
>          <literal>0</literal> to disable issuance of asynchronous I/O requests.
> 
> I'll guess the comment meant that "advice disabled" is a no-op for AIO, so we
> could reasonably argue to have effective_io_concurrency=0 distinguish itself
> from effective_io_concurrency=1 in some different way for AIO.  Equally,
> there's no hurry to use that freedom to distinguish them.

Thomas has since provided an implementation of what he was thinking of when
writing that comment:
https://postgr.es/m/CA%2BhUKG%2B8SC2%3DAD3bC0Pn85aMXm-PE2JSFGhC%3DMFVJvNQLObZeA%40mail.gmail.com

I squashed that into "aio: Basic read_stream adjustments for real AIO".


> > Subject: [PATCH v2.12 11/28] read_stream: Introduce and use optional batchmode
> >  support
> 
> > This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
> > uses it where appropriate.
> 
> I'd also use the new flag on the read_stream_begin_smgr_relation() call in
> RelationCopyStorageUsingBuffer().  It uses block_range_read_stream_cb, and
> other streams of that callback rightly use the flag.

Ah, yes. I had searched for all read_stream_begin_relation(), but not for
_smgr...


> > + * b) directly or indirectly start another batch pgaio_enter_batchmode()
> 
> Needs new wording from end of postgr.es/m/20250325155808.f7.nmisch@google.com

Locally it's that, just need to send out a new version...
 *
 * b) start another batch (without first exiting batchmode and re-entering
 *    before returning)


> > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> 
> Consider also updating this comment to stop focusing on prefetch; I think
> changing that aligns with the patch's other changes:
>
> /*
>  * How many buffers PrefetchBuffer callers should try to stay ahead of their
>  * ReadBuffer calls by.  Zero means "never prefetch".  This value is only used
>  * for buffers not belonging to tablespaces that have their
>  * effective_io_concurrency parameter set.
>  */
> int			effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;

Good point.  Although I suspect it might be worth adjusting this, and also the
config.sgml bit about effective_io_concurrency separately. That seems like it
might take an iteration or two.


> > -#io_combine_limit = 128kB		# usually 1-128 blocks (depends on OS)
> > +#io_combine_limit = 128kB		# usually 1-32 blocks (depends on OS)
> 
> I think "usually 1-128" remains right given:

> GUC_UNIT_BLOCKS
> #define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
> #define PG_IOV_MAX Min(IOV_MAX, 128)

You're right.  I think I got this wrong when rebasing over conflicts due to
06fb5612c97.


> > -         On systems without prefetch advice support, attempting to configure
> > -         any value other than <literal>0</literal> will error out.
> > +         On systems with prefetch advice support,
> > +         <varname>effective_io_concurrency</varname> also controls the prefetch distance.
> 
> Wrap the last line.

Done.


> > Subject: [PATCH v2.12 14/28] docs: Add acronym and glossary entries for I/O
> >  and AIO
> 
> > These could use a lot more polish.
> 
> To me, it's fine as-is.

Cool.


> > I did not actually reference the new entries yet, because I don't really
> > understand what our policy for that is.
> 
> I haven't seen much of a policy on that.

That's sure what it looks like to me :/


> 
> > Subject: [PATCH v2.12 15/28] aio: Add pg_aios view
> > +retry:
> > +
> > +		/*
> > +		 * There is no lock that could prevent the state of the IO to advance
> > +		 * concurrently - and we don't want to introduce one, as that would
> > +		 * introduce atomics into a very common path. Instead we
> > +		 *
> > +		 * 1) Determine the state + generation of the IO.
> > +		 *
> > +		 * 2) Copy the IO to local memory.
> > +		 *
> > +		 * 3) Check if state or generation of the IO changed. If the state
> > +		 * changed, retry, if the generation changed don't display the IO.
> > +		 */
> > +
> > +		/* 1) from above */
> > +		start_generation = live_ioh->generation;
> > +		pg_read_barrier();
> 
> Based on the "really started after this function was called" and "no risk of a
> livelock here" comments below, I think "retry:"  should be here.  We don't
> want to livelock in the form of chasing ever-growing start_generation numbers.

You're right.


> > +		 * scratch. There's no risk of a livelock here, as an IO has a limited
> > +		 * sets of states it can be in, and state changes go only in a single
> > +		 * direction.
> > +		 */
> > +		if (live_ioh->state != start_state)
> > +			goto retry;
> 
> > +      <entry role="catalog_table_entry"><para role="column_definition">
> > +       <structfield>target</structfield> <type>text</type>
> > +      </para>
> > +      <para>
> > +       What kind of object is the I/O targeting:
> > +       <itemizedlist spacing="compact">
> > +        <listitem>
> > +         <para>
> > +          <literal>smgr</literal>, I/O on postgres relations
> 
> s/postgres relations/relations/ since SGML docs don't use the term "postgres"
> that way.

Not sure what I was even trying to express with "postgres relations" vs plain
"relations" here...

> 
> > Subject: [PATCH v2.12 16/28] aio: Add test_aio module
> 
> > --- a/src/test/modules/meson.build
> > +++ b/src/test/modules/meson.build
> > @@ -1,5 +1,6 @@
> >  # Copyright (c) 2022-2025, PostgreSQL Global Development Group
> >  
> > +subdir('test_aio')
> >  subdir('brin')
> 
> List is alphabetized; please preserve that.
> 
> > +++ b/src/test/modules/test_aio/Makefile
> > @@ -0,0 +1,26 @@
> > +# src/test/modules/delay_execution/Makefile
> 
> Update filename in comment.
>
> > +++ b/src/test/modules/test_aio/meson.build
> > @@ -0,0 +1,37 @@
> > +# Copyright (c) 2022-2024, PostgreSQL Global Development Group
> 
> s/2024/2025/


Done.


> > --- /dev/null
> > +++ b/src/test/modules/test_aio/t/001_aio.pl
> 
> s/ {4}/\t/g on this file.  It's mostly \t now, with some exceptions.

Huh. No idea how that happened.


> > +	test_inject_worker('worker', $node_worker);
> 
> What do we expect to happen if autovacuum or checkpointer runs one of these
> injection points?  I'm guessing it would at most make that process fail
> without affecting the test outcome.  If so, that's fine.

Autovacuum I disabled on the relations, to prevent that.

I think checkpointer should behave as you describe, although I could wonder if
it could confuse wait_for_log() based checks - but even so, I think that would
at worst lead to a test missing a bug, in extremely rare circumstances.

I tried triggering that condition, but it's pretty hard to hit, even after
lowering checkpoint_timeout to 1s and looping in the tests.



> > +	# normal handle use
> > +	psql_like($io_method, $psql, "handle_get_release()",
> > +		qq(SELECT handle_get_release()),
> > +		qr/^$/, qr/^$/);
> > +
> > +	# should error out, API violation
> > +	psql_like($io_method, $psql, "handle_get_twice()",
> > +		qq(SELECT handle_get_release()),
> > +		qr/^$/, qr/^$/);
> 
> Last two lines are a clone of the previous psql_like() call.  I guess this
> wants to instead call handle_get_twice() and check for some stderr.

Indeed.


> > +			"read_rel_block_ll() of $tblname page",
> 
> What does "_ll" stand for?

"low level".  I added a C comment:

/*
 * A "low level" read. This does similar things to what
 * StartReadBuffers()/WaitReadBuffers() do, but provides more control (and
 * less sanity).
 */


> > +	# Issue IO without waiting for completion, then exit
> > +	$psql_a->query_safe(
> > +		qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
> > +	$psql_a->reconnect_and_clear();
> > +
> > +	# Check that another backend can read the relevant block
> > +	psql_like(
> > +		$io_method,
> > +		$psql_b,
> > +		"completing read started by exited backend",
> 
> I think the exiting backend's pgaio_shutdown() completed it.

I wrote the test precisely to exercise that path, otherwise it's pretty hard
to reach. It does seem to reach the path reasonably reliably, although it's
much harder to catch that causing problems, as the IO is typically too fast.


> > +sub test_inject
> 
> This deserves a brief comment on the behaviors being tested, like the previous
> functions have.  It seems to be about short reads and hard failures like EIO.

Done.

Greetings,

Andres Freund

  142. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-26T21:42:26Z

    Hi,

On 2025-03-26 21:20:47 +0000, Thom Brown wrote:
> I took a quick gander through this just out of curiosity (yes, I know
> I'm late), and found these show-stoppers:
> 
> v2.12-0015-aio-Add-pg_aios-view.patch:
> 
> +          <literal>ERROR</literal> mean the I/O failed with an error.
> 
> s/mean/means/
> 
> 
> v2.12-0021-bufmgr-Implement-AIO-write-support.patch
> 
> +shared buffer lock still allows some modification, e.g., for hint bits(see
> 
> s/bits\(see/bits \(see)
> 
> +buffers that can be used as the source / target for IO. A bounce buffer be
> 
> s/be/can be/

Thanks! Squashed into my local tree.

Greetings,

Andres Freund

  143. Re: AIO v2.5

    Thomas Munro <thomas.munro@gmail.com> — 2025-03-26T21:52:10Z

    On Thu, Mar 27, 2025 at 10:41 AM Andres Freund <andres@anarazel.de> wrote:
> > > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> >
> > Consider also updating this comment to stop focusing on prefetch; I think
> > changing that aligns with the patch's other changes:
> >
> > /*
> >  * How many buffers PrefetchBuffer callers should try to stay ahead of their
> >  * ReadBuffer calls by.  Zero means "never prefetch".  This value is only used
> >  * for buffers not belonging to tablespaces that have their
> >  * effective_io_concurrency parameter set.
> >  */
> > int                   effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;
>
> Good point.  Although I suspect it might be worth adjusting this, and also the
> config.sgml bit about effective_io_concurrency separately. That seems like it
> might take an iteration or two.

+1 for rewriting that separately from this work on the code (I can
have a crack at that if you want).  For the comment, my suggestion
would be something like:

"Default limit on the level of concurrency that each I/O stream
(currently, ReadStream but in future other kinds of streams) can use.
Zero means that I/O is always performed synchronously, ie not
concurrently with query execution. This value can be overridden at the
tablespace level with the parameter of the same name. Note that
streams performing I/O not classified as single-session work respect
maintenance_io_concurrency instead."

  144. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-27T01:07:40Z

    Hi,

Attached v2.13, with the following changes:
Changes:

- Pushed a fair number of commits

  A lot of thanks goes to Noah's detailed reviews!


- As Noah pointed out, the zero_damaged_pages warning could be emitted in an
  io worker or another backend, but omitted in the backend that started the IO

  To address that:

  1) I added a new commit "aio: Add WARNING result status"
     (itself trivial)

  2) I changed buffer_readv_complete() to encode the warning/error in a more
     detailed way than before (was_zeroed, first_invalid_off, count_invalid)

     As part of that I put the encoding/decoding into a static inline

  3) Tracking the number of invalid buffers was awkward with
     buffer_readv_complete_one() returning a PgAioResult. Now it just
     reports whether it found an invalid page with an out argument.

  4) As discussed there now is a different error messages for the case of
     multiple invalid pages

     The code is a bit awkward to avoid code duplication, curious whether
     that's seen as acceptable?  I could just duplicate the entire ereport()
     instead.

  5) The WARNING in the callback is now a LOG, as it will be sent to the
     client as a WARNING explicitly when the IO's results are processed

     I actually chose LOG_SERVER_ONLY - that seemed slightly better than just
     LOG? But not at all sure.

     There's a comment explaining this now too.


  Noah, I think this set of changes would benefit from another round of
  review. I left these changes in "squash-later: " commits, to make it easier
  to see / think about.


- Added a comment about the pgaio_result_report() in md_readv_complete(). I
  changed it to LOG_SERVER_ONLY as well , but I'm not at all sure about that.


- Previously the buffer completion callback checked zero_damaged_pages - but
  that's not right, the GUC hopefully is only set on a per-session basis

  I solved that by having AsyncReadBuffers() add ZERO_ON_ERROR to the flags if
  zero_damaged_pages is configured.

  Also added a comment explaining that we probably should eventually use a
  separate flag, so we can adjust the errcode etc differently.


- Explicit test for zero_damaged_pages and ZERO_ON_ERROR

  As part of that I made read_rel_block_ll() support reading multiple
  blocks. That makes it a lot easier to verify that we handle cases like a
  4-block read where 2,3 are invalid correctly.


- I removed the code that "localbuf: Track pincount in BufferDesc as well"
  added to ConditionalLockBufferForCleanup() and IsBufferCleanupOK() as discussed

  Right now the situations that the code was worried don't exist yet, as we
  only support reads.

  I added a comment about not needing to worry about that yet to "bufmgr:
  Implement AIO read support". And then changed that comment to a FIXME in the
  write patches.


- Squashed Thomas' change to make io_concurrency=0 really not use AIO


- Lots of other review comments by Noah addressed


- Merged typo fixes by Thom Brown



TODO:


- There are more tests in test_aio that should be expanded to run for temp
  tables as well, not just normal tables


- Add an explicit test for the checksum verification in the completion callback

  There is an existing test for testing an invalid page due to page header
  verification in test_aio, but not for checksum failures.

  I think it's indirectly covered (e.g. in amcheck), but seems better to test
  it explicitly.


- Add error context callbacks for io worker and "foreign" IO completion

Greetings,

Andres Freund

  145. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-27T20:58:11Z

    Hi,

On 2025-03-26 21:07:40 -0400, Andres Freund wrote:
> TODO
> ...
> - Add an explicit test for the checksum verification in the completion callback
>
>   There is an existing test for testing an invalid page due to page header
>   verification in test_aio, but not for checksum failures.
>
>   I think it's indirectly covered (e.g. in amcheck), but seems better to test
>   it explicitly.

Ah, for crying out loud.  As it turns out, no, we do not have *ANY* tests for
this for the server-side. Not a single one. I'm somewhat apoplectic,
data_checksums is a really complicated feature, which we just started *turning
on by default*, without a single test of the failure behaviour, when detecting
failures is the one thing the feature is supposed to do.


I now wrote some tests. And I both regret doing so (because it found problems,
which would have been apparent long ago, if the feature had come with *any*
tests, if I had gone the same way I could have just pushed stuff) and am glad
I did (because I dislike pushing broken stuff).

I have to admit, I was tempted to just ignore this issue and just not say
anything about tests for checksum failures anymore.


Problems:

1) PageIsVerifiedExtended() emits a WARNING, just like with ZERO_ON_ERROR, we
   don't want to emit it in a) io workers b) another backend if it completes
   the error.

   This isn't hard to address, we can add PIV_LOG_LOG (or something like that)
   to emit it at a different log level and an out-parameter to trigger sending
   a warning / adjust the warning/error message we already emit once the
   issuer completes the IO.


2) With IO workers (and "foreign completors", in rare cases), the checksum
   failures would be attributed wrongly, as it reports all stats to
   MyDatabaseId

   As it turns out, this is already borked on master for shared relations,
   since pg_stat_database.checksum_failures has existed, see [1].

   This isn't too hard to fix, if we adjust the signature to
   PageIsVerifiedExtended() to pass in the database oid. But see also 3)


3) We can't pgstat_report_checksum_failure() during the completion callback,
   as it *sometimes* allocates memory

   Aside from the allocation-in-critical-section asserts, I think this is
   *extremely* unlikely to actually cause a problem in practice. But we don't
   want to rely on that, obviously.



Addressing the first two is pretty simple and/or needs to be done anyway,
since it's a currently existing bug, as discussed in [1].


Addressing 3) is not at all trivial.  Here's what I've thought of so far:


Approach I)

My first thoughts were around trying to make the relevant pgstat
infrastructure either not need to allocate memory, or handle memory
allocation failures gracefully.

Unfortunately that seems not really viable:

The most successful approach I tried was to report stats directly to the
dshash table, and only report stats if there's already an entry (which
there just about always will, except for a very short period after stats
have been reset).

Unfortunately that fails because to access the shared memory with the stats
data we need to do dsa_get_address(), which can fail if the relevant dsm
segment wasn't already mapped in the current process (it allocates memory
in the process of mapping in the segment). There's no API to do that
without erroring out.

That aspect rules out a number of other approaches that sounded like they
could work - we e.g. could increase the refcount of the relevant pgstat
entry before issuing IO, ensuring that it's around by the time we need to
report. But that wouldn't get around the issue of needing to map in the dsm
segment.


Approach II)

Don't report the error in the completion callback.  The obvious place would be
to do it where we we'll raise the warning/error in the issuing process.  The
big disadvantage is that that that could lead to under-counting checksum
errors:

a) A read stream does 2+ concurrent reads for the same relation, and more than
one encounters checksum errors. When processing the results for the first
failed read, we raise an error and thus won't process the results of the
second+ reads with errors.

b) A read is started asynchronously, but before the backend gets around to
processing the result of the IO, it errors out during that other work
(possibly due to a cancellation). Because the backend never looked at the
results of the IOs, the checksum errors don't get accounted for.

b) doesn't overly bother me, but a) seems problematic.


Approach III)

Accumulate checksum errors in two backend local variables (one for database
specific errors, one for errors on shared relations), which will be flushed by
the backend that issued IO during the next pgstat_report_start().

Two disadvantages:

- Accumulation of errors will be delayed until the next
  pgstat_report_start(). That seems acceptable, after all we do so far a lot
  of other stats.

- We need to register a local callback for shared buffer reads, which don't
  need them today . That's a small bit of added overhead. It's a shame to do
  so for counters that approximately never get incremented.


Approach IV):

Embracy piercing abstractions / generic infrastructure and put two atomic
variables (one for shared one for the backend's database) in some
backend-specific shared memory (e.g. the backend's PgAioBackend or PGPROC) and
update that in the completion callback. Flush that variable to the shared
stats in pgstat_report_start() or such.

This would avoid the need for the local completion callback, and would also
allow to introduce a function see the number "unflushed" checksum errors. It
also doesn't require transporting the number of errors between the shared
callback and the local callback - but we might want to do have that for the
error message anyway.

I wish the new-to-18 pgstat_backend() were designed in a way to make this
possible in a nice way. But unfortunately it puts the backend-specific data in
the dshash table / dynamic shared memory, rather than in a MaxBackends +
NUM_AUX sized array array in plain shared memory. As explained in I), we can't
rely on having the entire array mapped.  Leaving the issue from this email
aside, that also adds a fair bit of overhead to other cases.



Does anybody have better ideas?


I think II), III) and IV) are all relatively simple to implement.

The most complicated bit is that a bit of bit-squeezing is necessary to fit
the number of checksum errors (in addition to the number of otherwise invalid
pages) into the available space for error data. It's doable. We could also
just increase the size of PgAioResult.

I've implemented II), but I'm not sure the disadvantages are acceptable.


Greetings,

Andres Freund

[1] https://postgr.es/m/mglpvvbhighzuwudjxzu4br65qqcxsnyvio3nl4fbog3qknwhg%40e4gt7npsohuz

  146. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-28T03:22:23Z

    On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> I now wrote some tests. And I both regret doing so (because it found problems,
> which would have been apparent long ago, if the feature had come with *any*
> tests, if I had gone the same way I could have just pushed stuff) and am glad
> I did (because I dislike pushing broken stuff).
> 
> I have to admit, I was tempted to just ignore this issue and just not say
> anything about tests for checksum failures anymore.

I don't blame you.

> 3) We can't pgstat_report_checksum_failure() during the completion callback,
>    as it *sometimes* allocates memory
> 
>    Aside from the allocation-in-critical-section asserts, I think this is
>    *extremely* unlikely to actually cause a problem in practice. But we don't
>    want to rely on that, obviously.

> Addressing 3) is not at all trivial.  Here's what I've thought of so far:
> 
> 
> Approach I)

> Unfortunately that fails because to access the shared memory with the stats
> data we need to do dsa_get_address()


> Approach II)
> 
> Don't report the error in the completion callback.  The obvious place would be
> to do it where we we'll raise the warning/error in the issuing process.  The
> big disadvantage is that that that could lead to under-counting checksum
> errors:
> 
> a) A read stream does 2+ concurrent reads for the same relation, and more than
> one encounters checksum errors. When processing the results for the first
> failed read, we raise an error and thus won't process the results of the
> second+ reads with errors.
> 
> b) A read is started asynchronously, but before the backend gets around to
> processing the result of the IO, it errors out during that other work
> (possibly due to a cancellation). Because the backend never looked at the
> results of the IOs, the checksum errors don't get accounted for.
> 
> b) doesn't overly bother me, but a) seems problematic.

While neither are great, I could live with both.  I guess I'm optimistic that
clusters experiencing checksum failures won't lose enough reports to these
loss sources to make the difference in whether monitoring catches them.  In
other words, a cluster will report N failures without these losses and N-K
after these losses.  If N is large enough for relevant monitoring to flag the
cluster appropriately, N-K will also be large enough.


> Approach III)
> 
> Accumulate checksum errors in two backend local variables (one for database
> specific errors, one for errors on shared relations), which will be flushed by
> the backend that issued IO during the next pgstat_report_start().
> 
> Two disadvantages:
> 
> - Accumulation of errors will be delayed until the next
>   pgstat_report_start(). That seems acceptable, after all we do so far a lot
>   of other stats.

Yep, acceptable.

> - We need to register a local callback for shared buffer reads, which don't
>   need them today . That's a small bit of added overhead. It's a shame to do
>   so for counters that approximately never get incremented.

Fair concern.  An idea is to let the complete_shared callback change the
callback list associated with the IO, so it could change
PGAIO_HCB_SHARED_BUFFER_READV to PGAIO_HCB_SHARED_BUFFER_READV_SLOW.  The
latter would differ from the former only in having the extra local callback.
Could that help?  I think the only overhead is using more PGAIO_HCB numbers.
We currently reserve 256 (uint8), but one could imagine trying to pack into
fewer bits.  That said, this wouldn't paint us into a corner.  We could change
the approach later.

pgaio_io_call_complete_local() starts a critical section.  Is that a problem
for this approach?


> Approach IV):
> 
> Embracy piercing abstractions / generic infrastructure and put two atomic
> variables (one for shared one for the backend's database) in some
> backend-specific shared memory (e.g. the backend's PgAioBackend or PGPROC) and
> update that in the completion callback. Flush that variable to the shared
> stats in pgstat_report_start() or such.

I could live with that.  I feel better about Approach III currently, though.
Overall, I'm feeling best about III long-term, but II may be the right
tactical choice.


> Does anybody have better ideas?

I think no, but here are some ideas I tossed around:

- Like your Approach III, but have the completing process store the count
  locally and flush it, instead of the staging process doing so.  Would need
  more than 2 slots, but we could have a fixed number of slots and just
  discard any reports that arrive with all slots full.  Reporting checksum
  failures in, say, 8 databases in quick succession probably tells the DBA
  there's "enough corruption to start worrying".  Missing the 9th database
  would be okay.

- Pre-warm the memory allocations and DSAs we could possibly need, so we can
  report those stats in critical sections, from the completing process.  Bad
  since there's an entry per database, hence no reasonable limit on how much
  memory a process might need to pre-warm.  We could even end up completing an
  IO for a database that didn't exist on entry to our critical section.

- Skip the checksum pgstats if we're completing in a critical section.
  Doesn't work since we _always_ make a critical section to complete I/O.

This email isn't as well-baked as I like, but the alternative was delaying it
24-48h depending on how other duties go over those hours.  My v2.13 review is
still in-progress, too.

  147. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-28T12:54:42Z

    Hi,

On 2025-03-27 20:22:23 -0700, Noah Misch wrote:
> On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> > Don't report the error in the completion callback.  The obvious place would be
> > to do it where we we'll raise the warning/error in the issuing process.  The
> > big disadvantage is that that that could lead to under-counting checksum
> > errors:
> >
> > a) A read stream does 2+ concurrent reads for the same relation, and more than
> > one encounters checksum errors. When processing the results for the first
> > failed read, we raise an error and thus won't process the results of the
> > second+ reads with errors.
> >
> > b) A read is started asynchronously, but before the backend gets around to
> > processing the result of the IO, it errors out during that other work
> > (possibly due to a cancellation). Because the backend never looked at the
> > results of the IOs, the checksum errors don't get accounted for.
> >
> > b) doesn't overly bother me, but a) seems problematic.
>
> While neither are great, I could live with both.  I guess I'm optimistic that
> clusters experiencing checksum failures won't lose enough reports to these
> loss sources to make the difference in whether monitoring catches them.  In
> other words, a cluster will report N failures without these losses and N-K
> after these losses.  If N is large enough for relevant monitoring to flag the
> cluster appropriately, N-K will also be large enough.

That's true.


> > Approach III)
> >
> > Accumulate checksum errors in two backend local variables (one for database
> > specific errors, one for errors on shared relations), which will be flushed by
> > the backend that issued IO during the next pgstat_report_start().

FWIW, two variables turn out to not quite suffice - as I realized later, we
actually can issue IO on behalf of arbitrary databases, due to
ScanSourceDatabasePgClass() and RelationCopyStorageUsingBuffer().

That unfortunately makes it much harder to be able to guarantee that the
completor of an IO has the DSM segment for a pg_stat_database stats entry
mapped.


> > - We need to register a local callback for shared buffer reads, which don't
> >   need them today . That's a small bit of added overhead. It's a shame to do
> >   so for counters that approximately never get incremented.
>
> Fair concern.  An idea is to let the complete_shared callback change the
> callback list associated with the IO, so it could change
> PGAIO_HCB_SHARED_BUFFER_READV to PGAIO_HCB_SHARED_BUFFER_READV_SLOW.  The
> latter would differ from the former only in having the extra local callback.
> Could that help?  I think the only overhead is using more PGAIO_HCB numbers.

I think changing the callback could work - I'll do some measurements in a
coffee or two, but I suspect the overhead is not worth being too worried about
for now.  There's a different aspect that worries me slightly more, see
further down.


> We currently reserve 256 (uint8), but one could imagine trying to pack into
> fewer bits.

Yea, my current local worktree reduces it to 6 bits for now, to make space for
keeping track of the number of checksum failures in error data (as part of
that adds defines for the bit widths).  If that becomes an issue we can make
PgAioResult wider, but I suspect that won't be too soon.

One simplification that we could make is to only ever report one checksum
failure for each IO, even if N buffers failed - after all that's what HEAD
does (by virtue of throwing an error after the first). Then we'd not track the
number of checksum errors.


> That said, this wouldn't paint us into a corner.  We could change the
> approach later.

Indeed - I think we mainly need something that works for now.  I think medium
term the right fix here would be to make sure that the stats can be accounted
for with just an atomic increment somewhere.

We've had several discussions around having an in-memory datastructure for
every relation that currently has buffer in shared_buffers, to store e.g. the
relation length and the sync requests. If we get that, I think Thomas has a
prototype, we can accumulate the number of checksum errors in there, for
example. It'd also allow to address the biggest blocker for writes, namely
that RememberSyncRequest() could fail, *after* IO comletion.


> pgaio_io_call_complete_local() starts a critical section.  Is that a problem
> for this approach?

I think we can make it not a problem - I added a
pgstat_prepare_report_checksum_failure(dboid) that ensures the calling backend
has a reference to the relevant shared memory stats entry. If we make the rule
that it has to be called *before* starting buffered IO (i.e. in
AsyncReadBuffers()), we can be sure the stats reference still exists by the
time local completion runs (as the isn't a way to have the stats entry dropped
without dropping the database, which isn't possible while a) the database
still is connected to, for normal IO b) the CREATE DATABASE is still running).

Unfortunately pgstat_prepare_report_checksum_failure() has to do a lookup in a
local hashtable. That's more expensive than an indirect function call
(i.e. the added local callback). I hope^Wsuspect it'll still be fine, and if
not we can apply a mini-cache for the current database, which is surely the
only thing that ever matters for performance.


> > Approach IV):
> >
> > Embracy piercing abstractions / generic infrastructure and put two atomic
> > variables (one for shared one for the backend's database) in some
> > backend-specific shared memory (e.g. the backend's PgAioBackend or PGPROC) and
> > update that in the completion callback. Flush that variable to the shared
> > stats in pgstat_report_start() or such.
>
> I could live with that.  I feel better about Approach III currently, though.
> Overall, I'm feeling best about III long-term, but II may be the right
> tactical choice.

I think it's easy to change between these approaches. Both require that we
encode the number of checksum failures in the result, which is where most of
the complexity lies (but still a rather surmountable amount of complexity).


> I think no, but here are some ideas I tossed around:
>
> - Like your Approach III, but have the completing process store the count
>   locally and flush it, instead of the staging process doing so.  Would need
>   more than 2 slots, but we could have a fixed number of slots and just
>   discard any reports that arrive with all slots full.  Reporting checksum
>   failures in, say, 8 databases in quick succession probably tells the DBA
>   there's "enough corruption to start worrying".  Missing the 9th database
>   would be okay.

Yea. I think that'd be an ok fallback, but if we can make III' work, it'd be
nicer.


> - Pre-warm the memory allocations and DSAs we could possibly need, so we can
>   report those stats in critical sections, from the completing process.  Bad
>   since there's an entry per database, hence no reasonable limit on how much
>   memory a process might need to pre-warm.  We could even end up completing an
>   IO for a database that didn't exist on entry to our critical section.

I experimented with this one - it works surprisingly well, because for IO
workers we could just do the pre-warming outside of the critical section, and
it's *exceedingly* rare that any other completor would ever need to complete
IO for another database than the current / a shared relation.

But it does leave a nasty edge case, that we'd just have to accept. I guess we
could just make it so that in that case stats aren't reported.

But it seems pretty ugly.


> This email isn't as well-baked as I like, but the alternative was delaying it
> 24-48h depending on how other duties go over those hours.  My v2.13 review is
> still in-progress, too.

It's appreciated!

Greetings,

Andres Freund

  148. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-28T12:57:25Z

    Hi,

On 2025-03-28 08:54:42 -0400, Andres Freund wrote:
> One simplification that we could make is to only ever report one checksum
> failure for each IO, even if N buffers failed - after all that's what HEAD
> does (by virtue of throwing an error after the first). Then we'd not track the
> number of checksum errors.

Just after sending, I thought of another variation: Report the number of
*invalid* pages (which we already track) as checksum errors, if there was at
least on checksum error.

It's imo rather weird that we track checksum errors but we don't track invalid
page headers, despite the latter being an even worse indication of something
having gone wrong...

Greetings,

Andres Freund

  149. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-29T03:35:23Z

    Hi,

On 2025-03-28 08:54:42 -0400, Andres Freund wrote:
> On 2025-03-27 20:22:23 -0700, Noah Misch wrote:
> > On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> > > - We need to register a local callback for shared buffer reads, which don't
> > >   need them today . That's a small bit of added overhead. It's a shame to do
> > >   so for counters that approximately never get incremented.
> >
> > Fair concern.  An idea is to let the complete_shared callback change the
> > callback list associated with the IO, so it could change
> > PGAIO_HCB_SHARED_BUFFER_READV to PGAIO_HCB_SHARED_BUFFER_READV_SLOW.  The
> > latter would differ from the former only in having the extra local callback.
> > Could that help?  I think the only overhead is using more PGAIO_HCB numbers.
>
> I think changing the callback could work - I'll do some measurements in a
> coffee or two, but I suspect the overhead is not worth being too worried about
> for now.  There's a different aspect that worries me slightly more, see
> further down.
> ...
> Unfortunately pgstat_prepare_report_checksum_failure() has to do a lookup in a
> local hashtable. That's more expensive than an indirect function call
> (i.e. the added local callback). I hope^Wsuspect it'll still be fine, and if
> not we can apply a mini-cache for the current database, which is surely the
> only thing that ever matters for performance.

I tried it and at ~30GB/s of read IO, with checksums disabled, I can't see a
difference of either having the unnecessary complete_local callback or having
the lookup in pgstat_prepare_report_checksum_failure(). In a profile there are
a few hits inside pgstat_get_entry_ref(), but not enough to matter.

Hence I think this isn't worth worrying about, at least for now. I think we
have far bigger fish to fry at this point than such a small performance
difference.

I've adjusted the comment above TRACE_POSTGRESQL_BUFFER_READ_DONE() to not
mention the overhead. I'm still inclined to think that it's better to call it
in the shared completion callback.


I also fixed support and added tests for ignore_checksum_failure, that also
needs to be determined at the start of the IO, not in the completion.  Once
more there were no tests, of course.


I spent the last 6 hours on the stupid error/warning messages around this,
somewhat ridiculous.

The number of combinations is annoyingly large. It's e.g. plausible to use
ignore_checksum_failure=on and zero_damaged_pages=on at the same time for
recovery. The same buffer could both be ignored *and* zeroed. Or somebody
could use ignore_checksum_failure=on but then still encounter a page that is
invalid.

But I finally got to a point where the code ends up readable, without undue
duplication.  It would, leaving some nasty hack aside, require a
errhint_internal() - but I can't imagine a reason against introducing that,
given we have it for the errmsg and errhint.

Here's the relevant code:

	/*
	 * Treat a read that had both zeroed buffers *and* ignored checksums as a
	 * special case, it's too irregular to be emitted the same way as the other
	 * cases.
	 */
	if (zeroed_any && ignored_any)
	{
		Assert(zeroed_any && ignored_any);
		Assert(nblocks > 1);	/* same block can't be both zeroed and ignored */
		Assert(result.status != PGAIO_RS_ERROR);
		affected_count = zeroed_or_error_count;

		ereport(elevel,
				errcode(ERRCODE_DATA_CORRUPTED),
				errmsg("zeroing %u pages and ignoring %u checksum failures among blocks %u..%u of relation %s",
					   affected_count, checkfail_count, first, last, rpath.str),
				affected_count > 1 ?
				errdetail("Block %u held first zeroed page.",
						  first + first_off) : 0,
				errhint("See server log for details about the other %u invalid blocks.",
						affected_count + checkfail_count - 1));
		return;
	}

	/*
	 * The other messages are highly repetitive. To avoid duplicating a long
	 * and complicated ereport(), gather the translated format strings
	 * separately and then do one common ereport.
	 */
	if (result.status == PGAIO_RS_ERROR)
	{
		Assert(!zeroed_any);	/* can't have invalid pages when zeroing them */
		affected_count = zeroed_or_error_count;
		msg_one = _("invalid page in block %u of relation %s");
		msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
		det_mult = _("Block %u held first invalid page.");
		hint_mult = _("See server log for the other %u invalid blocks.");
	}
	else if (zeroed_any && !ignored_any)
	{
		affected_count = zeroed_or_error_count;
		msg_one = _("invalid page in block %u of relation %s; zeroing out page");
		msg_mult = _("zeroing out %u invalid pages among blocks %u..%u of relation %s");
		det_mult = _("Block %u held first zeroed page.");
		hint_mult = _("See server log for the other %u zeroed blocks.");
	}
	else if (!zeroed_any && ignored_any)
	{
		affected_count = checkfail_count;
		msg_one = _("ignoring checksum failure in block %u of relation %s");
		msg_mult = _("ignoring %u checksum failures among blocks %u..%u of relation %s");
		det_mult = _("Block %u held first ignored page.");
		hint_mult = _("See server log for the other %u ignored blocks.");
	}
	else
		pg_unreachable();

	ereport(elevel,
			errcode(ERRCODE_DATA_CORRUPTED),
			affected_count == 1 ?
			errmsg_internal(msg_one, first + first_off, rpath.str) :
			errmsg_internal(msg_mult, affected_count, first, last, rpath.str),
			affected_count > 1 ? errdetail_internal(det_mult, first + first_off) : 0,
			affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);

Does that approach make sense?

What do you think about using
  "zeroing invalid page in block %u of relation %s"
instead of
  "invalid page in block %u of relation %s; zeroing out page"

I thought about instead translating "ignoring", "ignored", "zeroing",
"zeroed", etc separately, but I have doubts about how well that would actually
translate.

Greetings,

Andres Freund

  150. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-29T13:41:43Z

    On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:
> The number of combinations is annoyingly large. It's e.g. plausible to use
> ignore_checksum_failure=on and zero_damaged_pages=on at the same time for
> recovery.

That's intricate indeed.

> But I finally got to a point where the code ends up readable, without undue
> duplication.  It would, leaving some nasty hack aside, require a
> errhint_internal() - but I can't imagine a reason against introducing that,
> given we have it for the errmsg and errhint.

Introducing that is fine.

> Here's the relevant code:
> 
> 	/*
> 	 * Treat a read that had both zeroed buffers *and* ignored checksums as a
> 	 * special case, it's too irregular to be emitted the same way as the other
> 	 * cases.
> 	 */
> 	if (zeroed_any && ignored_any)
> 	{
> 		Assert(zeroed_any && ignored_any);
> 		Assert(nblocks > 1);	/* same block can't be both zeroed and ignored */
> 		Assert(result.status != PGAIO_RS_ERROR);
> 		affected_count = zeroed_or_error_count;
> 
> 		ereport(elevel,
> 				errcode(ERRCODE_DATA_CORRUPTED),
> 				errmsg("zeroing %u pages and ignoring %u checksum failures among blocks %u..%u of relation %s",
> 					   affected_count, checkfail_count, first, last, rpath.str),

Translation stumbles on this one, because each of the first two %u are
plural-sensitive.  I'd do one of:

- Call ereport() twice, once for zeroed pages and once for ignored checksums.
  Since elevel <= ERROR here, that doesn't lose the second call.

- s/pages/page(s)/ like msgid "There are %d other session(s) and %d prepared
  transaction(s) using the database."

- Something more like the style of VACUUM VERBOSE, e.g. "INTRO_TEXT: %u
  zeroed, %u checksums ignored".  I've not written INTRO_TEXT, and this
  doesn't really resolve pluralization.  Probably don't use this option.

> 				affected_count > 1 ?
> 				errdetail("Block %u held first zeroed page.",
> 						  first + first_off) : 0,
> 				errhint("See server log for details about the other %u invalid blocks.",
> 						affected_count + checkfail_count - 1));
> 		return;
> 	}
> 
> 	/*
> 	 * The other messages are highly repetitive. To avoid duplicating a long
> 	 * and complicated ereport(), gather the translated format strings
> 	 * separately and then do one common ereport.
> 	 */
> 	if (result.status == PGAIO_RS_ERROR)
> 	{
> 		Assert(!zeroed_any);	/* can't have invalid pages when zeroing them */
> 		affected_count = zeroed_or_error_count;
> 		msg_one = _("invalid page in block %u of relation %s");
> 		msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
> 		det_mult = _("Block %u held first invalid page.");
> 		hint_mult = _("See server log for the other %u invalid blocks.");

For each hint_mult, we would usually use ngettext() instead of _().  (Would be
errhint_plural() if not separated from its ereport().)  Alternatively,
s/blocks/block(s)/ is fine.

> 	}
> 	else if (zeroed_any && !ignored_any)
> 	{
> 		affected_count = zeroed_or_error_count;
> 		msg_one = _("invalid page in block %u of relation %s; zeroing out page");
> 		msg_mult = _("zeroing out %u invalid pages among blocks %u..%u of relation %s");
> 		det_mult = _("Block %u held first zeroed page.");
> 		hint_mult = _("See server log for the other %u zeroed blocks.");
> 	}
> 	else if (!zeroed_any && ignored_any)
> 	{
> 		affected_count = checkfail_count;
> 		msg_one = _("ignoring checksum failure in block %u of relation %s");
> 		msg_mult = _("ignoring %u checksum failures among blocks %u..%u of relation %s");
> 		det_mult = _("Block %u held first ignored page.");
> 		hint_mult = _("See server log for the other %u ignored blocks.");
> 	}
> 	else
> 		pg_unreachable();
> 
> 	ereport(elevel,
> 			errcode(ERRCODE_DATA_CORRUPTED),
> 			affected_count == 1 ?
> 			errmsg_internal(msg_one, first + first_off, rpath.str) :
> 			errmsg_internal(msg_mult, affected_count, first, last, rpath.str),
> 			affected_count > 1 ? errdetail_internal(det_mult, first + first_off) : 0,
> 			affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);
> 
> Does that approach make sense?

Yes.

> What do you think about using
>   "zeroing invalid page in block %u of relation %s"
> instead of
>   "invalid page in block %u of relation %s; zeroing out page"

I like the replacement.  It moves the important part to the front, and it's
shorter.

> I thought about instead translating "ignoring", "ignored", "zeroing",
> "zeroed", etc separately, but I have doubts about how well that would actually
> translate.

Agreed, I wouldn't have high hopes for that.  An approach like that would
probably need messages that separate the independently-translated part
grammatically, e.g.:

  /* last %s is translation of "ignore" or "zero-fill" */
  "invalid page in block %u of relation %s; resolved by method \"%s\""

(Again, I'm not recommending that.)

  151. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-29T14:48:10Z

    Hi,

On 2025-03-29 06:41:43 -0700, Noah Misch wrote:
> On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:
> > But I finally got to a point where the code ends up readable, without undue
> > duplication.  It would, leaving some nasty hack aside, require a
> > errhint_internal() - but I can't imagine a reason against introducing that,
> > given we have it for the errmsg and errhint.
>
> Introducing that is fine.

Cool.


> > Here's the relevant code:
> >
> > 	/*
> > 	 * Treat a read that had both zeroed buffers *and* ignored checksums as a
> > 	 * special case, it's too irregular to be emitted the same way as the other
> > 	 * cases.
> > 	 */
> > 	if (zeroed_any && ignored_any)
> > 	{
> > 		Assert(zeroed_any && ignored_any);
> > 		Assert(nblocks > 1);	/* same block can't be both zeroed and ignored */
> > 		Assert(result.status != PGAIO_RS_ERROR);
> > 		affected_count = zeroed_or_error_count;
> >
> > 		ereport(elevel,
> > 				errcode(ERRCODE_DATA_CORRUPTED),
> > 				errmsg("zeroing %u pages and ignoring %u checksum failures among blocks %u..%u of relation %s",
> > 					   affected_count, checkfail_count, first, last, rpath.str),
>
> Translation stumbles on this one, because each of the first two %u are
> plural-sensitive.

Fair. We don't generally seem to have been very careful around this in relate
code, but there's no reason to just continue down that road when it's easy.

E.g. in md.c we unconditionally output "could not read blocks %u..%u in file \"%s\": %m"
even if it's just a single block...


> I'd do one of:
>
> - Call ereport() twice, once for zeroed pages and once for ignored checksums.
>   Since elevel <= ERROR here, that doesn't lose the second call.
>
> - s/pages/page(s)/ like msgid "There are %d other session(s) and %d prepared
>   transaction(s) using the database."

I think I like this better.


> > 	/*
> > 	 * The other messages are highly repetitive. To avoid duplicating a long
> > 	 * and complicated ereport(), gather the translated format strings
> > 	 * separately and then do one common ereport.
> > 	 */
> > 	if (result.status == PGAIO_RS_ERROR)
> > 	{
> > 		Assert(!zeroed_any);	/* can't have invalid pages when zeroing them */
> > 		affected_count = zeroed_or_error_count;
> > 		msg_one = _("invalid page in block %u of relation %s");
> > 		msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
> > 		det_mult = _("Block %u held first invalid page.");
> > 		hint_mult = _("See server log for the other %u invalid blocks.");
>
> For each hint_mult, we would usually use ngettext() instead of _().  (Would be
> errhint_plural() if not separated from its ereport().)  Alternatively,
> s/blocks/block(s)/ is fine.

I will go with the (s) here as well, this stuff is too rare to be worth having
pluralized messages imo.


> > Does that approach make sense?
>
> Yes.
> ...
> > I like the replacement.  It moves the important part to the front, and it's
> shorter.

Cool, I squashed them with the relevant changes now.


Attached is v2.14:

Changes:

- Added a commit to fix stats attribution of checksum errors, previously the
  checksum errors detected bufmgr.c/storage.c were always attributed to the
  current database

  This would have caused bigger issues with worker based IO, as IO workers
  aren't connected to databases.


- Added a commit to allow checksum error reports to happen in critical
  sections. For that a pgstat_prepare_report_checksum_failure() has to be
  called in the same backend, to report the critical section.

  Other suggestions for the name welcome.


- Expanded on the idea in 13 to track the number of invalid buffers in the
  IO's result, by also tracking checksum errors. Combined with the previous
  point, this fixes the issue of an assert during checksum failure reporting
  outlined in:

  https://postgr.es/m/5tyic6epvdlmd6eddgelv47syg2b5cpwffjam54axp25xyq2ga%40ptwkinxqo3az

  This required being a bit more careful with space in the error, to be able
  to squeeze in the checksum number.


- The ignore_checksum_failure of the issuer needs to be used when completing
  IO, not the one of the completor, particularly when using io_method=worker

  For that the access to ignore_checksum_failure had to be moved from
  PageIsVerified() to its callers.

  I added tests for ignore_checksum_failure, including its interplay with
  zero_damaged_pages.


- Deduplicated the error reporting in buffer_readv_report() somewhat by only
  having the selection of format strings be done in branches. I think this
  ends up a lot more readable than the huge ereport before.


- Added details about the changed error/warning logging to "bufmgr: Implement
  AIO read support"'s commit message.


- polished the commit to add PGAIO_RS_WARNING a bit, adding defines for the
  bit-widths of PgAioResult portions and added static asserts to verify them


- Squashed the changes that I had kept separately in v2.13, it was too hard to
  do that while doing the above changes.

  I did make the encoding function cast the arguments to uint32 before
  shifting. I think that's implied by the C integer promotion rules, but it
  seemed fishy enough to not want to believe in that.

  I also added a StaticAssertStmt() to ensure we are only using the available
  bit space.


- Added a test for a) checksum errors being detected b) CREATE DATABASE
  ... STRATEGY WAL_LOG

  The latter is interesting because it also provides test coverage for doing
  IO for objects in other databases.


- Removed an obsoleted inclusion of pg_trace.h in localbuf.c


TODO:

- I think the tests around zero_damaged_pages, ignore_checksum_failure should
  be expanded a bit more. There's two FIXME in the tests about that.

  At the moment there are two different test functions for zero_damaged_pages
  and ignore_checksum_failure, I'm not sure how good that is.

  I wanted to get this version out, because I have to run some errands,
  otherwise I'd have implemente them first...


Next steps:

- push the checksums stats fix


- unless somebody sees a reason to not use LOG_SERVER_ONLY in
  "aio: Implement support for reads in smgr/md/fd", push that

  Besides that the only change since Noah's last review of that commit is an
  added comment.


- push acronym, glossary change


- push pg_aios view (depends a tiny bit on the smgr/md/fd change above)


- push "localbuf: Track pincount in BufferDesc as well" - I think I addressed
  all of Noah's review feedback


- address the above TODO



Greetings,

Andres Freund

  152. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-29T18:25:15Z

    Hi,

On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> Attached is v2.14:

FWIW, there was a last minute change in the test that fails in one task on CI,
due to reading across the smaller segment size configured for one of the
runs. Doesn't quite seem worth posting a new version for.


> - push the checksums stats fix

Done.


> - unless somebody sees a reason to not use LOG_SERVER_ONLY in
>   "aio: Implement support for reads in smgr/md/fd", push that
>
>   Besides that the only change since Noah's last review of that commit is an
>   added comment.

Also done.  If we want to change log level later, it's easy to do so.

I made some small changes since the version I had posted:
- I found one dangling reference to mdread() instead of mdreadv()
- I had accidentally squashed the fix to Noah's review comment about a comment
  above md_readv_report() to the wrong commit (smgr/md/fd.c write support)
- PGAIO_HCB_MD_WRITEV was added in "smgr/md/fd.c read support" instead of
  "smgr/md/fd.c write support"


> - push pg_aios view (depends a tiny bit on the smgr/md/fd change above)

I think I found an issue with this one - as it stands the view was viewable by
everyone. While it doesn't provide a *lot* of insight, it still seems a bit
too much for an unprivileged user to learn what part of a relation any other
user is currently reading.

There'd be two different ways to address that:
1) revoke view & function from public, grant to a limited role (presumably
   pg_read_all_stats)
2) copy pg_stat_activity's approach of using something like

   #define HAS_PGSTAT_PERMISSIONS(role)	 (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))

   on a per-IO basis.


Greetings,

Andres Freund

  153. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-29T20:16:50Z

    On Sat, Mar 29, 2025 at 2:25 PM Andres Freund <andres@anarazel.de> wrote:
>
> I think I found an issue with this one - as it stands the view was viewable by
> everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> too much for an unprivileged user to learn what part of a relation any other
> user is currently reading.
>
> There'd be two different ways to address that:
> 1) revoke view & function from public, grant to a limited role (presumably
>    pg_read_all_stats)
> 2) copy pg_stat_activity's approach of using something like
>
>    #define HAS_PGSTAT_PERMISSIONS(role)  (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
>
>    on a per-IO basis.

Is it easier to later change it to be more restrictive or less? If it
is easier to later lock it down more, then go with 2, otherwise go
with 1?

- Melanie

  154. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-29T21:29:29Z

    Flushing half-baked review comments before going offline for a few hours:

On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> Attached v2.13, with the following changes:

>   5) The WARNING in the callback is now a LOG, as it will be sent to the
>      client as a WARNING explicitly when the IO's results are processed
> 
>      I actually chose LOG_SERVER_ONLY - that seemed slightly better than just
>      LOG? But not at all sure.

LOG_SERVER_ONLY and its synonym COMMERR look to be used for:

- ProcessLogMemoryContextInterrupt()
- messages before successful authentication
- protocol sync loss, where we'd fail to send a client message
- client already gone

The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
If a client has decided to set client_min_messages that high, the client might
be interested in the fact that it got side-tracked completing someone else's
IO.  On the other hand, almost none of those sidetrack events will produce
messages.  The main argument I'd envision for LOG_SERVER_ONLY is that we
consider the message content sensitive, but I don't see the message content as
materially sensitive.

Since you committed LOG_SERVER_ONLY, let's keep that decision.  My last draft
review discouraged it, but it doesn't matter.  pgaio_result_report() should
assert elevel != LOG to avoid future divergence.

> - Previously the buffer completion callback checked zero_damaged_pages - but
>   that's not right, the GUC hopefully is only set on a per-session basis

Good catch.  I've now audited the complete_shared callbacks for other variable
references and actions not acceptable there.  I found nothing beyond what you
found by v2.14.

On Sat, Mar 29, 2025 at 10:48:10AM -0400, Andres Freund wrote:
> On 2025-03-29 06:41:43 -0700, Noah Misch wrote:
> > On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:

> Subject: [PATCH v2.14 01/29] Fix mis-attribution of checksum failure stats to
>  the wrong database

I've skipped reviewing this patch, since it's already committed.  If it needs
post-commit review, let me know.

> Subject: [PATCH v2.14 02/29] aio: Implement support for reads in smgr/md/fd

> +		/*
> +		 * Immediately log a message about the IO error, but only to the
> +		 * server log. The reason to do so immediately is that the originator
> +		 * might not process the query result immediately (because it is busy
> +		 * doing another part of query processing) or at all (e.g. if it was
> +		 * cancelled or errored out due to another IO also failing).  The
> +		 * issuer of the IO will emit an ERROR when processing the IO's

s/issuer/definer/ please, to avoid proliferating synonyms.  Likewise two other
places in the patches.

> +/*
> + * smgrstartreadv() -- asynchronous version of smgrreadv()
> + *
> + * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> + * `ioh` all parameters are the same as smgrreadv().

I would add a comment starting with:

  Compared to smgrreadv(), more responsibilities fall on layers above smgr.
  Higher layers handle partial reads.  smgr will ereport(LOG_SERVER_ONLY) some
  problems, but higher layers are responsible for pgaio_result_report() to
  mirror that news to the user and (for ERROR) abort the (sub)transaction.

md_readv_complete() comment "issuer of the IO will emit an ERROR" says some of
that, but someone adding a smgrstartreadv() call is less likely to find it
there.

I say "comment starting with", because I think there's a remaining decision
about who owns the zeroing currently tied to smgrreadv().  An audit of
mdreadv() vs. AIO counterparts found this part of mdreadv():

			if (nbytes == 0)
			{
				/*
				 * We are at or past EOF, or we read a partial block at EOF.
				 * Normally this is an error; upper levels should never try to
				 * read a nonexistent block.  However, if zero_damaged_pages
				 * is ON or we are InRecovery, we should instead return zeroes
				 * without complaining.  This allows, for example, the case of
				 * trying to update a block that was later truncated away.
				 */
				if (zero_damaged_pages || InRecovery)
				{

I didn't write a test to prove its absence, but I'm not finding such code in
the AIO path.  I wondered if we could just Assert(!InRecovery), but adding
that to md_readv_complete() failed 001_stream_rep.pl with this stack:

ExceptionalCondition at assert.c:52
md_readv_complete at md.c:2043
pgaio_io_call_complete_shared at aio_callback.c:258
pgaio_io_process_completion at aio.c:515
pgaio_io_perform_synchronously at aio_io.c:148
pgaio_io_stage at aio.c:453
pgaio_io_start_readv at aio_io.c:87
FileStartReadV at fd.c:2243
mdstartreadv at md.c:1005
smgrstartreadv at smgr.c:757
AsyncReadBuffers at bufmgr.c:1938
StartReadBuffersImpl at bufmgr.c:1422
StartReadBuffer at bufmgr.c:1515
ReadBuffer_common at bufmgr.c:1246
ReadBufferExtended at bufmgr.c:818
vm_readbuf at visibilitymap.c:584
visibilitymap_pin at visibilitymap.c:203
heap_xlog_insert at heapam_xlog.c:450
heap_redo at heapam_xlog.c:1195
ApplyWalRecord at xlogrecovery.c:1995
PerformWalRecovery at xlogrecovery.c:1825
StartupXLOG at xlog.c:5895

If this is a real problem, fix options may include:

- Implement the InRecovery zeroing for real.
- Make the InRecovery case somehow use real mdreadv(), not
  pgaio_io_perform_synchronously() to use AIO APIs with synchronous AIO.  I'll
  guess this is harder than the previous option, though.

> Subject: [PATCH v2.14 04/29] aio: Add pg_aios view

> +		/*
> +		 * There is no lock that could prevent the state of the IO to advance
> +		 * concurrently - and we don't want to introduce one, as that would
> +		 * introduce atomics into a very common path. Instead we
> +		 *
> +		 * 1) Determine the state + generation of the IO.
> +		 *
> +		 * 2) Copy the IO to local memory.
> +		 *
> +		 * 3) Check if state or generation of the IO changed. If the state
> +		 * changed, retry, if the generation changed don't display the IO.
> +		 */
> +
> +		/* 1) from above */
> +		start_generation = live_ioh->generation;
> +		pg_read_barrier();

I think "retry:" needs to be here, above start_state assignment.  Otherwise,
the "live_ioh->state != start_state" test will keep seeing a state mismatch.

> +		start_state = live_ioh->state;
> +
> +retry:
> +		if (start_state == PGAIO_HS_IDLE)
> +			continue;


> Subject: [PATCH v2.14 05/29] localbuf: Track pincount in BufferDesc as well
> Subject: [PATCH v2.14 07/29] aio: Add WARNING result status
> Subject: [PATCH v2.14 08/29] pgstat: Allow checksum errors to be reported in
>  critical sections
> Subject: [PATCH v2.14 09/29] Add errhint_internal()

Ready for commit


> Subject: [PATCH v2.14 10/29] bufmgr: Implement AIO read support

> Buffer reads executed this infrastructure will report invalid page / checksum
> errors / warnings differently than before:

s/this/through this/

> +	*zeroed_or_error_count = rem_error & ((1 << 7) - 1);
> +	rem_error >>= 7;

These raw "7" are good places to use your new #define values.  Likewise in
buffer_readv_encode_error().

> + *   that was errored or zerored or, if no errors/zeroes, the first ignored

s/zerored/zeroed/

> +	 * enough. If there is an error, the error is the integeresting offset,

typo "integeresting"

> +/*
> + * We need a backend-local completion callback for shared buffers, to be able
> + * to report checksum errors correctly. Unfortunately that can only safely
> + * happen if the reporting backend has previously called

Missing end of sentence.

> @@ -144,8 +144,8 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
>  	 */

There's an outdated comment ending here:

	/*
	 * Throw a WARNING if the checksum fails, but only after we've checked for
	 * the all-zeroes case.
	 */

>  	if (checksum_failure)
>  	{
> -		if ((flags & PIV_LOG_WARNING) != 0)
> -			ereport(WARNING,
> +		if ((flags & (PIV_LOG_WARNING | PIV_LOG_LOG)) != 0)
> +			ereport(flags & PIV_LOG_WARNING ? WARNING : LOG,


> Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
>  ignore_checksum_failure
> Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
> Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
> Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
>  support
> Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
>  wait time
> Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems

Ready for commit


> Subject: [PATCH v2.14 18/29] aio: Add test_aio module

I didn't yet re-review the v2.13 or 2.14 changes to this one.  That's still in
my queue.  One thing I noticed anyway:

> +# Tests using injection points. Mostly to exercise had IO errors that are

s/had/hard/


On Sat, Mar 29, 2025 at 02:25:15PM -0400, Andres Freund wrote:
> On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> > Attached is v2.14:

> > - push pg_aios view (depends a tiny bit on the smgr/md/fd change above)
> 
> I think I found an issue with this one - as it stands the view was viewable by
> everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> too much for an unprivileged user to learn what part of a relation any other
> user is currently reading.
> 
> There'd be two different ways to address that:
> 1) revoke view & function from public, grant to a limited role (presumably
>    pg_read_all_stats)
> 2) copy pg_stat_activity's approach of using something like
> 
>    #define HAS_PGSTAT_PERMISSIONS(role)	 (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
> 
>    on a per-IO basis.

No strong opinion.  I'm not really worried about any of this information
leaking.  Nothing in pg_aios comes close to the sensitivity of
pg_stat_activity.query.  pg_stat_activity is mighty cautious, hiding even
stuff like wait_event_type that I wouldn't worry about.  Hence, another valid
choice is (3) change nothing.

Meanwhile, I see substantially less need to monitor your own IOs than to
monitor your own pg_stat_activity rows, and even your own IOs potentially
reveal things happening in other sessions, e.g. evicting buffers that others
read and you never read.  So restrictions wouldn't be too painful, and (1)
arguably helps privacy more than (2).

I'd likely go with (1) today.

  155. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-30T00:39:54Z

    Hi,

On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> Flushing half-baked review comments before going offline for a few hours:
>
> On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> > Attached v2.13, with the following changes:
>
> >   5) The WARNING in the callback is now a LOG, as it will be sent to the
> >      client as a WARNING explicitly when the IO's results are processed
> >
> >      I actually chose LOG_SERVER_ONLY - that seemed slightly better than just
> >      LOG? But not at all sure.
>
> LOG_SERVER_ONLY and its synonym COMMERR look to be used for:
>
> - ProcessLogMemoryContextInterrupt()
> - messages before successful authentication
> - protocol sync loss, where we'd fail to send a client message
> - client already gone
>
> The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
> If a client has decided to set client_min_messages that high, the client might
> be interested in the fact that it got side-tracked completing someone else's
> IO.  On the other hand, almost none of those sidetrack events will produce
> messages.  The main argument I'd envision for LOG_SERVER_ONLY is that we
> consider the message content sensitive, but I don't see the message content as
> materially sensitive.

I don't think it's sensitive - it just seems a bit sillier to send the same
thing to the client twice than to the server log. I'm happy to change it to
LOG if you prefer. Your points below mean some comments need to be updated in
smgr/md.c anyway.


> > - Previously the buffer completion callback checked zero_damaged_pages - but
> >   that's not right, the GUC hopefully is only set on a per-session basis
>
> Good catch.  I've now audited the complete_shared callbacks for other variable
> references and actions not acceptable there.  I found nothing beyond what you
> found by v2.14.

I didn't find anything else either.



> > Subject: [PATCH v2.14 02/29] aio: Implement support for reads in smgr/md/fd
>
> > +		/*
> > +		 * Immediately log a message about the IO error, but only to the
> > +		 * server log. The reason to do so immediately is that the originator
> > +		 * might not process the query result immediately (because it is busy
> > +		 * doing another part of query processing) or at all (e.g. if it was
> > +		 * cancelled or errored out due to another IO also failing).  The
> > +		 * issuer of the IO will emit an ERROR when processing the IO's
>
> s/issuer/definer/ please, to avoid proliferating synonyms.  Likewise two other
> places in the patches.

Hm. Will do. Doesn't bother me personally, but happy to change it.


> > +/*
> > + * smgrstartreadv() -- asynchronous version of smgrreadv()
> > + *
> > + * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> > + * `ioh` all parameters are the same as smgrreadv().
>
> I would add a comment starting with:
>
>   Compared to smgrreadv(), more responsibilities fall on layers above smgr.
>   Higher layers handle partial reads.  smgr will ereport(LOG_SERVER_ONLY) some
>   problems, but higher layers are responsible for pgaio_result_report() to
>   mirror that news to the user and (for ERROR) abort the (sub)transaction.

Hm - if we document that in all the smgrstart* we'd end up with something like
that in a lot of places - but OTOH, this is the first one so far...



> I say "comment starting with", because I think there's a remaining decision
> about who owns the zeroing currently tied to smgrreadv().  An audit of
> mdreadv() vs. AIO counterparts found this part of mdreadv():
>
> 			if (nbytes == 0)
> 			{
> 				/*
> 				 * We are at or past EOF, or we read a partial block at EOF.
> 				 * Normally this is an error; upper levels should never try to
> 				 * read a nonexistent block.  However, if zero_damaged_pages
> 				 * is ON or we are InRecovery, we should instead return zeroes
> 				 * without complaining.  This allows, for example, the case of
> 				 * trying to update a block that was later truncated away.
> 				 */
> 				if (zero_damaged_pages || InRecovery)
> 				{
>
> I didn't write a test to prove its absence, but I'm not finding such code in
> the AIO path.

Yes, there is no such codepath

A while ago I had started a thread about whether the above codepath is
necessary, as the whole idea of putting a buffer into shared buffers that
doesn't exist on-disk is *extremely* ill conceived, it puts a buffer into
shared buffer that somehow wasn't readable on disk, *without* creating it on
disk. The problem is that an mdnblocks() wouldn't know about that
only-in-memory part of the relation and thus most parts of PG won't consider
that buffer to exist - it'd be just skipped in sequential scans etc, but then
it'd trigger errors when extending the relation ("unexpected data beyond
EOF"), etc.

I had planned to put in an error into mdreadv() at the time, but somehow lost
track of that - I kind of mentally put this issue into the "done" category :(

Based on my research, the InRecovery path is not reachable (most recovery
buffer reads go through XLogReadBufferExtended() which extends at that layer
files, and the exceptions like VM/FSM have explicit code to extend the
relation, c.f. vm_readbuf()). It actually looks to me like it *never* was
reachable, the XLogReadBufferExtended() predecessors, back to the initial
addition of WAL to PG, had that such an extension path, as did vm/fsm.

The zero_damaged_pages path hasn't reliably worked for a long time afaict,
because _mdfd_getseg() doesn't know about it (note we're not passing
EXTENSION_CREATE). So unless the buffer is just after the physical end of the
last segment, you'll just get an error at that point. To my knowledge we
haven't heard related complaints.

It makes some sense to have zero_damaged_pages for actually existing pages
reached from sequential / tid / COPY on the table level - after all that's the
only way you might get data out during data recovery. But those would never
reach this logic, as such scans rely on mdnblocks().  For index -> heap
fetches the option seems mainly dangerous, because that'll just create random
buffers in shared buffers that, as explained above, won't then be reached by
other scans.  And index scans during data recovery are not a good idea in the
first place, all that one should do in that situation is to dump out the data.


At the very least we need to add a comment about this though.  If we want to
implement it, it'd be easy enough, but I think that logic is so insane that I
think we shouldn't do it unless there is some *VERY* clear evidence that we
need it.



> I wondered if we could just Assert(!InRecovery), but adding that to
> md_readv_complete() failed 001_stream_rep.pl with this stack:

I'd expect that to fail in a lot of paths:
XLogReadBufferExtended() ->
ReadBufferWithoutRelcache() ->
ReadBuffer_common() ->
StartReadBuffer()


> > Subject: [PATCH v2.14 04/29] aio: Add pg_aios view
>
> > +		/*
> > +		 * There is no lock that could prevent the state of the IO to advance
> > +		 * concurrently - and we don't want to introduce one, as that would
> > +		 * introduce atomics into a very common path. Instead we
> > +		 *
> > +		 * 1) Determine the state + generation of the IO.
> > +		 *
> > +		 * 2) Copy the IO to local memory.
> > +		 *
> > +		 * 3) Check if state or generation of the IO changed. If the state
> > +		 * changed, retry, if the generation changed don't display the IO.
> > +		 */
> > +
> > +		/* 1) from above */
> > +		start_generation = live_ioh->generation;
> > +		pg_read_barrier();
>
> I think "retry:" needs to be here, above start_state assignment.  Otherwise,
> the "live_ioh->state != start_state" test will keep seeing a state mismatch.

Damn, you're right.


> > Subject: [PATCH v2.14 05/29] localbuf: Track pincount in BufferDesc as well
> > Subject: [PATCH v2.14 07/29] aio: Add WARNING result status
> > Subject: [PATCH v2.14 08/29] pgstat: Allow checksum errors to be reported in
> >  critical sections
> > Subject: [PATCH v2.14 09/29] Add errhint_internal()
>
> Ready for commit

Cool


> > Subject: [PATCH v2.14 10/29] bufmgr: Implement AIO read support
>
> > Buffer reads executed this infrastructure will report invalid page / checksum
> > errors / warnings differently than before:
>
> s/this/through this/

Fixed.


> > +	*zeroed_or_error_count = rem_error & ((1 << 7) - 1);
> > +	rem_error >>= 7;
>
> These raw "7" are good places to use your new #define values.  Likewise in
> buffer_readv_encode_error().

Which define value are you thinking of here? I don't think any of the ones I
added apply?  But I think you're right it'd be good to have some define for
it, at least locally.


> > + *   that was errored or zerored or, if no errors/zeroes, the first ignored
>
> s/zerored/zeroed/
>
> > +	 * enough. If there is an error, the error is the integeresting offset,
>
> typo "integeresting"

:(. Fixed.


> > +/*
> > + * We need a backend-local completion callback for shared buffers, to be able
> > + * to report checksum errors correctly. Unfortunately that can only safely
> > + * happen if the reporting backend has previously called
>
> Missing end of sentence.

It's now:

/*
 * We need a backend-local completion callback for shared buffers, to be able
 * to report checksum errors correctly. Unfortunately that can only safely
 * happen if the reporting backend has previously called
 * pgstat_prepare_report_checksum_failure(), which we can only guarantee in
 * the backend that started the IO. Hence this callback.
 */


> > @@ -144,8 +144,8 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
> >  	 */
>
> There's an outdated comment ending here:
>
> 	/*
> 	 * Throw a WARNING if the checksum fails, but only after we've checked for
> 	 * the all-zeroes case.
> 	 */

Updated to:
	/*
	 * Throw a WARNING/LOG, as instructed by PIV_LOG_*, if the checksum fails,
	 * but only after we've checked for the all-zeroes case.
	 */

I found one more, the newly added comment about checksum_failure_p was still
talking about ignore_checksum_failure, but it should now be IGNORE_CHECKSUM_FAILURE.


> > Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
> >  ignore_checksum_failure
> > Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
> > Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
> > Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
> >  support
> > Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
> >  wait time
> > Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems
>
> Ready for commit

Cool.



> > Subject: [PATCH v2.14 18/29] aio: Add test_aio module
>
> I didn't yet re-review the v2.13 or 2.14 changes to this one.  That's still in
> my queue.

That's good - I think some of the tests need to expand a bit more. Since
that's at the end of the dependency chain...


> One thing I noticed anyway:

> > +# Tests using injection points. Mostly to exercise had IO errors that are
>
> s/had/hard/

Fixed.


> On Sat, Mar 29, 2025 at 02:25:15PM -0400, Andres Freund wrote:
> > On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> > > Attached is v2.14:
>
> > > - push pg_aios view (depends a tiny bit on the smgr/md/fd change above)
> >
> > I think I found an issue with this one - as it stands the view was viewable by
> > everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> > too much for an unprivileged user to learn what part of a relation any other
> > user is currently reading.
> >
> > There'd be two different ways to address that:
> > 1) revoke view & function from public, grant to a limited role (presumably
> >    pg_read_all_stats)
> > 2) copy pg_stat_activity's approach of using something like
> >
> >    #define HAS_PGSTAT_PERMISSIONS(role)	 (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
> >
> >    on a per-IO basis.
>
> No strong opinion.

Same.


> I'm not really worried about any of this information leaking.  Nothing in
> pg_aios comes close to the sensitivity of pg_stat_activity.query.
> pg_stat_activity is mighty cautious, hiding even stuff like wait_event_type
> that I wouldn't worry about.  Hence, another valid choice is (3) change
> nothing.

I'd also be on board with that.


> Meanwhile, I see substantially less need to monitor your own IOs than to
> monitor your own pg_stat_activity rows, and even your own IOs potentially
> reveal things happening in other sessions, e.g. evicting buffers that others
> read and you never read.  So restrictions wouldn't be too painful, and (1)
> arguably helps privacy more than (2).
>
> I'd likely go with (1) today.

Sounds good to me. It also has the advantage of being much easier to test than
2).

Greetings,

Andres Freund

  156. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-03-30T02:45:13Z

    On Sat, Mar 29, 2025 at 08:39:54PM -0400, Andres Freund wrote:
> On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> > On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:

> > The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
> > If a client has decided to set client_min_messages that high, the client might
> > be interested in the fact that it got side-tracked completing someone else's
> > IO.  On the other hand, almost none of those sidetrack events will produce
> > messages.  The main argument I'd envision for LOG_SERVER_ONLY is that we
> > consider the message content sensitive, but I don't see the message content as
> > materially sensitive.
> 
> I don't think it's sensitive - it just seems a bit sillier to send the same
> thing to the client twice than to the server log.

Ah, that adds weight to the benefit of LOG_SERVER_ONLY.

> I'm happy to change it to
> LOG if you prefer. Your points below mean some comments need to be updated in
> smgr/md.c anyway.

Nah.

> > > +/*
> > > + * smgrstartreadv() -- asynchronous version of smgrreadv()
> > > + *
> > > + * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> > > + * `ioh` all parameters are the same as smgrreadv().
> >
> > I would add a comment starting with:
> >
> >   Compared to smgrreadv(), more responsibilities fall on layers above smgr.
> >   Higher layers handle partial reads.  smgr will ereport(LOG_SERVER_ONLY) some
> >   problems, but higher layers are responsible for pgaio_result_report() to
> >   mirror that news to the user and (for ERROR) abort the (sub)transaction.
> 
> Hm - if we document that in all the smgrstart* we'd end up with something like
> that in a lot of places - but OTOH, this is the first one so far...

Alternatively, to avoid duplication:

  See $PLACE for the tasks that the caller's layer owns, in contrast to smgr
  owning them for smgrreadv().

> > I say "comment starting with", because I think there's a remaining decision
> > about who owns the zeroing currently tied to smgrreadv().  An audit of
> > mdreadv() vs. AIO counterparts found this part of mdreadv():
> >
> > 			if (nbytes == 0)
> > 			{
> > 				/*
> > 				 * We are at or past EOF, or we read a partial block at EOF.
> > 				 * Normally this is an error; upper levels should never try to
> > 				 * read a nonexistent block.  However, if zero_damaged_pages
> > 				 * is ON or we are InRecovery, we should instead return zeroes
> > 				 * without complaining.  This allows, for example, the case of
> > 				 * trying to update a block that was later truncated away.
> > 				 */
> > 				if (zero_damaged_pages || InRecovery)
> > 				{
> >
> > I didn't write a test to prove its absence, but I'm not finding such code in
> > the AIO path.
> 
> Yes, there is no such codepath
> 
> A while ago I had started a thread about whether the above codepath is
> necessary

postgr.es/m/3qxxsnciyffyf3wyguiz4besdp5t5uxvv3utg75cbcszojlz7p@uibfzmnukkbd
which I had forgotten completely.

I've redone your audit, and I agree the InRecovery case is dead code.
check-world InRecovery reaches mdstartreadv() and mdreadv() only via
XLogReadBufferExtended(), vm_readbuf(), and fsm_readbuf().

The zero_damaged_pages case entails more of a judgment call about whether or
not its rule breakage eclipses its utility.  Fortunately, a disappointed
zero_damaged_pages user could work around that code's absence by stopping the
server and using "dd" to extend the segment with zeros.

> I had planned to put in an error into mdreadv() at the time, but somehow lost
> track of that - I kind of mentally put this issue into the "done" category :(

> At the very least we need to add a comment about this though.

I'm fine with either of:

1. Replace that mdreadv() code with an error.

2. Update comment in mdreadv() that we're phasing out this code due the
   InRecovery case's dead code status and the zero_damaged_pages problems; AIO
   intentionally doesn't replicate it.  Maybe add Assert(false).

I'd do (2) for v18, then do (1) first thing in v19 development.

> > > +	*zeroed_or_error_count = rem_error & ((1 << 7) - 1);
> > > +	rem_error >>= 7;
> >
> > These raw "7" are good places to use your new #define values.  Likewise in
> > buffer_readv_encode_error().
> 
> Which define value are you thinking of here? I don't think any of the ones I
> added apply?  But I think you're right it'd be good to have some define for
> it, at least locally.

It was just my imagination.  Withdrawn.

> It's now:
> 
> /*
>  * We need a backend-local completion callback for shared buffers, to be able
>  * to report checksum errors correctly. Unfortunately that can only safely
>  * happen if the reporting backend has previously called
>  * pgstat_prepare_report_checksum_failure(), which we can only guarantee in
>  * the backend that started the IO. Hence this callback.
>  */

Sounds good.

> Updated to:
> 	/*
> 	 * Throw a WARNING/LOG, as instructed by PIV_LOG_*, if the checksum fails,
> 	 * but only after we've checked for the all-zeroes case.
> 	 */
> 
> I found one more, the newly added comment about checksum_failure_p was still
> talking about ignore_checksum_failure, but it should now be IGNORE_CHECKSUM_FAILURE.

That works.

> > > Subject: [PATCH v2.14 18/29] aio: Add test_aio module
> >
> > I didn't yet re-review the v2.13 or 2.14 changes to this one.  That's still in
> > my queue.
> 
> That's good - I think some of the tests need to expand a bit more. Since
> that's at the end of the dependency chain...

Okay, I'll delay on re-reviewing that one.  When it's a good time, please put
the CF entry back in Needs Review.  The patches before it are all ready for
commit after the above points of this mail.

  157. Re: AIO v2.5

    Melanie Plageman <melanieplageman@gmail.com> — 2025-03-30T18:23:20Z

    On Tue, Mar 25, 2025 at 11:58 AM Andres Freund <andres@anarazel.de> wrote:
>
> > Another thought on complete_shared running on other backends: I wonder if we
> > should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> > other process" or similar, so people wonder less about how "SELECT FROM a" led
> > to a log message about IO on table "b".
>
> I've been wondering about that as well, and yes, we probably should.
>
> I'd add the pid of the backend that started the IO to the message - although
> need to check whether we're trying to keep PIDs of other processes from
> unprivileged users.
>
> I think we probably should add a similar, but not equivalent, context in io
> workers. Maybe "I/O worker executing I/O on behalf of process %d".

I think this has not yet been done. Attached patch is an attempt to
add error context for IO completions by another backend when using
io_uring and IO processing in general by an IO worker. It seems to
work -- that is, running the test_aio tests, you can see the context
in the logs.

I'm not certain that I did this in the way you both were envisioning, though.

- Melanie

  158. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-30T23:13:56Z

    Hi,

On 2025-03-27 10:52:10 +1300, Thomas Munro wrote:
> On Thu, Mar 27, 2025 at 10:41 AM Andres Freund <andres@anarazel.de> wrote:
> > > > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> > >
> > > Consider also updating this comment to stop focusing on prefetch; I think
> > > changing that aligns with the patch's other changes:
> > >
> > > /*
> > >  * How many buffers PrefetchBuffer callers should try to stay ahead of their
> > >  * ReadBuffer calls by.  Zero means "never prefetch".  This value is only used
> > >  * for buffers not belonging to tablespaces that have their
> > >  * effective_io_concurrency parameter set.
> > >  */
> > > int                   effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;
> >
> > Good point.  Although I suspect it might be worth adjusting this, and also the
> > config.sgml bit about effective_io_concurrency separately. That seems like it
> > might take an iteration or two.
> 
> +1 for rewriting that separately from this work on the code (I can
> have a crack at that if you want).

You taking a crack at that would be appreciated!

> For the comment, my suggestion would be something like:
> 
> "Default limit on the level of concurrency that each I/O stream
> (currently, ReadStream but in future other kinds of streams) can use.
> Zero means that I/O is always performed synchronously, ie not
> concurrently with query execution. This value can be overridden at the
> tablespace level with the parameter of the same name. Note that
> streams performing I/O not classified as single-session work respect
> maintenance_io_concurrency instead."

Generally sounds good. I do wonder if the last sentence could be made a bit
simpler, it took me a few seconds to parse "not classified as single-session".

Maybe "classified as performing work for multiple sessions respect
maintenance_io_concurrency instead."?

Greetings,

Andres Freund

  159. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-03-30T23:46:57Z

    Hi,

On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> > Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
> >  ignore_checksum_failure
> > Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> > Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
> > Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
> >  support
> > Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
> >  wait time
> > Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems
>
> Ready for commit

I've pushed most of these after some very light further editing.  Yay.  Thanks
a lot for all the reviews!

So far the buildfarm hasn't been complaining, but it's early days.


I didn't yet push

> > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design

because I want to integrate some language that could be referenced by
smgrstartreadv() (and more in the future), as we have been talking about.


Tomorrow I'll work on sending out a new version with the remaining patches. I
plan for that version to have:

- pg_aios view with the security checks (already done, trivial)

- a commit with updated language for smgrstartreadv(), probably referencing
  aio's README

- a change to mdreadv() around zero_damaged_pages, as Noah and I have been
  discussing

- updated tests, with the FIXMEs etc addressed

- a reviewed version of the errcontext callback patch that Melanie sent
  earlier today


Todo beyond that:

- The comment and docs updates we've been discussing in
  https://postgr.es/m/5fc6r4smanncsmqw7ib6s3uj6eoiqoioxbd5ibmhtqimcggtoe%40fyrok2gozsoq

- I think a good long search through the docs is in order, there probably are
  other things that should be updated, beyond concrete references to
  effective_io_concurrency etc.

- Whether we should do something, and if so what, about BAS_BULKREAD for
  18. Thomas may have some thoughts / code.

- Whether there's anything committable around Jelte's RLIMIT_NOFILE changes.


Greetings,

Andres Freund

  160. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-01T00:41:39Z

    Hi,

On 2025-03-30 19:46:57 -0400, Andres Freund wrote:
> I didn't yet push
>
> > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
>
> because I want to integrate some language that could be referenced by
> smgrstartreadv() (and more in the future), as we have been talking about.

I tried a bunch of variations and none of them seemed great. So I ended up
with a lightly polished version of your suggested comment above
smgrstartreadv().  We can later see about generalizing it.


> Tomorrow I'll work on sending out a new version with the remaining patches. I
> plan for that version to have:

Got a bit distracted with $work stuff today, but here we go.


The updated version has all of that:

> - pg_aios view with the security checks (already done, trivial)
>
> - a commit with updated language for smgrstartreadv(), probably referencing
>   aio's README
>
> - a change to mdreadv() around zero_damaged_pages, as Noah and I have been
>   discussing
>
> - updated tests, with the FIXMEs etc addressed
>
> - a reviewed version of the errcontext callback patch that Melanie sent
>   earlier today

Although I didn't actually find anything in that, other than one unnecessary
change.

Greetings,

Andres Freund

  161. Re: AIO v2.5

    Aleksander Alekseev <aleksander@timescale.com> — 2025-04-01T11:56:17Z

    Hi Andres,

> > I didn't yet push
> >
> > > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design

I have several notes about 0003 / README.md:

1. I noticed that the use of "Postgres" and "postgres" is inconsistent.

2.

```
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
```

Perhaps I'm a bit late here, but the name of the function is weird. It
registers a single callback, but the name is "_callbacks".

3. The use of "AIO Handle" and "AioHandle" is inconsistent.

4.

- pgaio_io_register_callbacks
- pgaio_io_set_handle_data_32

If I understand correctly one can register multiple callbacks per one
AIO Handle (right? ...). However I don't see an obvious way to match
handle data to the given callback. If all the callbacks get the same
handle data... well it's weird IMO, but we should explicitly say that.
On top of that we should probably explain in which order the callbacks
are going to be executed. If there are any guarantees in this respect
of course.

5. pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1)

Perhaps it's worth mentioning if `buffer` can be freed after the call
i.e. if it's stored by value or by reference. It's also worth
clarifying if the maximum number of buffers is limited or not.

6. It is worth clarifying if AIO allows reads and writes or only reads
at the moment. Perhaps it's also worth explicitly saying that AIO is
for disk IO only, not for network one.

7. It is worth clarifying how many times the callbacks are called when
reading multiple buffers. Is it guaranteed that the callbacks are
called ones, or if it somehow depends on the implementation, and also
what happens in the case if I/O succeeds partially.

8. I believe we should tell a bit more about the context in which the
callbacks are called. Particularly what happens to the memory contexts
and if I can allocate/free memory, can I throw ERRORs, can I create
new AIO Handles, is it expected that the callback should return
quickly, are the signals masked while the callback is executed, can I
use sockets, is it guaranteed that the callback is going to be called
in the same process (I guess so, but the text doesn't explicitly
promise that), etc.

9.

```
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed.
```

What pgaio_io_acquire() does if we are out of AIO Handles? Since it
always succeeds I guess it should block the caller in this case, but
IMO we should say this explicitly.

10.

> > because I want to integrate some language that could be referenced by
> > smgrstartreadv() (and more in the future), as we have been talking about.
>
> I tried a bunch of variations and none of them seemed great. So I ended up
> with a lightly polished version of your suggested comment above
> smgrstartreadv().  We can later see about generalizing it.

IMO the problem here is that README doesn't show the code that does IO
per se, and thus doesn't give the full picture of how AIO should be
used. Perhaps instead of referencing smgrstartreadv() it would be
better to provide a simple but complete example, one that opens a
binary file and reads 512 bytes from it by the given offset for
instance.

-- 
Best regards,
Aleksander Alekseev

  162. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-01T13:45:23Z

    Hi,

On 2025-04-01 14:56:17 +0300, Aleksander Alekseev wrote:
> Hi Andres,
>
> > > I didn't yet push
> > >
> > > > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
>
> I have several notes about 0003 / README.md:
>
> 1. I noticed that the use of "Postgres" and "postgres" is inconsistent.

:/

It probably should be consistent, but I have no idea which of the spellings we
should go for. Either looks ugly in some contexts.


> 2.
>
> ```
> +pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
> ```
>
> Perhaps I'm a bit late here, but the name of the function is weird. It
> registers a single callback, but the name is "_callbacks".

It registers a *set* of callbacks (stage, complete_shared, complete_local,
report_error) on the handle.



> 3. The use of "AIO Handle" and "AioHandle" is inconsistent.

This seems ok to me.


> 4.
>
> - pgaio_io_register_callbacks
> - pgaio_io_set_handle_data_32
>
> If I understand correctly one can register multiple callbacks per one
> AIO Handle (right? ...). However I don't see an obvious way to match
> handle data to the given callback. If all the callbacks get the same
> handle data... well it's weird IMO, but we should explicitly say that.

There is:

/*
 * Associate an array of data with the Handle. This is e.g. useful to the
 * transport knowledge about which buffers a multi-block IO affects to
 * completion callbacks.
 *
 * Right now this can be done only once for each IO, even though multiple
 * callbacks can be registered. There aren't any known usecases requiring more
 * and the required amount of shared memory does add up, so it doesn't seem
 * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
 */


> On top of that we should probably explain in which order the callbacks
> are going to be executed. If there are any guarantees in this respect
> of course.

Alongside PgAioHandleCallbacks:
	 *
	 * The latest registered callback is called first. This allows
	 * higher-level code to register callbacks that can rely on callbacks
	 * registered by lower-level code to already have been executed.
	 *


> 5. pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1)
>
> Perhaps it's worth mentioning if `buffer` can be freed after the call
> i.e. if it's stored by value or by reference.

By value.


> It's also worth clarifying if the maximum number of buffers is limited or
> not.

It's limited to PG_IOV_MAX, fwiw.


> 6. It is worth clarifying if AIO allows reads and writes or only reads
> at the moment.

We have patches for writes, I just ran out of time for 18. Im not particularly
excited about adding stuff that then needs to be removed in 19.


> Perhaps it's also worth explicitly saying that AIO is for disk IO only, not
> for network one.

I'd like to integrate network IO too.  I have a local prototype, fwiw.


> 7. It is worth clarifying how many times the callbacks are called when
> reading multiple buffers. Is it guaranteed that the callbacks are
> called ones, or if it somehow depends on the implementation, and also
> what happens in the case if I/O succeeds partially.

The aio subsystem doesn't know anything about buffers.  Callbacks are executed
exactly once, with the exception of the error reporting callback, which could
be called multiple times.


> 8. I believe we should tell a bit more about the context in which the
> callbacks are called. Particularly what happens to the memory contexts
> and if I can allocate/free memory, can I throw ERRORs, can I create
> new AIO Handles, is it expected that the callback should return
> quickly, are the signals masked while the callback is executed, can I
> use sockets, is it guaranteed that the callback is going to be called
> in the same process (I guess so, but the text doesn't explicitly
> promise that), etc.

There is the following above pgaio_io_register_callbacks()

 * Note that callbacks are executed in critical sections.  This is necessary
 * to be able to execute IO in critical sections (consider e.g. WAL
 * logging). To perform AIO we first need to acquire a handle, which, if there
 * are no free handles, requires waiting for IOs to complete and to execute
 * their completion callbacks.
 *
 * Callbacks may be executed in the issuing backend but also in another
 * backend (because that backend is waiting for the IO) or in IO workers (if
 * io_method=worker is used).

And also a bunch of detail along struct PgAioHandleCallbacks.


> 9.
>
> ```
> +Because acquisition of an IO handle
> +[must always succeed](#io-can-be-started-in-critical-sections)
> +and the number of AIO Handles
> +[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
> +AIO handles can be reused as soon as they have completed.
> ```
>
> What pgaio_io_acquire() does if we are out of AIO Handles? Since it
> always succeeds I guess it should block the caller in this case, but
> IMO we should say this explicitly.

That's documented above pgaio_io_acquire().


> 10.
>
> > > because I want to integrate some language that could be referenced by
> > > smgrstartreadv() (and more in the future), as we have been talking about.
> >
> > I tried a bunch of variations and none of them seemed great. So I ended up
> > with a lightly polished version of your suggested comment above
> > smgrstartreadv().  We can later see about generalizing it.
>
> IMO the problem here is that README doesn't show the code that does IO
> per se, and thus doesn't give the full picture of how AIO should be
> used. Perhaps instead of referencing smgrstartreadv() it would be
> better to provide a simple but complete example, one that opens a
> binary file and reads 512 bytes from it by the given offset for
> instance.

IMO the example is already long enough, if you want that level of detail, you
can just look at smgrstartreadv() etc. The idea about explaining it at that
level is that that is basically what is required to use AIO in a new place,
whereas implementing AIO for a new target, or a new IO operation, requires a
bit more care.

Greetings,

Andres Freund

  163. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-04-01T15:11:59Z

    On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> updated version

All non-write patches (1-7) are ready for commit, though I have some cosmetic
recommendations below.  I've marked the commitfest entry Ready for Committer.

> +		# Check a page validity error in another block, to ensure we report
> +		# the correct block number
> +		$psql_a->query_safe(
> +			qq(
> +SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
> +));
> +		psql_like(
> +			$io_method,
> +			$psql_a,
> +			"$persistency: test zeroing of invalid block 3",
> +			qq(SELECT read_rel_block_ll('tbl_zero', 3, zero_on_error=>true);),
> +			qr/^$/,
> +			qr/^psql:<stdin>:\d+: WARNING:  invalid page in block 3 of relation base\/.*\/.*; zeroing out page$/
> +		);
> +
> +
> +		# Check a page validity error in another block, to ensure we report
> +		# the correct block number

This comment is a copy of the previous test's comment.  While the comment is
not false, consider changing it to:

		# Check one read reporting multiple invalid blocks.

> +		$psql_a->query_safe(
> +			qq(
> +SELECT modify_rel_block('tbl_zero', 2, corrupt_header=>true);
> +SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
> +));
> +		# First test error
> +		psql_like(
> +			$io_method,
> +			$psql_a,
> +			"$persistency: test reading of invalid block 2,3 in larger read",
> +			qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)),
> +			qr/^$/,
> +			qr/^psql:<stdin>:\d+: ERROR:  2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL:  Block 2 held first invalid page\.\nHINT:[^\n]+$/
> +		);
> +
> +		# Then test zeroing via ZERO_ON_ERROR flag
> +		psql_like(
> +			$io_method,
> +			$psql_a,
> +			"$persistency: test zeroing of invalid block 2,3 in larger read, ZERO_ON_ERROR",
> +			qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>true)),
> +			qr/^$/,
> +			qr/^psql:<stdin>:\d+: WARNING:  zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL:  Block 2 held first zeroed page\.\nHINT:[^\n]+$/
> +		);
> +
> +		# Then test zeroing vio zero_damaged_pages

s/vio/via/

> +# Verify checksum handling when creating database from an invalid database.
> +# This also serves as a minimal check that cross-database IO is handled
> +# reasonably.

To me, "invalid database" is a term of art from the message "cannot connect to
invalid database".  Hence, I would change "invalid database" to "database w/
invalid block" or similar, here and below.  (Alternatively, just delete "from
an invalid database".  It's clear from the context.)

> +	if (corrupt_checksum)
> +	{
> +		bool		successfully_corrupted = 0;
> +
> +		/*
> +		 * Any single modification of the checksum could just end up being
> +		 * valid again. To be sure
> +		 */

Unfinished sentence.  That said, I'm not following why we'd need this loop.
If this test code were changing the input to the checksum, it's true that an
input bit flip might reach the same pd_checksum.  The test case is changing
pd_checksum, not the input bits.  I don't see how changing pd_checksum could
leave the page still valid.  There's only one valid pd_checksum value for a
given input page.

> +			/*
> +			 * The underlying IO actually completed OK, and thus the "invalid"
> +			 * portion of the IOV actually contains valid data. That can hide
> +			 * a lot of problems, e.g. if we were to wrongly mark a buffer,
> +			 * that wasn't read according to the shortened-read, IO as valid,
> +			 * the contents would look valid and we might miss a bug.

Minimally s/read, IO/read IO,/ but I'd edit a bit further:

			 * a lot of problems, e.g. if we were to wrongly mark-valid a
			 * buffer that wasn't read according to the shortened-read IO, the
			 * contents would look valid and we might miss a bug.

> Subject: [PATCH v2.15 05/18] md: Add comment & assert to buffer-zeroing path
>  in md[start]readv()

> The zero_damaged_pages path is incomplete, as as missing segments are not

s/as as/as/

> For now, put an Assert(false) comments documenting this choice into mdreadv()

s/comments/and comments/

> +				 * For PG 18, we are putting an Assert(false) in into
> +				 * mdreadv() (triggering failures in assertion-enabled builds,

s/in into/in/

> Subject: [PATCH v2.15 06/18] aio: comment polishing

> + * - Partial reads need to be handle by the caller re-issuing IO for the
> + *   unread blocks

s/handle/handled/

> Subject: [PATCH v2.15 07/18] aio: Add errcontext for processing I/Os for
>  another backend

  164. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-01T15:55:20Z

    Hi,

On 2025-04-01 08:11:59 -0700, Noah Misch wrote:
> On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> > updated version
>
> All non-write patches (1-7) are ready for commit, though I have some cosmetic
> recommendations below.  I've marked the commitfest entry Ready for Committer.

Thanks!

I haven't yet pushed the changes, but will work on that in the afternoon.

I plan to afterwards close the CF entry and will eventually create a new one
for write support, although probably only rebasing onto
https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf%40gcnactj4z56m
and addressing some of the locking issues.

WRT the locking issues, I've been wondering whether we could make
LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
Probably better to get rid of the LWLock*Var functions and go for the approach
I had in v1, namely a version of LWLockAcquire() with a callback that gets
called between LWLockQueueSelf() and PGSemaphoreLock(), which can cause the
lock acquisition to abort.


> This comment is a copy of the previous test's comment.  While the comment is
> not false, consider changing it to:
>
> 		# Check one read reporting multiple invalid blocks.

> > +		# Then test zeroing vio zero_damaged_pages
>
> s/vio/via/
>

These make sense.


> > +# Verify checksum handling when creating database from an invalid database.
> > +# This also serves as a minimal check that cross-database IO is handled
> > +# reasonably.
>
> To me, "invalid database" is a term of art from the message "cannot connect to
> invalid database".  Hence, I would change "invalid database" to "database w/
> invalid block" or similar, here and below.  (Alternatively, just delete "from
> an invalid database".  It's clear from the context.)

Yea, I agree, this is easy to misunderstand when stepping back. I went for "with
an invalid block".


> > +	if (corrupt_checksum)
> > +	{
> > +		bool		successfully_corrupted = 0;
> > +
> > +		/*
> > +		 * Any single modification of the checksum could just end up being
> > +		 * valid again. To be sure
> > +		 */
>
> Unfinished sentence.

Oops. See below.


> That said, I'm not following why we'd need this loop.  If this test code
> were changing the input to the checksum, it's true that an input bit flip
> might reach the same pd_checksum.  The test case is changing pd_checksum,
> not the input bits.

We might be changing the input, due to the zero/corrupt_header options. Or we
might be called on a page that is *already* corrupted. I did encounter that
situation once while writing tests, where the tests only passed if I made the
+ 1 a + 2. Which was, uh, rather confusing and left me feel like I was cursed
that day.


> I don't see how changing pd_checksum could leave the
> page still valid.  There's only one valid pd_checksum value for a given
> input page.

I updated the comment to:
		/*
		 * Any single modification of the checksum could just end up being
		 * valid again, due to e.g. corrupt_header changing the data in a way
		 * that'd result in the "corrupted" checksum, or the checksum already
		 * being invalid. Retry in that, unlikely, case.
		 */


> > +			/*
> > +			 * The underlying IO actually completed OK, and thus the "invalid"
> > +			 * portion of the IOV actually contains valid data. That can hide
> > +			 * a lot of problems, e.g. if we were to wrongly mark a buffer,
> > +			 * that wasn't read according to the shortened-read, IO as valid,
> > +			 * the contents would look valid and we might miss a bug.
>
> Minimally s/read, IO/read IO,/ but I'd edit a bit further:
>
> 			 * a lot of problems, e.g. if we were to wrongly mark-valid a
> 			 * buffer that wasn't read according to the shortened-read IO, the
> 			 * contents would look valid and we might miss a bug.

Adopted.


> > Subject: [PATCH v2.15 05/18] md: Add comment & assert to buffer-zeroing path
> >  in md[start]readv()
>
> > The zero_damaged_pages path is incomplete, as as missing segments are not
>
> s/as as/as/
>
> > For now, put an Assert(false) comments documenting this choice into mdreadv()
>
> s/comments/and comments/
>
> > +				 * For PG 18, we are putting an Assert(false) in into
> > +				 * mdreadv() (triggering failures in assertion-enabled builds,
>
> s/in into/in/

> > Subject: [PATCH v2.15 06/18] aio: comment polishing
>
> > + * - Partial reads need to be handle by the caller re-issuing IO for the
> > + *   unread blocks
>
> s/handle/handled/

All adopted.  I'm sorry that you had to see so much of tiredness-enhanced
dyslexia :(.

Greetings,

Andres Freund

  165. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-04-01T16:07:27Z

    On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote:
> On 2025-04-01 08:11:59 -0700, Noah Misch wrote:
> > On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:

> I haven't yet pushed the changes, but will work on that in the afternoon.
> 
> I plan to afterwards close the CF entry and will eventually create a new one
> for write support, although probably only rebasing onto
> https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf%40gcnactj4z56m
> and addressing some of the locking issues.

Sounds good.

> WRT the locking issues, I've been wondering whether we could make
> LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
> Probably better to get rid of the LWLock*Var functions and go for the approach
> I had in v1, namely a version of LWLockAcquire() with a callback that gets
> called between LWLockQueueSelf() and PGSemaphoreLock(), which can cause the
> lock acquisition to abort.

What are the best thing(s) to read to understand the locking issues?

> > > +# Verify checksum handling when creating database from an invalid database.
> > > +# This also serves as a minimal check that cross-database IO is handled
> > > +# reasonably.
> >
> > To me, "invalid database" is a term of art from the message "cannot connect to
> > invalid database".  Hence, I would change "invalid database" to "database w/
> > invalid block" or similar, here and below.  (Alternatively, just delete "from
> > an invalid database".  It's clear from the context.)
> 
> Yea, I agree, this is easy to misunderstand when stepping back. I went for "with
> an invalid block".

Sounds good.

> > > +	if (corrupt_checksum)
> > > +	{
> > > +		bool		successfully_corrupted = 0;
> > > +
> > > +		/*
> > > +		 * Any single modification of the checksum could just end up being
> > > +		 * valid again. To be sure
> > > +		 */
> >
> > Unfinished sentence.
> 
> Oops. See below.
> 
> 
> > That said, I'm not following why we'd need this loop.  If this test code
> > were changing the input to the checksum, it's true that an input bit flip
> > might reach the same pd_checksum.  The test case is changing pd_checksum,
> > not the input bits.
> 
> We might be changing the input, due to the zero/corrupt_header options. Or we
> might be called on a page that is *already* corrupted. I did encounter that
> situation once while writing tests, where the tests only passed if I made the
> + 1 a + 2. Which was, uh, rather confusing and left me feel like I was cursed
> that day.

Got it.

> > I don't see how changing pd_checksum could leave the
> > page still valid.  There's only one valid pd_checksum value for a given
> > input page.
> 
> I updated the comment to:
> 		/*
> 		 * Any single modification of the checksum could just end up being
> 		 * valid again, due to e.g. corrupt_header changing the data in a way
> 		 * that'd result in the "corrupted" checksum, or the checksum already
> 		 * being invalid. Retry in that, unlikely, case.
> 		 */

Works for me.

  166. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-01T16:51:53Z

    Hi,

On 2025-04-01 09:07:27 -0700, Noah Misch wrote:
> On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote:
> > WRT the locking issues, I've been wondering whether we could make
> > LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
> > Probably better to get rid of the LWLock*Var functions and go for the approach
> > I had in v1, namely a version of LWLockAcquire() with a callback that gets
> > called between LWLockQueueSelf() and PGSemaphoreLock(), which can cause the
> > lock acquisition to abort.
>
> What are the best thing(s) to read to understand the locking issues?

Unfortunately I think it's our discussion from a few days/weeks ago.

The problem basically is that functions like LockBuffer(EXCLUSIVE) need to be able
to non-racily

a) wait for in-fligth IOs
b) acquire the content lock

If you just do it naively like this:

    else if (mode == BUFFER_LOCK_EXCLUSIVE)
    {
        if (pg_atomic_read_u32(&buf->state) &_IO_IN_PROGRESS)
            WaitIO(buf);
        LWLockAcquire(content_lock, LW_EXCLUSIVE);
   }

you obviously could have another backend start new IO between the WaitIO() and
the LWLockAcquire().  If that other backend then doesn't consume the
completion of that IO, the current backend could end up endlessly waiting for
the IO.  I don't see a way to avoid with narrow changes just to LockBuffer().


We need some infrastructure that allows to avoid that issue.  One approach
could be to integrate more tightly with lwlock.c. If

1) anyone starting IO were to wake up all waiters for the LWLock

2) The waiting side checked that there is no IO in progress *after*
   LWLockQueueSelf(), but before PGSemaphoreLock()

The backend doing LockBuffer() would be guaranteed to have the chance to wait
for the IO, rather than the lwlock.


But there might be better approaches.

I'm not really convinced that using generic lwlocks for buffer locking is the
best idea. There's just too many special things about buffers. E.g. we have
rather massive NUMA scalability issues due to the amount of lock traffic from
buffer header and content lock atomic operations, particuly on things like the
uppermost levels of a btree.  I've played with ideas like super-pinning and
locking btree root pages, which move all the overhead to the side that wants
to exclusively lock such a page - but that doesn't really make sense for
lwlocks in general.


Greetings,

Andres Freund

  167. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-01T21:47:51Z

    Hi,

On 2025-04-01 11:55:20 -0400, Andres Freund wrote:
> I haven't yet pushed the changes, but will work on that in the afternoon.

There are three different types of failures in the test_aio test so far:

1) TEMP_CONFIG

See https://postgr.es/m/zh5u22wbpcyfw2ddl3lsvmsxf4yvsrvgxqwwmfjddc4c2khsgp%40gfysyjsaelr5


2) Failure on at least some windows BF machines:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2025-04-01%2020%3A15%3A19
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2025-04-01%2019%3A03%3A07

Afaict the error is independent of AIO, instead just related CREATE DATABASE
... STRATEGY wal_log failing on windows.  In contrast to dropdb(), which does

    /*
     * Force a checkpoint to make sure the checkpointer has received the
     * message sent by ForgetDatabaseSyncRequests.
     */
    RequestCheckpoint(CHECKPOINT_IMMEDIATE | CHECKPOINT_FORCE | CHECKPOINT_WAIT);

    /* Close all smgr fds in all backends. */
    WaitForProcSignalBarrier(EmitProcSignalBarrier(PROCSIGNAL_BARRIER_SMGRRELEASE));

createdb_failure_callback() does no such thing.  But it's rather likely that
we, bgwriter, checkpointer (and now IO workers) have files open for the target
database.

Note that the test is failing even with "io_method=sync", which obviously
doesn't use IO workers, so it's not related to that.


It's probably not a good idea to blockingly request a checkpoint and a barrier
inside a PG_TRY/PG_ENSURE_ERROR_CLEANUP() though, so this would need a bit
more rearchitecting.


I think I'm just going to make the test more lenient by not insisting that the
error is the first thing on psql's stderr.


3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07

# +++ tap check in src/test/modules/test_aio +++

#   Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
#   at t/001_aio.pl line 318.
#                   'psql:<stdin>:4: ERROR:  starting batch while batch already in progress'
#     doesn't match '(?^:open AIO batch at end)'


The problem is basically that the test intentionally forgets to exit batchmode
- normally that would trigger an error at the end of the transaction, which
the test verifies.  However, with RELCACHE_FORCE_RELEASE and
CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
erroring out because batchmode isn't allowed to be entered recursively.


#0  pgaio_enter_batchmode () at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/aio.c:997
#1  0x000055ec847959bf in read_stream_look_ahead (stream=0x55ecbcfda098)
    at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:438
#2  0x000055ec84796514 in read_stream_next_buffer (stream=0x55ecbcfda098, per_buffer_data=0x0)
    at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:890
#3  0x000055ec8432520b in heap_fetch_next_buffer (scan=0x55ecbcfd1c00, dir=ForwardScanDirection)
    at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:679
#4  0x000055ec843259a4 in heapgettup_pagemode (scan=0x55ecbcfd1c00, dir=ForwardScanDirection, nkeys=1, key=0x55ecbcfd1620)
    at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1041
#5  0x000055ec843263ba in heap_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
    at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1420
#6  0x000055ec8434ebe5 in table_scan_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
    at ../../../../../home/andres/src/postgresql/src/include/access/tableam.h:1041
#7  0x000055ec8434f786 in systable_getnext (sysscan=0x55ecbcfd8088) at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:541
#8  0x000055ec849c784a in SearchCatCacheMiss (cache=0x55ecbcf81000, nkeys=1, hashValue=3830081846, hashIndex=2, v1=403, v2=0, v3=0, v4=0)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1543
#9  0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcf81000, nkeys=1, v1=403, v2=0, v3=0, v4=0)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
#10 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcf81000, v1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
#11 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=2, key1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
#12 0x000055ec849d8c78 in RelationInitIndexAccessInfo (relation=0x7f6a85901c20)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1456
#13 0x000055ec849d8471 in RelationBuildDesc (targetRelId=2703, insertIt=true)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1201
#14 0x000055ec849d9e9c in RelationIdGetRelation (relationId=2703) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2100
#15 0x000055ec842d219f in relation_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/common/relation.c:58
#16 0x000055ec8435043c in index_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:137
#17 0x000055ec8434f2f9 in systable_beginscan (heapRelation=0x7f6a859353a8, indexId=2703, indexOK=true, snapshot=0x0, nkeys=1, key=0x7ffc11aa7c90)
    at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:400
#18 0x000055ec849c782c in SearchCatCacheMiss (cache=0x55ecbcfa0e80, nkeys=1, hashValue=2659955452, hashIndex=60, v1=2278, v2=0, v3=0, v4=0)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1533
#19 0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcfa0e80, nkeys=1, v1=2278, v2=0, v3=0, v4=0)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
#20 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcfa0e80, v1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
#21 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=82, key1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
#22 0x000055ec849d0375 in getTypeOutputInfo (type=2278, typOutput=0x55ecbcfd15d0, typIsVarlena=0x55ecbcfd15d8)
    at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/lsyscache.c:2995
#23 0x000055ec842d1a57 in printtup_prepare_info (myState=0x55ecbcfcec00, typeinfo=0x55ecbcfd0588, numAttrs=1)
    at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:277
#24 0x000055ec842d1ba6 in printtup (slot=0x55ecbcfd0b28, self=0x55ecbcfcec00)
    at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:315
#25 0x000055ec84541f54 in ExecutePlan (queryDesc=0x55ecbced4290, operation=CMD_SELECT, sendTuples=true, numberTuples=0, direction=ForwardScanDirection,
    dest=0x55ecbcfcec00) at ../../../../../home/andres/src/postgresql/src/backend/executor/execMain.c:1814


I don't really have a good idea how to deal with that yet.


Greetings,

Andres

  168. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-01T22:25:28Z

    Hi,

On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
> 
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
> 
> # +++ tap check in src/test/modules/test_aio +++
> 
> #   Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
> #   at t/001_aio.pl line 318.
> #                   'psql:<stdin>:4: ERROR:  starting batch while batch already in progress'
> #     doesn't match '(?^:open AIO batch at end)'
> 
> 
> The problem is basically that the test intentionally forgets to exit batchmode
> - normally that would trigger an error at the end of the transaction, which
> the test verifies.  However, with RELCACHE_FORCE_RELEASE and
> CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
> erroring out because batchmode isn't allowed to be entered recursively.
> 
> 
> #0  pgaio_enter_batchmode () at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/aio.c:997
> #1  0x000055ec847959bf in read_stream_look_ahead (stream=0x55ecbcfda098)
>     at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:438
> #2  0x000055ec84796514 in read_stream_next_buffer (stream=0x55ecbcfda098, per_buffer_data=0x0)
>     at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:890
> #3  0x000055ec8432520b in heap_fetch_next_buffer (scan=0x55ecbcfd1c00, dir=ForwardScanDirection)
>     at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:679
> #4  0x000055ec843259a4 in heapgettup_pagemode (scan=0x55ecbcfd1c00, dir=ForwardScanDirection, nkeys=1, key=0x55ecbcfd1620)
>     at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1041
> #5  0x000055ec843263ba in heap_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
>     at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1420
> #6  0x000055ec8434ebe5 in table_scan_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
>     at ../../../../../home/andres/src/postgresql/src/include/access/tableam.h:1041
> #7  0x000055ec8434f786 in systable_getnext (sysscan=0x55ecbcfd8088) at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:541
> #8  0x000055ec849c784a in SearchCatCacheMiss (cache=0x55ecbcf81000, nkeys=1, hashValue=3830081846, hashIndex=2, v1=403, v2=0, v3=0, v4=0)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1543
> #9  0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcf81000, nkeys=1, v1=403, v2=0, v3=0, v4=0)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
> #10 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcf81000, v1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
> #11 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=2, key1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
> #12 0x000055ec849d8c78 in RelationInitIndexAccessInfo (relation=0x7f6a85901c20)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1456
> #13 0x000055ec849d8471 in RelationBuildDesc (targetRelId=2703, insertIt=true)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1201
> #14 0x000055ec849d9e9c in RelationIdGetRelation (relationId=2703) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2100
> #15 0x000055ec842d219f in relation_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/common/relation.c:58
> #16 0x000055ec8435043c in index_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:137
> #17 0x000055ec8434f2f9 in systable_beginscan (heapRelation=0x7f6a859353a8, indexId=2703, indexOK=true, snapshot=0x0, nkeys=1, key=0x7ffc11aa7c90)
>     at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:400
> #18 0x000055ec849c782c in SearchCatCacheMiss (cache=0x55ecbcfa0e80, nkeys=1, hashValue=2659955452, hashIndex=60, v1=2278, v2=0, v3=0, v4=0)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1533
> #19 0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcfa0e80, nkeys=1, v1=2278, v2=0, v3=0, v4=0)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
> #20 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcfa0e80, v1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
> #21 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=82, key1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
> #22 0x000055ec849d0375 in getTypeOutputInfo (type=2278, typOutput=0x55ecbcfd15d0, typIsVarlena=0x55ecbcfd15d8)
>     at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/lsyscache.c:2995
> #23 0x000055ec842d1a57 in printtup_prepare_info (myState=0x55ecbcfcec00, typeinfo=0x55ecbcfd0588, numAttrs=1)
>     at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:277
> #24 0x000055ec842d1ba6 in printtup (slot=0x55ecbcfd0b28, self=0x55ecbcfcec00)
>     at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:315
> #25 0x000055ec84541f54 in ExecutePlan (queryDesc=0x55ecbced4290, operation=CMD_SELECT, sendTuples=true, numberTuples=0, direction=ForwardScanDirection,
>     dest=0x55ecbcfcec00) at ../../../../../home/andres/src/postgresql/src/backend/executor/execMain.c:1814
> 
> 
> I don't really have a good idea how to deal with that yet.

Hm. Making the query something like

SELECT * FROM (VALUES (NULL), (batch_start()));

avoids the wrong output, because the type lookup happens for the first row
already. But that's pretty magical and probably fragile.

Greetings,

Andres Freund

  169. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-04-02T00:13:24Z

    On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote:
> On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
> > 
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
> > 
> > # +++ tap check in src/test/modules/test_aio +++
> > 
> > #   Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
> > #   at t/001_aio.pl line 318.
> > #                   'psql:<stdin>:4: ERROR:  starting batch while batch already in progress'
> > #     doesn't match '(?^:open AIO batch at end)'
> > 
> > 
> > The problem is basically that the test intentionally forgets to exit batchmode
> > - normally that would trigger an error at the end of the transaction, which
> > the test verifies.  However, with RELCACHE_FORCE_RELEASE and
> > CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
> > erroring out because batchmode isn't allowed to be entered recursively.

> > I don't really have a good idea how to deal with that yet.
> 
> Hm. Making the query something like
> 
> SELECT * FROM (VALUES (NULL), (batch_start()));
> 
> avoids the wrong output, because the type lookup happens for the first row
> already. But that's pretty magical and probably fragile.

Hmm.  Some options:

a. VALUES() trick above.  For test code, it's hard to argue with something
   that seems to solve it in practice.

b. Encapsulate the test in a PROCEDURE, so perhaps less happens between the
   batch_start() and the procedure-managed COMMIT.  Maybe less fragile than
   (a), maybe more fragile.

c. Move RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE to be
   GUC-controlled, like how CLOBBER_CACHE_ALWAYS changed into the
   debug_discard_caches GUC.  Then disable them for relevant parts of
   test_aio.  This feels best long-term, but it's bigger.  I also wanted this
   in syscache-update-pruned.spec[1].

d. Have test_aio deduce whether these are set, probably by observing memory
   contexts or DEBUG messages.  Maybe have every postmaster startup print a
   DEBUG message about these settings being enabled.  Skip relevant parts of
   test_aio.  This sounds messy.

Each of those feels defensible to me.  I'd probably do (a) or (b) to start.


[1] For that spec, an alternative expected output sufficed.  Incidentally,
I'll soon fix that spec flaking on valgrind/skink.

  170. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-02T11:58:21Z

    Hi,

I've pushed fixes for 1) and 2) and am working on 3).


On 2025-04-01 17:13:24 -0700, Noah Misch wrote:
> On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote:
> > On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> > > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
> > > 
> > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
> > > 
> > > # +++ tap check in src/test/modules/test_aio +++
> > > 
> > > #   Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
> > > #   at t/001_aio.pl line 318.
> > > #                   'psql:<stdin>:4: ERROR:  starting batch while batch already in progress'
> > > #     doesn't match '(?^:open AIO batch at end)'
> > > 
> > > 
> > > The problem is basically that the test intentionally forgets to exit batchmode
> > > - normally that would trigger an error at the end of the transaction, which
> > > the test verifies.  However, with RELCACHE_FORCE_RELEASE and
> > > CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
> > > erroring out because batchmode isn't allowed to be entered recursively.
> 
> > > I don't really have a good idea how to deal with that yet.
> > 
> > Hm. Making the query something like
> > 
> > SELECT * FROM (VALUES (NULL), (batch_start()));
> > 
> > avoids the wrong output, because the type lookup happens for the first row
> > already. But that's pretty magical and probably fragile.
> 
> Hmm.  Some options:
> 
> a. VALUES() trick above.  For test code, it's hard to argue with something
>    that seems to solve it in practice.

I think I'll go for a slightly nicer version of that, namely
  SELECT WHERE batch_start() IS NULL
I think that ends up the least verbose of the ideas we've been discussing.


> c. Move RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE to be
>    GUC-controlled, like how CLOBBER_CACHE_ALWAYS changed into the
>    debug_discard_caches GUC.  Then disable them for relevant parts of
>    test_aio.  This feels best long-term, but it's bigger.  I also wanted this
>    in syscache-update-pruned.spec[1].

Yea, that'd probably be a good thing medium-term.

Greetings,

Andres Freund

  171. Re: AIO v2.5

    Ranier Vilela <ranier.vf@gmail.com> — 2025-04-03T16:46:39Z

    Hi.

Em qua., 2 de abr. de 2025 às 08:58, Andres Freund <andres@anarazel.de>
escreveu:

> Hi,
>
> I've pushed fixes for 1) and 2) and am working on 3).
>
Coverity has one report about this.

CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
13. uninit_use_in_call: Using uninitialized value result_one. Field
result_one.result is uninitialized when calling pgaio_result_report.

Below not is a fix, but some suggestion:

diff --git a/src/backend/storage/buffer/bufmgr.c
b/src/backend/storage/buffer/bufmgr.c
index 1c37d7dfe2..b0f9ce452c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6786,6 +6786,8 @@ buffer_readv_encode_error(PgAioResult *result,
  else
  result->status = PGAIO_RS_WARNING;

+ result->result = 0;
+
  /*
  * The encoding is complicated enough to warrant cross-checking it against
  * the decode function.
@@ -6868,8 +6870,6 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
buf_off, Buffer buffer,
  /* Check for garbage data. */
  if (!failed)
  {
- PgAioResult result_one;
-
  if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
  failed_checksum))
  {
@@ -6904,6 +6904,8 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
buf_off, Buffer buffer,
  */
  if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
  {
+ PgAioResult result_one;
+
  buffer_readv_encode_error(&result_one, is_temp,
   *zeroed_buffer,
   *ignored_checksum,


1. I couldn't find the correct value to initialize the *result* field.
2. result_one can be reduced scope.

best regards,
Ranier Vilela

  172. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-03T18:19:43Z

    Hi,

On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> There are three different types of failures in the test_aio test so far:

And a fourth, visible after I enabled liburing support for skink.

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=skink&dt=2025-04-03%2007%3A06%3A19&stg=pg_upgrade-check
(ignore the pg_upgrade and oauth failures, they're independent, I've raised
them separately)

4a)

2025-04-03 10:58:32.978 UTC [2486740][client backend][3/6:0] LOG:  short read injection point called, is enabled: 0
==2486740== VALGRINDERROR-BEGIN
==2486740== Invalid read of size 2
==2486740==    at 0x59C8AC: PageIsNew (bufpage.h:237)
==2486740==    by 0x59C8AC: PageIsVerified (bufpage.c:108)
==2486740==    by 0x567870: buffer_readv_complete_one (bufmgr.c:6873)
==2486740==    by 0x567870: buffer_readv_complete (bufmgr.c:6996)
==2486740==    by 0x567870: shared_buffer_readv_complete (bufmgr.c:7153)
==2486740==    by 0x55DDB2: pgaio_io_call_complete_shared (aio_callback.c:256)
==2486740==    by 0x55D6F1: pgaio_io_process_completion (aio.c:512)
==2486740==    by 0x55F53A: pgaio_uring_drain_locked (method_io_uring.c:370)
==2486740==    by 0x55F7B8: pgaio_uring_wait_one (method_io_uring.c:449)
==2486740==    by 0x55C702: pgaio_io_wait (aio.c:587)
==2486740==    by 0x55C8B0: pgaio_wref_wait (aio.c:900)
==2486740==    by 0x8639240: read_rel_block_ll (test_aio.c:440)
==2486740==    by 0x3B915C: ExecInterpExpr (execExprInterp.c:953)
==2486740==    by 0x3B4E4E: ExecInterpExprStillValid (execExprInterp.c:2299)
==2486740==    by 0x3F7E97: ExecEvalExprNoReturn (executor.h:445)
==2486740==  Address 0x8fa400e is in a rw- anonymous segment
==2486740==
==2486740== VALGRINDERROR-END

The reason for this is that this test unpins the buffer (from the backend's
view), before waiting for the IO. While the AIO subsystem holds a pin,
UnpinBufferNoOwner() marked the buffer as inaccessible:
		/*
		 * Mark buffer non-accessible to Valgrind.
		 *
		 * Note that the buffer may have already been marked non-accessible
		 * within access method code that enforces that buffers are only
		 * accessed while a buffer lock is held.
		 */
		VALGRIND_MAKE_MEM_NOACCESS(BufHdrGetBlock(buf), BLCKSZ);


I think to fix this we need to mark buffers as accessible around the
PageIsVerified() call in buffer_readv_complete_one(), IFF they're not pinned
by the backend.  Unfortunately, this is complicated by the fact that local
buffers do not have valgrind integration :(, so we should only do that for
local buffers, as otherwise the local buffer stays inaccessible the next time
it is pinned.


4b)

That's not all though, after getting past this failure, I see uninitialized
memory errors for reads into temporary buffers:

==3334031== VALGRINDERROR-BEGIN
==3334031== Conditional jump or move depends on uninitialised value(s)
==3334031==    at 0xD7C859: PageIsVerified (bufpage.c:108)
==3334031==    by 0xD381CA: buffer_readv_complete_one (bufmgr.c:6876)
==3334031==    by 0xD385D1: buffer_readv_complete (bufmgr.c:7002)
==3334031==    by 0xD38D2E: local_buffer_readv_complete (bufmgr.c:7210)
==3334031==    by 0xD265FA: pgaio_io_call_complete_local (aio_callback.c:306)
==3334031==    by 0xD24720: pgaio_io_reclaim (aio.c:644)
==3334031==    by 0xD24400: pgaio_io_process_completion (aio.c:521)
==3334031==    by 0xD28D3D: pgaio_uring_drain_locked (method_io_uring.c:382)
==3334031==    by 0xD2905F: pgaio_uring_wait_one (method_io_uring.c:461)
==3334031==    by 0xD245E0: pgaio_io_wait (aio.c:587)
==3334031==    by 0xD24FFE: pgaio_wref_wait (aio.c:900)
==3334031==    by 0xD2F471: WaitReadBuffers (bufmgr.c:1695)
==3334031==    by 0xD2BCF4: read_stream_next_buffer (read_stream.c:898)
==3334031==    by 0x8B4861: heap_fetch_next_buffer (heapam.c:654)
==3334031==    by 0x8B4FFA: heapgettup_pagemode (heapam.c:1016)
==3334031==    by 0x8B594F: heap_getnextslot (heapam.c:1375)
==3334031==    by 0xB28AA4: table_scan_getnextslot (tableam.h:1031)
==3334031==    by 0xB29177: SeqNext (nodeSeqscan.c:81)
==3334031==    by 0xB28F75: ExecScanFetch (execScan.h:126)
==3334031==    by 0xB28FDD: ExecScanExtended (execScan.h:170)


The reason for this one is, I think, that valgrind doesn't understand io_uring
sufficiently. Which isn't surprising, io_uring's nature of an in-memory queue
of commands is somewhat hard to intercept by tools like valgrind and rr.

The best fix for that one would, I think, be to have method_io_uring() iterate
over the IOV and mark the relevant regions as defined?  That does fix the
issue at least and does seem to make sense?   Not quite sure if we should mark
the entire IOV is efined or just the portion that was actually read - the
latter is additional fiddly code, and it's not clear it's likely to be helpful?


4c)

Unfortunately, once 4a) is addressed, the VALGRIND_MAKE_MEM_NOACCESS() after
PageIsVerified() causes the *next* read into the same buffer in an IO worker
to fail:

==3339904== Syscall param pread64(buf) points to unaddressable byte(s)
==3339904==    at 0x5B3B687: __internal_syscall_cancel (cancellation.c:64)
==3339904==    by 0x5B3B6AC: __syscall_cancel (cancellation.c:75)
==3339904==    by 0x5B93C83: pread (pread64.c:25)
==3339904==    by 0xD274F4: pg_preadv (pg_iovec.h:56)
==3339904==    by 0xD2799A: pgaio_io_perform_synchronously (aio_io.c:137)
==3339904==    by 0xD2A6D7: IoWorkerMain (method_worker.c:538)
==3339904==    by 0xC91E26: postmaster_child_launch (launch_backend.c:290)
==3339904==    by 0xC99594: StartChildProcess (postmaster.c:3972)
==3339904==    by 0xC99EE3: maybe_adjust_io_workers (postmaster.c:4403)
==3339904==    by 0xC958A8: PostmasterMain (postmaster.c:1381)
==3339904==    by 0xB69622: main (main.c:227)
==3339904==  Address 0x7f936d386000 is in a rw- anonymous segment

Because, from the view of the IO worker, that memory is still marked NOACCESS,
even if it since has been marked accessible in the backend.


We could adress this by conditioning the VALGRIND_MAKE_MEM_NOACCESS() on not
being in an IO worker, but it seems better to instead explicitly mark the
region accessible in the worker, before executing the IO.

In a first hack, I did that in pgaio_io_perform_synchronously(), but that is
likely too broad.  I don't think the same scenario exists when IOs are
executed synchronously in the foreground.


Questions:

1) It'd be cleaner to implement valgrind support in localbuf.c, so we don't
   need to have special-case logic for that. But it also makes the change less
   localized and more "impactful", who knows what kind of skullduggery we have
   been getting away with unnoticed.

   I haven't written the code up yet, but I don't think it'd be all that much
   code to add valgrind support to localbuf.

2) Any better ideas to handle the above issues than what I outlined?

Greetings,

Andres Freund

  173. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-03T18:35:04Z

    Hi,

On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> Em qua., 2 de abr. de 2025 às 08:58, Andres Freund <andres@anarazel.de>
> escreveu:
> 
> > Hi,
> >
> > I've pushed fixes for 1) and 2) and am working on 3).
> >
> Coverity has one report about this.
> 
> CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
> 13. uninit_use_in_call: Using uninitialized value result_one. Field
> result_one.result is uninitialized when calling pgaio_result_report.

Isn't this a rather silly thing to warn about for coverity?  The field isn't
used in pgaio_result_report().  It can't be a particularly rare thing to have
struct fields that aren't always used?


> Below not is a fix, but some suggestion:
> 
> diff --git a/src/backend/storage/buffer/bufmgr.c
> b/src/backend/storage/buffer/bufmgr.c
> index 1c37d7dfe2..b0f9ce452c 100644
> --- a/src/backend/storage/buffer/bufmgr.c
> +++ b/src/backend/storage/buffer/bufmgr.c
> @@ -6786,6 +6786,8 @@ buffer_readv_encode_error(PgAioResult *result,
>   else
>   result->status = PGAIO_RS_WARNING;
> 
> + result->result = 0;
> +

That'd be completely wrong - and the tests indeed fail if you do that. The
read might succeed with a warning (e.g. due to zero_damaged_pages) in which
case the result still carries important information about how many blocks were
successfully read.


>   /*
>   * The encoding is complicated enough to warrant cross-checking it against
>   * the decode function.
> @@ -6868,8 +6870,6 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
> buf_off, Buffer buffer,
>   /* Check for garbage data. */
>   if (!failed)
>   {
> - PgAioResult result_one;
> -
>   if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
>   failed_checksum))
>   {
> @@ -6904,6 +6904,8 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
> buf_off, Buffer buffer,
>   */
>   if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
>   {
> + PgAioResult result_one;
> +
>   buffer_readv_encode_error(&result_one, is_temp,
>    *zeroed_buffer,
>    *ignored_checksum,
> 
> 
> 1. I couldn't find the correct value to initialize the *result* field.

It is not accessed in this path.  I guess we can just zero-initialize
result_one to shut up coverity.


> 2. result_one can be reduced scope.

True.


Greetings,

Andres Freund

  174. Re: AIO v2.5

    Ranier Vilela <ranier.vf@gmail.com> — 2025-04-03T19:16:50Z

    Em qui., 3 de abr. de 2025 às 15:35, Andres Freund <andres@anarazel.de>
escreveu:

> Hi,
>
> On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> > Em qua., 2 de abr. de 2025 às 08:58, Andres Freund <andres@anarazel.de>
> > escreveu:
> >
> > > Hi,
> > >
> > > I've pushed fixes for 1) and 2) and am working on 3).
> > >
> > Coverity has one report about this.
> >
> > CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
> > 13. uninit_use_in_call: Using uninitialized value result_one. Field
> > result_one.result is uninitialized when calling pgaio_result_report.
>
> Isn't this a rather silly thing to warn about for coverity?

Personally, I consider every warning to be important.


>   The field isn't
> used in pgaio_result_report().  It can't be a particularly rare thing to
> have
> struct fields that aren't always used?
>
Always considered a risk, someone may start using it.


>
>
> > Below not is a fix, but some suggestion:
> >
> > diff --git a/src/backend/storage/buffer/bufmgr.c
> > b/src/backend/storage/buffer/bufmgr.c
> > index 1c37d7dfe2..b0f9ce452c 100644
> > --- a/src/backend/storage/buffer/bufmgr.c
> > +++ b/src/backend/storage/buffer/bufmgr.c
> > @@ -6786,6 +6786,8 @@ buffer_readv_encode_error(PgAioResult *result,
> >   else
> >   result->status = PGAIO_RS_WARNING;
> >
> > + result->result = 0;
> > +
>
> That'd be completely wrong - and the tests indeed fail if you do that. The
> read might succeed with a warning (e.g. due to zero_damaged_pages) in which
> case the result still carries important information about how many blocks
> were
> successfully read.
>
That's exactly why it's not a patch.


>
>
> >   /*
> >   * The encoding is complicated enough to warrant cross-checking it
> against
> >   * the decode function.
> > @@ -6868,8 +6870,6 @@ buffer_readv_complete_one(PgAioTargetData *td,
> uint8
> > buf_off, Buffer buffer,
> >   /* Check for garbage data. */
> >   if (!failed)
> >   {
> > - PgAioResult result_one;
> > -
> >   if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
> >   failed_checksum))
> >   {
> > @@ -6904,6 +6904,8 @@ buffer_readv_complete_one(PgAioTargetData *td,
> uint8
> > buf_off, Buffer buffer,
> >   */
> >   if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
> >   {
> > + PgAioResult result_one;
> > +
> >   buffer_readv_encode_error(&result_one, is_temp,
> >    *zeroed_buffer,
> >    *ignored_checksum,
> >
> >
> > 1. I couldn't find the correct value to initialize the *result* field.
>
> It is not accessed in this path.  I guess we can just zero-initialize
> result_one to shut up coverity.
>
Very good.


>
>
> > 2. result_one can be reduced scope.
>
> True.
>
Ok.

best regards,
Ranier Vilela

  175. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-04-03T19:40:23Z

    On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> 4b)
> 
> That's not all though, after getting past this failure, I see uninitialized
> memory errors for reads into temporary buffers:
> 
> ==3334031== VALGRINDERROR-BEGIN
> ==3334031== Conditional jump or move depends on uninitialised value(s)
> ==3334031==    at 0xD7C859: PageIsVerified (bufpage.c:108)
> ==3334031==    by 0xD381CA: buffer_readv_complete_one (bufmgr.c:6876)
> ==3334031==    by 0xD385D1: buffer_readv_complete (bufmgr.c:7002)
> ==3334031==    by 0xD38D2E: local_buffer_readv_complete (bufmgr.c:7210)
> ==3334031==    by 0xD265FA: pgaio_io_call_complete_local (aio_callback.c:306)
> ==3334031==    by 0xD24720: pgaio_io_reclaim (aio.c:644)
> ==3334031==    by 0xD24400: pgaio_io_process_completion (aio.c:521)
> ==3334031==    by 0xD28D3D: pgaio_uring_drain_locked (method_io_uring.c:382)
> ==3334031==    by 0xD2905F: pgaio_uring_wait_one (method_io_uring.c:461)
> ==3334031==    by 0xD245E0: pgaio_io_wait (aio.c:587)
> ==3334031==    by 0xD24FFE: pgaio_wref_wait (aio.c:900)
> ==3334031==    by 0xD2F471: WaitReadBuffers (bufmgr.c:1695)
> ==3334031==    by 0xD2BCF4: read_stream_next_buffer (read_stream.c:898)
> ==3334031==    by 0x8B4861: heap_fetch_next_buffer (heapam.c:654)
> ==3334031==    by 0x8B4FFA: heapgettup_pagemode (heapam.c:1016)
> ==3334031==    by 0x8B594F: heap_getnextslot (heapam.c:1375)
> ==3334031==    by 0xB28AA4: table_scan_getnextslot (tableam.h:1031)
> ==3334031==    by 0xB29177: SeqNext (nodeSeqscan.c:81)
> ==3334031==    by 0xB28F75: ExecScanFetch (execScan.h:126)
> ==3334031==    by 0xB28FDD: ExecScanExtended (execScan.h:170)
> 
> 
> The reason for this one is, I think, that valgrind doesn't understand io_uring
> sufficiently. Which isn't surprising, io_uring's nature of an in-memory queue
> of commands is somewhat hard to intercept by tools like valgrind and rr.
> 
> The best fix for that one would, I think, be to have method_io_uring() iterate
> over the IOV and mark the relevant regions as defined?  That does fix the
> issue at least and does seem to make sense?

Makes sense.  Valgrind knows that read() makes its target bytes "defined".  It
probably doesn't have an io_uring equivalent for that.

I expect we only need this for local buffers, and it's unclear to me how the
fix for (4a) didn't fix this.  Before bufmgr Valgrind integration (1e0dfd1 of
2020-07) there was no explicit handling of shared_buffers.  I suspect that
worked because the initial mmap() of shared memory was considered "defined"
(zeros), and steps like PageAddItem() copy only defined bytes into buffers.
Hence, shared_buffers remained defined without explicit Valgrind client
requests.  This example uses local buffers.  Storage for those comes from
MemoryContextAlloc() in GetLocalBufferStorage().  That memory starts
undefined, but it becomes defined at PageInit() or read().  Hence, I expected
the fix for (4a) to make the buffer defined after io_uring read.  What makes
the outcome different?

In the general case, we could want client requests as follows:

- If completor==definer and has not dropped pin:
  - Make defined before verifying page.  That's all.  It might be cleaner to
    do this when first retrieving a return value from io_uring, since this
    just makes up for what Valgrind already does for readv().

- If completor!=definer or has dropped pin:
  - Make NOACCESS in definer when definer cedes its own pin.
  - For io_method=worker, make UNDEFINED before starting readv().  It might be
    cleanest to do this when the worker first acts as the owner of the AIO
    subsystem pin, if that's a clear moment earlier than readv().
  - Make DEFINED in completor before verifying page.  It might be cleaner to
    do this when the completor first retrieves a return value from io_uring,
    since this just makes up for what Valgrind already does for readv().
  - Make NOACCESS in completor after verifying page.  Similarly, it might be
    cleaner to do this when the completor releases the AIO subsystem pin.

> Not quite sure if we should mark
> the entire IOV is efined or just the portion that was actually read - the
> latter is additional fiddly code, and it's not clear it's likely to be helpful?

Seems fine to do the simpler way if that saves fiddly code.

> 4c)
> 
> Unfortunately, once 4a) is addressed, the VALGRIND_MAKE_MEM_NOACCESS() after
> PageIsVerified() causes the *next* read into the same buffer in an IO worker
> to fail:
> 
> ==3339904== Syscall param pread64(buf) points to unaddressable byte(s)
> ==3339904==    at 0x5B3B687: __internal_syscall_cancel (cancellation.c:64)
> ==3339904==    by 0x5B3B6AC: __syscall_cancel (cancellation.c:75)
> ==3339904==    by 0x5B93C83: pread (pread64.c:25)
> ==3339904==    by 0xD274F4: pg_preadv (pg_iovec.h:56)
> ==3339904==    by 0xD2799A: pgaio_io_perform_synchronously (aio_io.c:137)
> ==3339904==    by 0xD2A6D7: IoWorkerMain (method_worker.c:538)
> ==3339904==    by 0xC91E26: postmaster_child_launch (launch_backend.c:290)
> ==3339904==    by 0xC99594: StartChildProcess (postmaster.c:3972)
> ==3339904==    by 0xC99EE3: maybe_adjust_io_workers (postmaster.c:4403)
> ==3339904==    by 0xC958A8: PostmasterMain (postmaster.c:1381)
> ==3339904==    by 0xB69622: main (main.c:227)
> ==3339904==  Address 0x7f936d386000 is in a rw- anonymous segment
> 
> Because, from the view of the IO worker, that memory is still marked NOACCESS,
> even if it since has been marked accessible in the backend.
> 
> 
> We could adress this by conditioning the VALGRIND_MAKE_MEM_NOACCESS() on not
> being in an IO worker, but it seems better to instead explicitly mark the
> region accessible in the worker, before executing the IO.

Sounds good.  Since the definer gave the AIO subsystem a pin on the worker's
behalf, it's like the worker is doing an implicit pin and explicit unpin.

> In a first hack, I did that in pgaio_io_perform_synchronously(), but that is
> likely too broad.  I don't think the same scenario exists when IOs are
> executed synchronously in the foreground.
> 
> 
> Questions:
> 
> 1) It'd be cleaner to implement valgrind support in localbuf.c, so we don't
>    need to have special-case logic for that. But it also makes the change less
>    localized and more "impactful", who knows what kind of skullduggery we have
>    been getting away with unnoticed.
> 
>    I haven't written the code up yet, but I don't think it'd be all that much
>    code to add valgrind support to localbuf.

It would be the right thing long-term, and it's not a big deal if it causes
some false positives initially.  So if you're leaning that way, that's good.

> 2) Any better ideas to handle the above issues than what I outlined?

Not here, unless the discussion under (4b) differs usefully from what you
planned.

  176. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-03T22:30:19Z

    Hi,

On 2025-04-03 16:16:50 -0300, Ranier Vilela wrote:
> Em qui., 3 de abr. de 2025 às 15:35, Andres Freund <andres@anarazel.de>
> escreveu:> > On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> > > Em qua., 2 de abr. de 2025 às 08:58, Andres Freund <andres@anarazel.de>
> > > escreveu:
> > >
> > > > Hi,
> > > >
> > > > I've pushed fixes for 1) and 2) and am working on 3).
> > > >
> > > Coverity has one report about this.
> > >
> > > CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
> > > 13. uninit_use_in_call: Using uninitialized value result_one. Field
> > > result_one.result is uninitialized when calling pgaio_result_report.
> >
> > Isn't this a rather silly thing to warn about for coverity?
> 
> Personally, I consider every warning to be important.

If the warning is wrong, then it's not helpful. Warning quality really
matters.

Zero-initializing everything *REDUCES* what static analysis and sanitizers can
do. The analyzer/sanitizer can't tell that you just silenced a warning by
zero-initializing something that shouldn't be accessed. If later there is an
access, the zero is probably the wrong value, but the no tool can tell you,
because you did initialize it after all.

> 
> >   The field isn't
> > used in pgaio_result_report().  It can't be a particularly rare thing to
> > have
> > struct fields that aren't always used?
> >
> Always considered a risk, someone may start using it.

That makes it worse! E.g. valgrind won't raise errors about it anymore.

Greetings,

Andres Freund

  177. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-04T19:16:18Z

    Hi,

Sorry for the slow work on this. The cycle times are humonguous due to
valgrind being so slow...


On 2025-04-03 12:40:23 -0700, Noah Misch wrote:
> On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> > The best fix for that one would, I think, be to have method_io_uring() iterate
> > over the IOV and mark the relevant regions as defined?  That does fix the
> > issue at least and does seem to make sense?
> 
> Makes sense.  Valgrind knows that read() makes its target bytes "defined".  It
> probably doesn't have an io_uring equivalent for that.

Correct - and I think it would be nontrivial to add, because there's not easy
syscall to intercept...


> I expect we only need this for local buffers, and it's unclear to me how the
> fix for (4a) didn't fix this.

At that time I didn't apply the fix in 4a) to local buffers, because local
buffers, in HEAD, don't have the valgrind integration. Without that marking
the buffer as NOACCESS would cause all sorts of issues, because it'd be
considered inaccessible even after pinning.  As you analyzed, that then ends
up considered undefined due to the MemoryContextAlloc().


> In the general case, we could want client requests as follows:
> 
> - If completor==definer and has not dropped pin:
>   - Make defined before verifying page.  That's all.  It might be cleaner to
>     do this when first retrieving a return value from io_uring, since this
>     just makes up for what Valgrind already does for readv().

Yea, I think it's better to do that in io_uring. It's what I have done in the
attached.


> - If completor!=definer or has dropped pin:
>   - Make NOACCESS in definer when definer cedes its own pin.

That's the current behaviour for shared buffers, right?


>   - For io_method=worker, make UNDEFINED before starting readv().  It might be
>     cleanest to do this when the worker first acts as the owner of the AIO
>     subsystem pin, if that's a clear moment earlier than readv().

Hm, what do we need this for?


>   - Make DEFINED in completor before verifying page.  It might be cleaner to
>     do this when the completor first retrieves a return value from io_uring,
>     since this just makes up for what Valgrind already does for readv().

I think we can't rely on the marking during retrieving it from io_uring, as
that might have happened in a different backend for a temp buffer. That'd only
happen if we got io_uring events for *another* IO that involved a shared rel,
but it can happen.



> > Not quite sure if we should mark
> > the entire IOV is efined or just the portion that was actually read - the
> > latter is additional fiddly code, and it's not clear it's likely to be helpful?
> 
> Seems fine to do the simpler way if that saves fiddly code.

Can't quite decide, it's just at the border of what I consider too
fiddly... See the change to method_io_uring.c in the attached patch.


> > Questions:
> > 
> > 1) It'd be cleaner to implement valgrind support in localbuf.c, so we don't
> >    need to have special-case logic for that. But it also makes the change less
> >    localized and more "impactful", who knows what kind of skullduggery we have
> >    been getting away with unnoticed.
> > 
> >    I haven't written the code up yet, but I don't think it'd be all that much
> >    code to add valgrind support to localbuf.
> 
> It would be the right thing long-term, and it's not a big deal if it causes
> some false positives initially.  So if you're leaning that way, that's good.

It was easy enough.

I saw one related failure, FlushRelationBuffers() didn't pin temporary buffers
before flushing them. Pinning the buffers fixed that.

I don't think it's a real problem to not pin the local buffer during
FlushRelationBuffers(), at least not today. But it seems unnecessarily odd to
not pin it.


I wish valgrind had a way to mark the buffer as inaccessible and then
accessible again, without loosing the defined-ness information...


Greetings,

Andres Freund

  178. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-04-04T21:18:02Z

    On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> On 2025-04-03 12:40:23 -0700, Noah Misch wrote:
> > On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:

> > In the general case, we could want client requests as follows:
> > 
> > - If completor==definer and has not dropped pin:
> >   - Make defined before verifying page.  That's all.  It might be cleaner to
> >     do this when first retrieving a return value from io_uring, since this
> >     just makes up for what Valgrind already does for readv().
> 
> Yea, I think it's better to do that in io_uring. It's what I have done in the
> attached.
> 
> 
> > - If completor!=definer or has dropped pin:
> >   - Make NOACCESS in definer when definer cedes its own pin.
> 
> That's the current behaviour for shared buffers, right?

Yes.

> >   - For io_method=worker, make UNDEFINED before starting readv().  It might be
> >     cleanest to do this when the worker first acts as the owner of the AIO
> >     subsystem pin, if that's a clear moment earlier than readv().
> 
> Hm, what do we need this for?

At the time, we likely didn't need it:

- If the worker does its own PinBuffer*()+unpin, we don't need it.  Those
  functions do the Valgrind client requests.
- If the worker relies on the AIO-subsystem-owned pin and does neither regular
  pin nor regular unpin, we don't need it.  Buffers are always "defined".
- If the worker relies on the AIO-subsystem-owned pin to skip PinBuffer*() but
  uses regular unpin code, then the buffer may be NOACCESS.  Then one would
  need this.  But this would be questionable for other reasons.

Your proposed change to set NOACCESS in buffer_readv_complete_one() interacts
with things further, making the UNDEFINED necessary.

> >   - Make DEFINED in completor before verifying page.  It might be cleaner to
> >     do this when the completor first retrieves a return value from io_uring,
> >     since this just makes up for what Valgrind already does for readv().
> 
> I think we can't rely on the marking during retrieving it from io_uring, as
> that might have happened in a different backend for a temp buffer. That'd only
> happen if we got io_uring events for *another* IO that involved a shared rel,
> but it can happen.

Good point.  I think the VALGRIND_MAKE_MEM_DEFINED() in
pgaio_uring_drain_locked() isn't currently needed at all.  If
completor-subxact==definer-subxact, PinBuffer() already did what Valgrind
needs.  Otherwise, buffer_readv_complete_one() does what Valgrind needs.

If that's right, it would still be nice to reach the right
VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.  That helps future,
non-bufmgr AIO use cases.  It's tricky to pick the right place for that
VALGRIND_MAKE_MEM_DEFINED():

- pgaio_uring_drain_locked() is problematic, I think.  In the localbuf case,
  the iovec base address is relevant only in the ioh-defining process.  In the
  shmem completor!=definer case, this runs only in the completor.

- A complete_local callback solves those problems.  However, if the
  AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
  the buffer mapping may have changed by the time of complete_local.

- Putting it in the place that would call pgaio_result_report(ERROR) if
  needed, e.g. ProcessReadBuffersResult(), solves the problem of the buffer
  mapping having moved.  ProcessReadBuffersResult() doesn't even need this,
  since PinBuffer() already did it.  Each future AIO use case will have a
  counterpart of ProcessReadBuffersResult() that consumes the result and
  proceeds with tasks that depend on the AIO.  That's the place.

Is that right?  I got this wrong a few times while trying to think through it,
so I'm not too confident in the above.

> > > Not quite sure if we should mark
> > > the entire IOV is efined or just the portion that was actually read - the
> > > latter is additional fiddly code, and it's not clear it's likely to be helpful?
> > 
> > Seems fine to do the simpler way if that saves fiddly code.
> 
> Can't quite decide, it's just at the border of what I consider too
> fiddly... See the change to method_io_uring.c in the attached patch.

It is at the border, as you say, but I'd tend to keep it.


> Subject: [PATCH v1 1/3] localbuf: Add Valgrind buffer access instrumentation

Ready for commit


> Subject: [PATCH v1 2/3] aio: Make AIO compatible with valgrind

See above about pgaio_uring_drain_locked().

> related code until it is pinned bu "user" code again. But it requires some

s/bu/by/

> + * Return the iovecand its length. Currently only expected to be used by

s/iovecand/iovec and/

> @@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
>  		for (int i = 0; i < ncqes; i++)
>  		{
>  			struct io_uring_cqe *cqe = cqes[i];
> +			int32		res;
>  			PgAioHandle *ioh;
>  
>  			ioh = io_uring_cqe_get_data(cqe);
>  			errcallback.arg = ioh;
> +			res = cqe->res;
> +
>  			io_uring_cqe_seen(&context->io_uring_ring, cqe);
>  
> -			pgaio_io_process_completion(ioh, cqe->res);
> +			pgaio_uring_io_process_completion(ioh, res);

I guess this is a distinct cleanup, done to avoid any suspicion of cqe being
reused asynchronously after io_uring_cqe_seen().  Is that right?


> Subject: [PATCH v1 3/3] aio: Avoid spurious coverity warning

Ready for commit

  179. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-05T03:53:13Z

    Hi,

On 2025-04-04 14:18:02 -0700, Noah Misch wrote:
> On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> > >   - Make DEFINED in completor before verifying page.  It might be cleaner to
> > >     do this when the completor first retrieves a return value from io_uring,
> > >     since this just makes up for what Valgrind already does for readv().
> > 
> > I think we can't rely on the marking during retrieving it from io_uring, as
> > that might have happened in a different backend for a temp buffer. That'd only
> > happen if we got io_uring events for *another* IO that involved a shared rel,
> > but it can happen.
> 
> Good point.  I think the VALGRIND_MAKE_MEM_DEFINED() in
> pgaio_uring_drain_locked() isn't currently needed at all.  If
> completor-subxact==definer-subxact, PinBuffer() already did what Valgrind
> needs.  Otherwise, buffer_readv_complete_one() does what Valgrind needs.

We did need it - but only because I bungled something in the earlier patch to
add valgrind support.  The problem is that in PinLocalBuffer() there may not
actually be any storage allocated for the buffer yet, so
VALGRIND_MAKE_MEM_DEFINED() doesn't work. In the first use of the buffer the
allocation happens a bit later, in GetLocalVictimBuffer(), namely during the
call to GetLocalBufferStorage().

Not quite sure yet how to best deal with it.  Putting the PinLocalBuffer()
slightly later into GetLocalVictimBuffer() fixes the issue, but also doesn't
really seem great.


> If that's right, it would still be nice to reach the right
> VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.

I think that would be possible if we didn't do VALGRIND_MAKE_MEM_NOACCESS() in
UnpinBuffer()/UnpinLocalBuffer(). But with that I don't see how we can avoid
needing to remark the region as accessible?


> That helps future, non-bufmgr AIO use cases.  It's tricky to pick the right
> place for that VALGRIND_MAKE_MEM_DEFINED():

> - pgaio_uring_drain_locked() is problematic, I think.  In the localbuf case,
>   the iovec base address is relevant only in the ioh-defining process.  In the
>   shmem completor!=definer case, this runs only in the completor.

You're right :(


> - A complete_local callback solves those problems.  However, if the
>   AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
>   the buffer mapping may have changed by the time of complete_local.

I don't think that is possible, due to the aio subsystem owned pin?


> - Putting it in the place that would call pgaio_result_report(ERROR) if
>   needed, e.g. ProcessReadBuffersResult(), solves the problem of the buffer
>   mapping having moved.  ProcessReadBuffersResult() doesn't even need this,
>   since PinBuffer() already did it.  Each future AIO use case will have a
>   counterpart of ProcessReadBuffersResult() that consumes the result and
>   proceeds with tasks that depend on the AIO.  That's the place.

I don't really follow - at the point something like ProcessReadBuffersResult()
gets involved, we'll already have done the accesses that needed the memory to
be accessible and defined?


I think the point about non-aio uses is a fair one, but I don't quite know how
to best solve it right now, due to the local buffer issue you mentioned. I'd
guess that we'd best put it somewhere
a) in pgaio_io_process_completion(), if definer==completor || !PGAIO_HF_REFERENCES_LOCAL
b) pgaio_io_call_complete_local(), just before calling
   pgaio_io_call_complete_local() if PGAIO_HF_REFERENCES_LOCAL



> > related code until it is pinned bu "user" code again. But it requires some
> 
> s/bu/by/
> 
> > + * Return the iovecand its length. Currently only expected to be used by
> 
> s/iovecand/iovec and/

Fixed.


> > @@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
> >  		for (int i = 0; i < ncqes; i++)
> >  		{
> >  			struct io_uring_cqe *cqe = cqes[i];
> > +			int32		res;
> >  			PgAioHandle *ioh;
> >  
> >  			ioh = io_uring_cqe_get_data(cqe);
> >  			errcallback.arg = ioh;
> > +			res = cqe->res;
> > +
> >  			io_uring_cqe_seen(&context->io_uring_ring, cqe);
> >  
> > -			pgaio_io_process_completion(ioh, cqe->res);
> > +			pgaio_uring_io_process_completion(ioh, res);
> 
> I guess this is a distinct cleanup, done to avoid any suspicion of cqe being
> reused asynchronously after io_uring_cqe_seen().  Is that right?

I don't think there is any such danger - there's no background thing
processing things on the ring, if there were, it'd get corrupted, but it
seemed cleaner to do it that way when I introduced
pgaio_uring_io_process_completion().


> > Subject: [PATCH v1 3/3] aio: Avoid spurious coverity warning
> 
> Ready for commit

Thanks!


Greetings,

Andres Freund

  180. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-04-05T13:43:52Z

    On Fri, Apr 04, 2025 at 11:53:13PM -0400, Andres Freund wrote:
> On 2025-04-04 14:18:02 -0700, Noah Misch wrote:
> > On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> > > >   - Make DEFINED in completor before verifying page.  It might be cleaner to
> > > >     do this when the completor first retrieves a return value from io_uring,
> > > >     since this just makes up for what Valgrind already does for readv().
> > > 
> > > I think we can't rely on the marking during retrieving it from io_uring, as
> > > that might have happened in a different backend for a temp buffer. That'd only
> > > happen if we got io_uring events for *another* IO that involved a shared rel,
> > > but it can happen.
> > 
> > Good point.  I think the VALGRIND_MAKE_MEM_DEFINED() in
> > pgaio_uring_drain_locked() isn't currently needed at all.  If
> > completor-subxact==definer-subxact, PinBuffer() already did what Valgrind
> > needs.  Otherwise, buffer_readv_complete_one() does what Valgrind needs.
> 
> We did need it - but only because I bungled something in the earlier patch to
> add valgrind support.  The problem is that in PinLocalBuffer() there may not
> actually be any storage allocated for the buffer yet, so
> VALGRIND_MAKE_MEM_DEFINED() doesn't work. In the first use of the buffer the
> allocation happens a bit later, in GetLocalVictimBuffer(), namely during the
> call to GetLocalBufferStorage().
> 
> Not quite sure yet how to best deal with it.  Putting the PinLocalBuffer()
> slightly later into GetLocalVictimBuffer() fixes the issue, but also doesn't
> really seem great.

Yeah.  Maybe this (untested):

diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index fb44d75..3de13e7 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -813,7 +813,8 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
 		pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
 
 		/* see comment in PinBuffer() */
-		VALGRIND_MAKE_MEM_DEFINED(LocalBufHdrGetBlock(buf_hdr), BLCKSZ);
+		if (LocalBufHdrGetBlock(bufHdr) != NULL)
+			VALGRIND_MAKE_MEM_DEFINED(LocalBufHdrGetBlock(buf_hdr), BLCKSZ);
 	}
 	LocalRefCount[bufid]++;
 	ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -932,6 +933,15 @@ GetLocalBufferStorage(void)
 	next_buf_in_block++;
 	total_bufs_allocated++;
 
+	/*
+	 * Caller's PinBuffer() was too early for Valgrind updates, so do it here.
+	 * The block is actually undefined, but we want consistency with the
+	 * regular case of not needing to allocate memory.  This is specifically
+	 * needed when method_io_uring.c fills the block, because Valgrind doesn't
+	 * recognize io_uring reads causing undefined memory to become defined.
+	 */
+	VALGRIND_MAKE_MEM_DEFINED(this_buf, BLCKSZ);
+
 	return (Block) this_buf;
 }
 
> > If that's right, it would still be nice to reach the right
> > VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.
> 
> I think that would be possible if we didn't do VALGRIND_MAKE_MEM_NOACCESS() in
> UnpinBuffer()/UnpinLocalBuffer(). But with that I don't see how we can avoid
> needing to remark the region as accessible?

Yes, it's not that we should remove VALGRIND_MAKE_MEM_DEFINED() from bufmgr.
I was trying to think about future AIO callers (e.g. RelationCopyStorage())
and how they'd want things to work.  That said, perhaps we should just omit
the io_uring-level Valgrind calls and delegate the problem to higher layers
until there's a concrete use case:

--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -742,14 +742,16 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
  * Compared to smgrreadv(), more responsibilities fall on the caller:
  * - Partial reads need to be handled by the caller re-issuing IO for the
  *   unread blocks
  * - smgr will ereport(LOG_SERVER_ONLY) some problems, but higher layers are
  *   responsible for pgaio_result_report() to mirror that news to the user (if
  *   the IO results in PGAIO_RS_WARNING) or abort the (sub)transaction (if
  *   PGAIO_RS_ERROR).
+ * - Under Valgrind, the "buffers" memory may or may not change status to
+ *   DEFINED, depending on io_method and concurrent activity.
  */
 void
 smgrstartreadv(PgAioHandle *ioh,

> > - A complete_local callback solves those problems.  However, if the
> >   AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
> >   the buffer mapping may have changed by the time of complete_local.
> 
> I don't think that is possible, due to the aio subsystem owned pin?

We currently drop the shared buffer AIO subsystem pin in complete_shared
(buffer_readv_complete_one() -> TerminateBufferIO()).  I was trying to say
that if we put a VALGRIND_MAKE_MEM_DEFINED in complete_local without changing
anything else, it would have this problem.  We probably wouldn't want to move
the shared buffer pin drop to complete_local without a strong reason.

> > - Putting it in the place that would call pgaio_result_report(ERROR) if
> >   needed, e.g. ProcessReadBuffersResult(), solves the problem of the buffer
> >   mapping having moved.  ProcessReadBuffersResult() doesn't even need this,
> >   since PinBuffer() already did it.  Each future AIO use case will have a
> >   counterpart of ProcessReadBuffersResult() that consumes the result and
> >   proceeds with tasks that depend on the AIO.  That's the place.
> 
> I don't really follow - at the point something like ProcessReadBuffersResult()
> gets involved, we'll already have done the accesses that needed the memory to
> be accessible and defined?

(I was referring to future AIO callers, and this part of the discussion is
obsolete if we just delegate the problem.)  The bufmgr subsystem does the
PageIsVerified() accesses before ProcessReadBuffersResult(), but I think the
client of the bufmgr subsystem, e.g. read_stream.c, does accesses only after
that.  The client must do accesses only after that, since any
pgaio_result_report(ERROR) doesn't happen until ProcessReadBuffersResult().

> I think the point about non-aio uses is a fair one, but I don't quite know how
> to best solve it right now, due to the local buffer issue you mentioned. I'd
> guess that we'd best put it somewhere
> a) in pgaio_io_process_completion(), if definer==completor || !PGAIO_HF_REFERENCES_LOCAL
> b) pgaio_io_call_complete_local(), just before calling
>    pgaio_io_call_complete_local() if PGAIO_HF_REFERENCES_LOCAL

I think that would do nothing wrong today, but it uses
!PGAIO_HF_REFERENCES_LOCAL as a proxy for "target memory is already DEFINED,
or a higher layer will make it DEFINED".  While !PGAIO_HF_REFERENCES_LOCAL
does imply that today, I think that's a bufmgr-specific conclusion.  I have no
particular reason to expect that to hold for future AIO use cases.  As above,
I'd be inclined to omit the io_uring-level Valgrind calls until we have a
concrete use case to drive their design.  How do you see it?

> > > @@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
> > >  		for (int i = 0; i < ncqes; i++)
> > >  		{
> > >  			struct io_uring_cqe *cqe = cqes[i];
> > > +			int32		res;
> > >  			PgAioHandle *ioh;
> > >  
> > >  			ioh = io_uring_cqe_get_data(cqe);
> > >  			errcallback.arg = ioh;
> > > +			res = cqe->res;
> > > +
> > >  			io_uring_cqe_seen(&context->io_uring_ring, cqe);
> > >  
> > > -			pgaio_io_process_completion(ioh, cqe->res);
> > > +			pgaio_uring_io_process_completion(ioh, res);
> > 
> > I guess this is a distinct cleanup, done to avoid any suspicion of cqe being
> > reused asynchronously after io_uring_cqe_seen().  Is that right?
> 
> I don't think there is any such danger - there's no background thing
> processing things on the ring, if there were, it'd get corrupted, but it
> seemed cleaner to do it that way when I introduced
> pgaio_uring_io_process_completion().

I agree there's no concrete danger.

  181. Re: AIO v2.5

    Alexander Lakhin <exclusion@gmail.com> — 2025-04-06T20:00:00Z

    Hello Andres,

02.04.2025 14:58, Andres Freund wrote:
> Hi,
>
> I've pushed fixes for 1) and 2) and am working on 3).

When running multiple installcheck's against a single server (please find
the ready-to-use script attached (I use more sophisticated version with
additional patches to make installcheck pass cleanly, but that's not
required for this case)), I've encountered an interesting error related to
AIO/uring:
iteration 8: Sun Apr  6 19:22:39 UTC 2025
installchecks finished: Sun Apr  6 19:23:47 UTC 2025
2025-04-06 19:22:44.216 UTC [349525] LOG:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [349525] ERROR:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled

server.log contains:
2025-04-06 19:22:44.215 UTC [38231] LOG:  checkpoint complete: wrote ...
2025-04-06 19:22:44.216 UTC [38231] LOG:  checkpoint starting: immediate force wait flush-all
2025-04-06 19:22:44.216 UTC [349525] LOG:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [349525] STATEMENT:  alter table parted_copytest attach partition parted_copytest_a1 for 
values in(1);
2025-04-06 19:22:44.216 UTC [349525] ERROR:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [349525] STATEMENT:  alter table parted_copytest attach partition parted_copytest_a1 for 
values in(1);

It's reproduced better on tmpfs for me; probably you would need to increase
NUM_INSTALLCHECKS/NUM_ITERATIONS for your machine. I can reduce the testing
procedure to something trivial, if it makes sense for you. Probably, the
same effect can be also achieved with just pgbench...

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

  182. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-07T16:20:15Z

    Hi,

On 2025-04-06 23:00:00 +0300, Alexander Lakhin wrote:
> 02.04.2025 14:58, Andres Freund wrote:
> When running multiple installcheck's against a single server (please find
> the ready-to-use script attached (I use more sophisticated version with
> additional patches to make installcheck pass cleanly, but that's not
> required for this case)), I've encountered an interesting error related to
> AIO/uring:
> iteration 8: Sun Apr  6 19:22:39 UTC 2025
> installchecks finished: Sun Apr  6 19:23:47 UTC 2025
> 2025-04-06 19:22:44.216 UTC [349525] LOG:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> 2025-04-06 19:22:44.216 UTC [349525] ERROR:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled

Thanks for the report, clearly something isn't right.


> It's reproduced better on tmpfs for me; probably you would need to increase
> NUM_INSTALLCHECKS/NUM_ITERATIONS for your machine.

I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
tmpfs.


> server.log contains:
> 2025-04-06 19:22:44.215 UTC [38231] LOG:  checkpoint complete: wrote ...
> 2025-04-06 19:22:44.216 UTC [38231] LOG:  checkpoint starting: immediate force wait flush-all
> 2025-04-06 19:22:44.216 UTC [349525] LOG:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:  alter table parted_copytest
> attach partition parted_copytest_a1 for values in(1);
> 2025-04-06 19:22:44.216 UTC [349525] ERROR:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:  alter table parted_copytest
> attach partition parted_copytest_a1 for values in(1);

Hm. Does the failure vary between occurrences?
- is it always the same statement? Probably not?
- is it always 2606 (i.e. pg_constraint)?
- does the failure always happen around a checkpoint? If so, is it always
  immediate?
- I do assume it's always ECANCELED?


> I can reduce the testing procedure to something trivial, if it makes sense
> for you. Probably, the same effect can be also achieved with just pgbench...

That'd be very helpful!

Greetings,

Andres Freund

  183. Re: AIO v2.5

    Alexander Lakhin <exclusion@gmail.com> — 2025-04-07T19:10:01Z

    Hello Andres,

07.04.2025 19:20, Andres Freund wrote:
>> iteration 8: Sun Apr  6 19:22:39 UTC 2025
>> installchecks finished: Sun Apr  6 19:23:47 UTC 2025
>> 2025-04-06 19:22:44.216 UTC [349525] LOG:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
>> 2025-04-06 19:22:44.216 UTC [349525] ERROR:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> Thanks for the report, clearly something isn't right.

Thank you for your attention to it!
>> It's reproduced better on tmpfs for me; probably you would need to increase
>> NUM_INSTALLCHECKS/NUM_ITERATIONS for your machine.
> I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
> tmpfs.

Before sharing the script I tested it on two my machines, but I had
anticipated that the error can be hard to reproduce. Will try to reduce
the reproducer...

>> server.log contains:
>> 2025-04-06 19:22:44.215 UTC [38231] LOG:  checkpoint complete: wrote ...
>> 2025-04-06 19:22:44.216 UTC [38231] LOG:  checkpoint starting: immediate force wait flush-all
>> 2025-04-06 19:22:44.216 UTC [349525] LOG:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
>> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:  alter table parted_copytest
>> attach partition parted_copytest_a1 for values in(1);
>> 2025-04-06 19:22:44.216 UTC [349525] ERROR:  could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
>> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:  alter table parted_copytest
>> attach partition parted_copytest_a1 for values in(1);
> Hm. Does the failure vary between occurrences?
> - is it always the same statement? Probably not?
> - is it always 2606 (i.e. pg_constraint)?
> - does the failure always happen around a checkpoint? If so, is it always
>    immediate?
> - I do assume it's always ECANCELED?

I've got three such failures with three runs of of the script to
answer your questions:
...
iteration 6: Mon Apr  7 18:08:35 UTC 2025
installchecks finished: Mon Apr  7 18:09:45 UTC 2025
2025-04-07 18:08:45.369 UTC [1941260] LOG:  could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
2025-04-07 18:08:45.369 UTC [1941260] ERROR:  could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
server.log:
...
2025-04-07 18:08:44.120 UTC [1713945] LOG:  checkpoint starting: wal

2025-04-07 18:08:45.369 UTC [1941260] LOG:  could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
2025-04-07 18:08:45.369 UTC [1941260] STATEMENT:  SELECT routine_name, ordinal_position, parameter_name, parameter_default
         FROM information_schema.parameters JOIN information_schema.routines USING (specific_schema, specific_name)
         WHERE routine_schema = 'temp_func_test' AND routine_name ~ '^functest_is_'
         ORDER BY 1, 2;
2025-04-07 18:08:45.369 UTC [1941260] ERROR:  could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
2025-04-07 18:08:45.369 UTC [1941260] STATEMENT:  SELECT routine_name, ordinal_position, parameter_name, parameter_default
         FROM information_schema.parameters JOIN information_schema.routines USING (specific_schema, specific_name)
         WHERE routine_schema = 'temp_func_test' AND routine_name ~ '^functest_is_'
         ORDER BY 1, 2;
...
2025-04-07 18:08:51.836 UTC [1713945] LOG:  checkpoint complete: wrote 1558 buffers (9.5%), wrote 22 SLRU buffers; 0 WAL 
file(s) added, 0 removed, 33 recycled; write=7.544 s, sync=0.019 s, total=7.720 s; sync files=0, longest=0.000 s, 
average=0.000 s; distance=533837 kB, estimate=533837 kB; lsn=C/E532D478, redo lsn=C/C7054358
...

---
iteration 8: Mon Apr  7 18:26:47 UTC 2025
installchecks finished: Mon Apr  7 18:27:59 UTC 2025
2025-04-07 18:26:53.252 UTC [2359398] LOG:  could not read blocks 0..12 in file "base/4/1255": Operation canceled
2025-04-07 18:26:53.255 UTC [2359398] ERROR:  could not read blocks 0..12 in file "base/4/1255": Operation canceled
server.log:
...
2025-04-07 18:26:53.249 UTC [2048053] LOG:  checkpoint complete: wrote 3061 buffers (18.7%), wrote 3 SLRU buffers; 0 WAL 
file(s) added, 2 removed, 0 recycled; write=0.051 s, sync=0.002 s, total=0.068 s; sync files=0, longest=0.000 s, 
average=0.000 s; distance=18937 kB, estimate=120161 kB; lsn=11/6D42ED30, redo lsn=11/6C17AC30
2025-04-07 18:26:53.251 UTC [2359308] WARNING:  "most_common_elems" must be specified when "most_common_elem_freqs" is 
specified
2025-04-07 18:26:53.251 UTC [2359351] WARNING: "elem_count_histogram" array cannot contain NULL values
2025-04-07 18:26:53.252 UTC [2359375] ERROR:  cannot modify statistics for relation "testseq"
2025-04-07 18:26:53.252 UTC [2359375] DETAIL:  This operation is not supported for sequences.
2025-04-07 18:26:53.252 UTC [2359375] STATEMENT:  SELECT pg_catalog.pg_restore_relation_stats(
             'schemaname', 'stats_import',
             'relname', 'testseq');
2025-04-07 18:26:53.252 UTC [2359375] ERROR:  cannot modify statistics for relation "testseq"
2025-04-07 18:26:53.252 UTC [2359375] DETAIL:  This operation is not supported for sequences.
2025-04-07 18:26:53.252 UTC [2359375] STATEMENT:  SELECT pg_catalog.pg_clear_relation_stats(schemaname => 
'stats_import', relname => 'testseq');
2025-04-07 18:26:53.252 UTC [2359398] LOG:  could not read blocks 0..12 in file "base/4/1255": Operation canceled
2025-04-07 18:26:53.252 UTC [2359398] STATEMENT:  CREATE DATABASE regress020_tbd
         ENCODING utf8 LC_COLLATE "C" LC_CTYPE "C" TEMPLATE template0;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:  invalid input syntax for type pg_snapshot: "31:12:" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:  select '31:12:'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:  invalid input syntax for type pg_snapshot: "0:1:" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:  select '0:1:'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:  invalid input syntax for type pg_snapshot: "12:13:0" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:  select '12:13:0'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:  invalid input syntax for type pg_snapshot: "12:16:14,13" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:  select '12:16:14,13'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359398] ERROR:  could not read blocks 0..12 in file "base/4/1255": Operation canceled
2025-04-07 18:26:53.255 UTC [2359398] STATEMENT:  CREATE DATABASE regress020_tbd
         ENCODING utf8 LC_COLLATE "C" LC_CTYPE "C" TEMPLATE template0;
...
---
iteration 2: Mon Apr  7 19:02:57 UTC 2025
installchecks finished: Mon Apr  7 19:04:07 UTC 2025
2025-04-07 19:03:02.133 UTC [3537488] LOG:  could not read blocks 0..0 in file "base/902183/2607": Operation canceled
2025-04-07 19:03:02.133 UTC [3537488] ERROR:  could not read blocks 0..0 in file "base/902183/2607": Operation canceled
server.log:
2025-04-07 19:03:02.129 UTC [3487651] LOG:  checkpoint complete: wrote 337 buffers (2.1%), wrote 2 SLRU buffers; 0 WAL 
file(s) added, 0 removed, 0 recycled; write=0.013 s, sync=0.001 s, total=0.024 s; sync files=0, longest=0.000 s, 
average=0.000 s; distance=2555 kB, estimate=293963 kB; lsn=2/9D3DECE8, redo lsn=2/9D2AAC00
2025-04-07 19:03:02.129 UTC [3537184] ERROR:  invalid regular expression: invalid backreference number
2025-04-07 19:03:02.129 UTC [3537184] STATEMENT:  select 'xyz' ~ 'x(\w)(?=(\1))';
2025-04-07 19:03:02.129 UTC [3537184] ERROR:  invalid regular expression: invalid escape \ sequence
2025-04-07 19:03:02.129 UTC [3537184] STATEMENT:  select 'a' ~ '\x7fffffff';
2025-04-07 19:03:02.130 UTC [3537651] ERROR:  operator does not exist: point = box at character 23
2025-04-07 19:03:02.130 UTC [3537651] HINT:  No operator matches the given name and argument types. You might need to 
add explicit type casts.
2025-04-07 19:03:02.130 UTC [3537651] STATEMENT:  select '(0,0)'::point in ('(0,0,0,0)'::box, point(0,0));
2025-04-07 19:03:02.133 UTC [3537362] ERROR:  invalid regular expression: invalid backreference number
2025-04-07 19:03:02.133 UTC [3537362] STATEMENT:  select 'xyz' ~ 'x(\w)(?=\1)';
2025-04-07 19:03:02.133 UTC [3537362] ERROR:  invalid regular expression: invalid backreference number
2025-04-07 19:03:02.133 UTC [3537362] STATEMENT:  select 'xyz' ~ 'x(\w)(?=(\1))';
2025-04-07 19:03:02.133 UTC [3537362] ERROR:  invalid regular expression: invalid escape \ sequence
2025-04-07 19:03:02.133 UTC [3537362] STATEMENT:  select 'a' ~ '\x7fffffff';
2025-04-07 19:03:02.133 UTC [3537488] LOG:  could not read blocks 0..0 in file "base/902183/2607": Operation canceled
2025-04-07 19:03:02.133 UTC [3537488] STATEMENT:  SELECT p1.oid, p1.proname
     FROM pg_proc as p1
     WHERE 'cstring'::regtype = ANY (p1.proargtypes)
         AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typinput = p1.oid)
         AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
         AND p1.oid != 'shell_in(cstring)'::regprocedure
     ORDER BY 1;
2025-04-07 19:03:02.133 UTC [3537488] ERROR:  could not read blocks 0..0 in file "base/902183/2607": Operation canceled
2025-04-07 19:03:02.133 UTC [3537488] STATEMENT:  SELECT p1.oid, p1.proname
     FROM pg_proc as p1
     WHERE 'cstring'::regtype = ANY (p1.proargtypes)
         AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typinput = p1.oid)
         AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
         AND p1.oid != 'shell_in(cstring)'::regprocedure
     ORDER BY 1;
2025-04-07 19:03:02.135 UTC [3537671] WARNING:  TIME(7) WITH TIME ZONE precision reduced to maximum allowed, 6
2025-04-07 19:03:02.135 UTC [3537671] WARNING:  TIMESTAMP(7) WITH TIME ZONE precision reduced to maximum allowed, 6
2025-04-07 19:03:02.135 UTC [3537671] WARNING:  TIME(7) precision reduced to maximum allowed, 6
2025-04-07 19:03:02.136 UTC [3487651] LOG:  checkpoint starting: immediate force wait
...
So I suspect checkpointer, but I'm not sure yet.

(Sometimes (2 out of 5 runs) 10 iterations pass without the error.)

That script  with s/grep 'could not read blocks '/grep 'was terminated '/
also discovers another anomaly (right now on the third run, with 10
iterations of 30 installchecks):
Core was generated by `postgres: postgres regress005 [local] SELECT                  '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000591086d5ccfa in pgaio_io_get_target_description (ioh=ioh@entry=0x7ffcf961bc20) at aio_target.c:85
#2  0x0000591086d5bfdd in pg_get_aios (fcinfo=<optimized out>) at aio_funcs.c:218
#3  0x0000591086bb64c1 in ExecMakeTableFunctionResult (setexpr=0x5910bddcbf48, econtext=0x5910bddcbde8, 
argContext=<optimized out>,
     expectedDesc=0x5910bddff918, randomAccess=false) at execSRF.c:234
#4  0x0000591086bc9673 in FunctionNext (node=node@entry=0x5910bddcbbd8) at nodeFunctionscan.c:94
#5  0x0000591086bb7093 in ExecScanFetch (recheckMtd=0x591086bc9522 <FunctionRecheck>, accessMtd=0x591086bc95a2 
<FunctionNext>, epqstate=0x0,
     node=0x5910bddcbbd8) at ../../../src/include/executor/execScan.h:126
#6  ExecScanExtended (projInfo=0x0, qual=0x0, epqstate=0x0, recheckMtd=0x591086bc9522 <FunctionRecheck>, 
accessMtd=0x591086bc95a2 <FunctionNext>,
     node=0x5910bddcbbd8) at ../../../src/include/executor/execScan.h:170
#7  ExecScan (node=0x5910bddcbbd8, accessMtd=accessMtd@entry=0x591086bc95a2 <FunctionNext>, 
recheckMtd=recheckMtd@entry=0x591086bc9522 <FunctionRecheck>)
     at execScan.c:59
#8  0x0000591086bc9580 in ExecFunctionScan (pstate=<optimized out>) at nodeFunctionscan.c:269
#9  0x0000591086bb38f3 in ExecProcNodeFirst (node=0x5910bddcbbd8) at execProcnode.c:469
#10 0x0000591086bc0195 in ExecProcNode (node=0x5910bddcbbd8) at ../../../src/include/executor/executor.h:341
#11 0x0000591086bc021b in fetch_input_tuple (aggstate=aggstate@entry=0x5910bddcb480) at nodeAgg.c:563
#12 0x0000591086bc3825 in agg_retrieve_direct (aggstate=aggstate@entry=0x5910bddcb480) at nodeAgg.c:2450
#13 0x0000591086bc3a19 in ExecAgg (pstate=<optimized out>) at nodeAgg.c:2265
#14 0x0000591086bb38f3 in ExecProcNodeFirst (node=0x5910bddcb480) at execProcnode.c:469
#15 0x0000591086baac23 in ExecProcNode (node=node@entry=0x5910bddcb480) at ../../../src/include/executor/executor.h:341
#16 0x0000591086baacd0 in ExecutePlan (queryDesc=queryDesc@entry=0x5910bdde0e00, operation=operation@entry=CMD_SELECT, 
sendTuples=sendTuples@entry=true,
     numberTuples=numberTuples@entry=0, direction=direction@entry=ForwardScanDirection, dest=dest@entry=0x5910bddf6070) 
at execMain.c:1783
#17 0x0000591086bab79d in standard_ExecutorRun (queryDesc=0x5910bdde0e00, direction=ForwardScanDirection, count=0) at 
execMain.c:435
#18 0x0000591086bab7e0 in ExecutorRun (queryDesc=queryDesc@entry=0x5910bdde0e00, 
direction=direction@entry=ForwardScanDirection, count=count@entry=0)
     at execMain.c:371
#19 0x0000591086da9e9f in PortalRunSelect (portal=portal@entry=0x5910bdd72f10, forward=forward@entry=true, count=0, 
count@entry=9223372036854775807,
     dest=dest@entry=0x5910bddf6070) at pquery.c:953
#20 0x0000591086dab83b in PortalRun (portal=portal@entry=0x5910bdd72f10, count=count@entry=9223372036854775807, 
isTopLevel=isTopLevel@entry=true,
     dest=dest@entry=0x5910bddf6070, altdest=altdest@entry=0x5910bddf6070, qc=qc@entry=0x7ffcf961c9f0) at pquery.c:797
#21 0x0000591086da74a4 in exec_simple_query (query_string=query_string@entry=0x5910bdcebe60 "SELECT COUNT(*) >= 0 AS ok 
FROM pg_aios;") at postgres.c:1274

But I'm yet to construct a more reliable reproducer for it. Hope I could
do this during the current week.

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

  184. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-07T19:24:42Z

    Hi,


On 2025-04-05 06:43:52 -0700, Noah Misch wrote:
> Yeah.  Maybe this (untested):

Something like that works. I adopted your formulation of this, mine was in
GetLocalVictimBuffer(), which seems slightly less future proof.


> > > If that's right, it would still be nice to reach the right
> > > VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.
> >
> > I think that would be possible if we didn't do VALGRIND_MAKE_MEM_NOACCESS() in
> > UnpinBuffer()/UnpinLocalBuffer(). But with that I don't see how we can avoid
> > needing to remark the region as accessible?
>
> Yes, it's not that we should remove VALGRIND_MAKE_MEM_DEFINED() from bufmgr.
> I was trying to think about future AIO callers (e.g. RelationCopyStorage())
> and how they'd want things to work.

Ah, I now understand what you mean.  I'm inclined to leave that out for now,
we can do that later. I spent a bit of time experimenting with going bigger,
but I think it's important to get skink a bit less red.


> That said, perhaps we should just omit the io_uring-level Valgrind calls and
> delegate the problem to higher layers until there's a concrete use case:

Yes, I think that's the right answer for now. Applied your suggested comment.


I looked for a good place to add a comment to method_io_uring.c, but couldn't
really come up with anything convincing.  Then I thought the
 * "Start" routines for individual IO operations
comment in aio_io.c might be a good place, but also failed to come up with
anything particularly convincing.


> > > - A complete_local callback solves those problems.  However, if the
> > >   AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
> > >   the buffer mapping may have changed by the time of complete_local.
> >
> > I don't think that is possible, due to the aio subsystem owned pin?
>
> We currently drop the shared buffer AIO subsystem pin in complete_shared
> (buffer_readv_complete_one() -> TerminateBufferIO()).  I was trying to say
> that if we put a VALGRIND_MAKE_MEM_DEFINED in complete_local without changing
> anything else, it would have this problem.

Ah, yes, I see what you mean.


> We probably wouldn't want to move the shared buffer pin drop to
> complete_local without a strong reason.

Agreed.


> > I think the point about non-aio uses is a fair one, but I don't quite know how
> > to best solve it right now, due to the local buffer issue you mentioned. I'd
> > guess that we'd best put it somewhere
> > a) in pgaio_io_process_completion(), if definer==completor || !PGAIO_HF_REFERENCES_LOCAL
> > b) pgaio_io_call_complete_local(), just before calling
> >    pgaio_io_call_complete_local() if PGAIO_HF_REFERENCES_LOCAL
>
> I think that would do nothing wrong today, but it uses
> !PGAIO_HF_REFERENCES_LOCAL as a proxy for "target memory is already DEFINED,
> or a higher layer will make it DEFINED".  While !PGAIO_HF_REFERENCES_LOCAL
> does imply that today, I think that's a bufmgr-specific conclusion.  I have no
> particular reason to expect that to hold for future AIO use cases.  As above,
> I'd be inclined to omit the io_uring-level Valgrind calls until we have a
> concrete use case to drive their design.  How do you see it?

Yea, I'm not sure either.  I think we'll best wait until we have non-bufmgr
AIO to address all this. That'll make it easier to shake out.



I think eventually we should do an explicit
- VALGRIND_CHECK_MEM_IS_ADDRESSABLE() in pgaio_io_start_readv

- VALGRIND_CHECK_MEM_IS_DEFINED() in pgaio_io_start_writev

- VALGRIND_MAKE_MEM_DEFINED() in when a READV completes, although some of the
  details of when/where to do that aren't entirely clear to me yet. I suspect
  we might have to do it in pgaio_io_start_readv(), because it's harder to
  reliably do it later


I've pushed the patches. Thanks for the discussion, somehow the mix of shared
memory with valgrind tracking accessibility/definedness in a process local
manner is somewhat mindbending.

Greetings,

Andres Freund

  185. Re: AIO v2.5

    Alexander Lakhin <exclusion@gmail.com> — 2025-04-13T06:00:01Z

    Hello Andres,

07.04.2025 22:10, Alexander Lakhin wrote:
>> I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
>> tmpfs.
>
> Before sharing the script I tested it on two my machines, but I had
> anticipated that the error can be hard to reproduce. Will try to reduce
> the reproducer...

I've managed to reduce it to the following:
ulimit -n 4096

echo "
fsync = off
autovacuum = off

checkpoint_timeout = 30s

io_max_concurrency = 10
io_method = io_uring
" >> $PGDATA/postgresql.conf

pg_ctl -l server.log start

for i in `seq 1000`; do
   numjobs=$((20 + $RANDOM % 60))
   echo "iteration $i (jobs: $numjobs)"
   date
   for ((j=1;j<=numjobs;j++)); do
     (
       createdb db$j;
       for ((n=1;n<=50;n++)); do
         cat << EOF | psql -d db$j -a >>/dev/null 2>&1
DROP TABLE IF EXISTS tenk1;
CREATE TABLE tenk1 (
     unique1     int4,
     unique2     int4,
     two         int4,
     four        int4,
     ten         int4,
     twenty      int4,
     hundred     int4,
     thousand    int4,
     twothousand int4,
     fivethous   int4,
     tenthous    int4,
     odd         int4,
     even        int4,
     stringu1    name,
     stringu2    name,
     string4     name
);
COPY tenk1 FROM '.../src/test/regress/data/tenk.data';
EOF
       done;
     ) &
   done
   wait

   for ((j=1;j<=numjobs;j++)); do dropdb db$j & done
   wait
   grep -A3 -E '(ERROR|could not read blocks )' server.log && break;
done

pg_ctl stop

It fails for me as below:
iteration 13 (jobs: 25)
Sun Apr 13 05:31:47 AM UTC 2025
iteration 14 (jobs: 67)
Sun Apr 13 05:31:50 AM UTC 2025
dropdb: error: database removal failed: ERROR:  could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-13 05:31:58.930 UTC [1153451] LOG:  could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-13 05:31:58.930 UTC [1153451] CONTEXT:  completing I/O on behalf of process 1153456
2025-04-13 05:31:58.930 UTC [1153451] STATEMENT:  DROP DATABASE db5;
2025-04-13 05:31:58.930 UTC [1153456] ERROR:  could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-13 05:31:58.930 UTC [1153456] STATEMENT:  DROP DATABASE db6;
2025-04-13 05:31:58.931 UTC [1034758] LOG:  checkpoint complete: wrote 3 buffers (0.0%), wrote 0 SLRU buffers; 0 WAL 
file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.002 s; sync files=0, longest=0.000 s, 
average=0.000 s; distance=18 kB, estimate=458931 kB; lsn=16/54589E08, redo lsn=16/54586F88
2025-04-13 05:31:58.931 UTC [1034758] LOG:  checkpoint starting: immediate force wait


I reproduced this error on three different machines (all are running
Ubuntu 24.04, two with kernel version 6.8, one with 6.11), with PGDATA
located on tmpfs.

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

  186. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-14T16:06:24Z

    Hi,

On 2025-04-13 09:00:01 +0300, Alexander Lakhin wrote:
> 07.04.2025 22:10, Alexander Lakhin wrote:
> > > I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
> > > tmpfs.
> > 
> > Before sharing the script I tested it on two my machines, but I had
> > anticipated that the error can be hard to reproduce. Will try to reduce
> > the reproducer...
> 
> I've managed to reduce it to the following:

Thanks a lot for working on that!


> [reproducer]
> 
> It fails for me as below:
> iteration 13 (jobs: 25)
> Sun Apr 13 05:31:47 AM UTC 2025
> iteration 14 (jobs: 67)
> Sun Apr 13 05:31:50 AM UTC 2025
> dropdb: error: database removal failed: ERROR:  could not read blocks 0..0 in file "global/1213": Operation canceled
> 2025-04-13 05:31:58.930 UTC [1153451] LOG:  could not read blocks 0..0 in file "global/1213": Operation canceled
> 2025-04-13 05:31:58.930 UTC [1153451] CONTEXT:  completing I/O on behalf of process 1153456
> 2025-04-13 05:31:58.930 UTC [1153451] STATEMENT:  DROP DATABASE db5;
> 2025-04-13 05:31:58.930 UTC [1153456] ERROR:  could not read blocks 0..0 in file "global/1213": Operation canceled
> 2025-04-13 05:31:58.930 UTC [1153456] STATEMENT:  DROP DATABASE db6;
> 2025-04-13 05:31:58.931 UTC [1034758] LOG:  checkpoint complete: wrote 3
> buffers (0.0%), wrote 0 SLRU buffers; 0 WAL file(s) added, 0 removed, 0
> recycled; write=0.002 s, sync=0.001 s, total=0.002 s; sync files=0,
> longest=0.000 s, average=0.000 s; distance=18 kB, estimate=458931 kB;
> lsn=16/54589E08, redo lsn=16/54586F88
> 2025-04-13 05:31:58.931 UTC [1034758] LOG:  checkpoint starting: immediate force wait

Unfortunately I'm several hundred iterations in, without reproducing the
issue. I'm bad at statistics, but I think that makes it rather unlikely that I
will, without changing some aspect.

Was this an assert enabled build? What compiler and what optimization settings
did you use? Do you have huge pages configured (so that the default
huge_pages=try would end up with huge pages)?

So far I've been trying to use a cassert enabled build built with -O0, without
huge pages. After the current test run I'll switch to cassert+-O2.



> I reproduced this error on three different machines (all are running
> Ubuntu 24.04, two with kernel version 6.8, one with 6.11), with PGDATA
> located on tmpfs.

That's another variable to try - so far I've been trying this on 6.15.0-rc1
[1].  I guess I'll have to set up a ubuntu 24.04 VM and try with that.

Greetings,

Andres Freund


[1] I wanted to play with io_uring changes that were recently merged. Namely
support for readv/writev of "fixed" buffers. That avoids needing to pin/unpin
buffers while IO is ongoing, which turns out to be a noticeable bottleneck in
some workloads, particularly when using 1GB huge pages.

  187. Re: AIO v2.5

    Alexander Lakhin <exclusion@gmail.com> — 2025-04-15T18:00:00Z

    Hello Andres,

14.04.2025 19:06, Andres Freund wrote:
> Unfortunately I'm several hundred iterations in, without reproducing the
> issue. I'm bad at statistics, but I think that makes it rather unlikely that I
> will, without changing some aspect.
>
> Was this an assert enabled build? What compiler and what optimization settings
> did you use? Do you have huge pages configured (so that the default
> huge_pages=try would end up with huge pages)?

Yes, I used --enable-cassert; no explicit optimization setting and no huge
pages configured. pg_config says:
CONFIGURE =  '--enable-debug' '--enable-cassert' '--enable-tap-tests' '--with-liburing'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels 
-Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wshadow=compatible-local -Wformat-security 
-fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2

Please look at the complete script attached. I've just run it and got:
iteration 56 (jobs: 44)
Tue Apr 15 06:30:52 PM CEST 2025
dropdb: error: database removal failed: ERROR:  could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-15 18:31:00.650 CEST [1612266] LOG:  could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-15 18:31:00.650 CEST [1612266] CONTEXT:  completing I/O on behalf of process 1612271
2025-04-15 18:31:00.650 CEST [1612266] STATEMENT:  DROP DATABASE db3;

I used gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, but now I've also
reproduced the issue with CC=clang (18.1.3 (1ubuntu1)).

Please take a look also at the simple reproducer for the crash inside
pg_get_aios() I mentioned upthread:
for i in {1..100}; do
   numjobs=12
   echo "iteration $i"
   date
   for ((j=1;j<=numjobs;j++)); do
     ( createdb db$j; for k in {1..300}; do
         echo "CREATE TABLE t (a INT); CREATE INDEX ON t (a); VACUUM t;
               SELECT COUNT(*) >= 0 AS ok FROM pg_aios; " \
         | psql -d db$j >/dev/null 2>&1;
       done; dropdb db$j; ) &
   done
   wait
   psql -c 'SELECT 1' || break;
done

it fails for me as follows:
iteration 20
Tue Apr 15 07:21:29 PM EEST 2025
dropdb: error: connection to server on socket "/tmp/.s.PGSQL.55432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
...
2025-04-15 19:21:30.675 EEST [3111699] LOG:  client backend (PID 3320979) was terminated by signal 11: Segmentation fault
2025-04-15 19:21:30.675 EEST [3111699] DETAIL:  Failed process was running: SELECT COUNT(*) >= 0 AS ok FROM pg_aios;
2025-04-15 19:21:30.675 EEST [3111699] LOG:  terminating any other active server processes

>> I reproduced this error on three different machines (all are running
>> Ubuntu 24.04, two with kernel version 6.8, one with 6.11), with PGDATA
>> located on tmpfs.
> That's another variable to try - so far I've been trying this on 6.15.0-rc1
> [1].  I guess I'll have to set up a ubuntu 24.04 VM and try with that.
>
> Greetings,
>
> Andres Freund
>
>
> [1] I wanted to play with io_uring changes that were recently merged. Namely
> support for readv/writev of "fixed" buffers. That avoids needing to pin/unpin
> buffers while IO is ongoing, which turns out to be a noticeable bottleneck in
> some workloads, particularly when using 1GB huge pages.

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

  188. Re: AIO v2.5

    Alexander Lakhin <exclusion@gmail.com> — 2025-04-20T15:00:00Z

    Hello Andres,

31.03.2025 02:46, Andres Freund wrote:
> I've pushed most of these after some very light further editing.  Yay.  Thanks
> a lot for all the reviews!
>
> So far the buildfarm hasn't been complaining, but it's early days.

I found one complaint against commit 12ce89fd0, expressed as:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=scorpion&dt=2025-04-08%2001%3A36%3A41

149/334 postgresql:recovery / recovery/032_relfilenode_reuse ERROR           206.41s   (exit status 255 or signal 127 
SIGinvalid)

[01:41:53.249](0.620s) ok 8 - standby: post move contents as expected
[01:44:55.175](181.927s) Bail out!  aborting wait: program timed out
stream contents: >><<
pattern searched for: (?^m:warmed_buffers)

032_relfilenode_reuse_standby.log contains:
2025-04-08 01:41:54.674 UTC [4013024][startup][33/0:0] DEBUG: waiting for all backends to process ProcSignalBarrier 
generation 7
2025-04-08 01:41:54.674 UTC [4013024][startup][33/0:0] CONTEXT:  WAL redo at 0/43E5360 for Database/DROP: dir 1663/50001
2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for self with 0 pending
2025-04-08 01:41:54.674 UTC [4013251][client backend][0/2:0] ERROR: no free IOs despite no in-flight IOs
2025-04-08 01:41:54.674 UTC [4013251][client backend][0/2:0] STATEMENT:  SELECT SUM(pg_prewarm(oid)) warmed_buffers FROM 
pg_class WHERE pg_relation_filenode(oid) != 0;

I could reproduce this error with the following TEMP_CONFIG:
log_min_messages = DEBUG4

log_connections = on
log_disconnections = on
log_statement = 'all'
log_line_prefix = '%m [%d][%p:%l][%b] %q[%a] '

fsync = on
io_method=io_uring
backtrace_functions = 'pgaio_io_wait_for_free'

When running 032_relfilenode_reuse.pl in a loop, I got failures on
iterations 10, 18, 6:
2025-04-20 15:56:01.310 CEST [][1517068:122][startup] DEBUG: updated min recovery point to 0/4296E90 on timeline 1
2025-04-20 15:56:01.310 CEST [][1517068:123][startup] CONTEXT:  WAL redo at 0/4296E48 for Transaction/COMMIT: 2025-04-20 
15:56:01.09363+02; inval msgs: catcache 21; sync
2025-04-20 15:56:01.310 CEST [][1517068:124][startup] DEBUG: waiting for all backends to process ProcSignalBarrier 
generation 5
2025-04-20 15:56:01.310 CEST [][1517068:125][startup] CONTEXT:  WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
2025-04-20 15:56:01.312 CEST [postgres][1517075:144][client backend] [032_relfilenode_reuse.pl] DEBUG:  waiting for self 
with 0 pending
2025-04-20 15:56:01.312 CEST [postgres][1517075:145][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
pgaio_io_wait_for_free at aio.c:703:2
  (inlined by) pgaio_io_acquire at aio.c:186:3
AsyncReadBuffers at bufmgr.c:1854:9
StartReadBuffersImpl at bufmgr.c:1425:18
  (inlined by) StartReadBuffers at bufmgr.c:1497:9
read_stream_start_pending_read at read_stream.c:333:25
read_stream_look_ahead at read_stream.c:475:3
read_stream_next_buffer at read_stream.c:840:6
pg_prewarm at pg_prewarm.c:214:10
ExecInterpExpr at execExprInterp.c:927:7
ExecEvalExprNoReturn at executor.h:447:2
...
2025-04-20 15:56:01.312 CEST [][1517068:126][startup] DEBUG: finished waiting for all backends to process 
ProcSignalBarrier generation 5
2025-04-20 15:56:01.312 CEST [][1517068:127][startup] CONTEXT:  WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
2025-04-20 15:56:01.314 CEST [postgres][1517075:146][client backend] [032_relfilenode_reuse.pl] ERROR:  no free IOs 
despite no in-flight IOs
2025-04-20 15:56:01.314 CEST [postgres][1517075:147][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
pgaio_io_wait_for_free at aio.c:735:3
  (inlined by) pgaio_io_acquire at aio.c:186:3
AsyncReadBuffers at bufmgr.c:1854:9
StartReadBuffersImpl at bufmgr.c:1425:18
  (inlined by) StartReadBuffers at bufmgr.c:1497:9
read_stream_start_pending_read at read_stream.c:333:25
read_stream_look_ahead at read_stream.c:475:3
read_stream_next_buffer at read_stream.c:840:6
pg_prewarm at pg_prewarm.c:214:10
ExecInterpExpr at execExprInterp.c:927:7
ExecEvalExprNoReturn at executor.h:447:2
...

I configured the build with:
CPPFLAGS="-O1" ./configure --enable-injection-points --enable-cassert --enable-debug --enable-tap-tests --with-liburing

I also encountered another failure when running this test:
t/032_relfilenode_reuse.pl .. 1/? Bailout called.  Further testing stopped:  aborting wait: program died

Core was generated by `postgres: standby: law postgres [local] SELECT                   '.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x0000700a5a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x0000700a5a2288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x0000630ff2c5dcea in errfinish (filename=filename@entry=0x630ff2cdd139 "aio.c", lineno=lineno@entry=559,
     funcname=funcname@entry=0x630ff2e85da8 <__func__.9> "pgaio_io_wait") at elog.c:599
#6  0x0000630ff2aae79c in pgaio_io_wait (ioh=ioh@entry=0x700a57b59700, ref_generation=3972) at aio.c:559
#7  0x0000630ff2aaefab in pgaio_io_wait_for_free () at aio.c:771
#8  pgaio_io_acquire (resowner=0x63100114dee8, ret=ret@entry=0x6310011efec8) at aio.c:186
#9  0x0000630ff2ab9137 in AsyncReadBuffers (operation=operation@entry=0x6310011efe88, 
nblocks_progress=nblocks_progress@entry=0x7ffd8d262034) at bufmgr.c:1854
#10 0x0000630ff2abc6cc in StartReadBuffersImpl (allow_forwarding=true, flags=0, nblocks=0x7ffd8d262034, blockNum=0, 
buffers=0x6310011efe3c,
     operation=0x6310011efe88) at bufmgr.c:1425
#11 StartReadBuffers (operation=0x6310011efe88, buffers=buffers@entry=0x6310011efe3c, blockNum=0, 
nblocks=nblocks@entry=0x7ffd8d262034, flags=flags@entry=0)
     at bufmgr.c:1497
#12 0x0000630ff2ab2f3f in read_stream_start_pending_read (stream=stream@entry=0x6310011efde0) at read_stream.c:328
#13 0x0000630ff2ab3363 in read_stream_look_ahead (stream=stream@entry=0x6310011efde0) at read_stream.c:475
#14 0x0000630ff2ab37e5 in read_stream_next_buffer (stream=stream@entry=0x6310011efde0, 
per_buffer_data=per_buffer_data@entry=0x0) at read_stream.c:837
#15 0x0000700a5adae60c in pg_prewarm (fcinfo=<optimized out>) at pg_prewarm.c:214
#16 0x0000630ff29004cc in ExecInterpExpr (state=0x6310012239c8, econtext=0x6310011e1920, isnull=0x0) at execExprInterp.c:926
#17 0x0000630ff291bfb4 in ExecEvalExprNoReturn (econtext=<optimized out>, state=<optimized out>) at 
../../../src/include/executor/executor.h:445
#18 ExecEvalExprNoReturnSwitchContext (econtext=<optimized out>, state=<optimized out>) at 
../../../src/include/executor/executor.h:486
#19 advance_aggregates (aggstate=aggstate@entry=0x6310011e14f8) at nodeAgg.c:820
#20 0x0000630ff291db18 in agg_retrieve_direct (aggstate=0x6310011e14f8) at nodeAgg.c:2540
#21 ExecAgg (pstate=0x6310011e14f8) at nodeAgg.c:2265
#22 0x0000630ff290deef in ExecProcNodeFirst (node=0x6310011e14f8) at execProcnode.c:469
...

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

  189. Re: AIO v2.5

    Tomas Vondra <tomas@vondra.me> — 2025-04-22T22:38:01Z

    On 3/18/25 21:12, Andres Freund wrote:
> Hi,
> 
> Attached is v2.10, with the following changes:
> 
>  ...
> 
> - committed io_method=worker
> 

There's an open item related to this commit (247ce06b883d), based on:

    For now the default io_method is changed to "worker". We should re-
    evaluate that around beta1, we might want to be careful and set the
    default to "sync" for 18.

FWIW the open item is in the "recheck mid-beta" section, but the commit
message says "around beta1" which is not that far away (especially if we
choose to do beta1 on May 8, with the minor releases).

What information we need to gather to make the decision and who's
expected to do it? I assume someone needs to do run some benchmarks, but
is anyone working on it already? What benchmarks, what platforms?

FWIW I'm asking because of the RMT, but I'm also willing to do some of
the tests, if needed - but it'd be good to get some guidance.


regards

-- 
Tomas Vondra

  190. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-24T00:40:48Z

    Hi,

On 2025-04-20 18:00:00 +0300, Alexander Lakhin wrote:
> 31.03.2025 02:46, Andres Freund wrote:
> > I've pushed most of these after some very light further editing.  Yay.  Thanks
> > a lot for all the reviews!
> > 
> > So far the buildfarm hasn't been complaining, but it's early days.
> 
> I found one complaint against commit 12ce89fd0, expressed as:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=scorpion&dt=2025-04-08%2001%3A36%3A41

Thanks for pointing that out and repro'ing it!


> 2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for self with 0 pending

I'd like to extend this debug message with the number of in-flight IOs.  I
assume nobody will mind me doing that.



> I could reproduce this error with the following TEMP_CONFIG:
> log_min_messages = DEBUG4
> 
> log_connections = on
> log_disconnections = on
> log_statement = 'all'
> log_line_prefix = '%m [%d][%p:%l][%b] %q[%a] '
> 
> fsync = on
> io_method=io_uring
> backtrace_functions = 'pgaio_io_wait_for_free'

Hm. Several hundred iterations in without triggering the issue even once, with
that added config. Given your reproducer used fsync, I assume this was on a
real filesystem not tmpfs? What FS?  I tried on XFS.

very well could just be a timing dependent issue, the time for an fsync will
differ heavily betweeen devices...


> When running 032_relfilenode_reuse.pl in a loop, I got failures on
> iterations 10, 18, 6:
> 2025-04-20 15:56:01.310 CEST [][1517068:122][startup] DEBUG: updated min recovery point to 0/4296E90 on timeline 1
> 2025-04-20 15:56:01.310 CEST [][1517068:123][startup] CONTEXT:  WAL redo at
> 0/4296E48 for Transaction/COMMIT: 2025-04-20 15:56:01.09363+02; inval msgs:
> catcache 21; sync
> 2025-04-20 15:56:01.310 CEST [][1517068:124][startup] DEBUG: waiting for all
> backends to process ProcSignalBarrier generation 5
> 2025-04-20 15:56:01.310 CEST [][1517068:125][startup] CONTEXT:  WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
> 2025-04-20 15:56:01.312 CEST [postgres][1517075:144][client backend]
> [032_relfilenode_reuse.pl] DEBUG:  waiting for self with 0 pending
> 2025-04-20 15:56:01.312 CEST [postgres][1517075:145][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
> pgaio_io_wait_for_free at aio.c:703:2
>  (inlined by) pgaio_io_acquire at aio.c:186:3
> AsyncReadBuffers at bufmgr.c:1854:9
> StartReadBuffersImpl at bufmgr.c:1425:18
>  (inlined by) StartReadBuffers at bufmgr.c:1497:9
> read_stream_start_pending_read at read_stream.c:333:25
> read_stream_look_ahead at read_stream.c:475:3
> read_stream_next_buffer at read_stream.c:840:6
> pg_prewarm at pg_prewarm.c:214:10
> ExecInterpExpr at execExprInterp.c:927:7
> ExecEvalExprNoReturn at executor.h:447:2
> ...
> 2025-04-20 15:56:01.312 CEST [][1517068:126][startup] DEBUG: finished
> waiting for all backends to process ProcSignalBarrier generation 5
> 2025-04-20 15:56:01.312 CEST [][1517068:127][startup] CONTEXT:  WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
> 2025-04-20 15:56:01.314 CEST [postgres][1517075:146][client backend]
> [032_relfilenode_reuse.pl] ERROR:  no free IOs despite no in-flight IOs

The earlier report also had a ProcSignalBarrier wait just before. That's
doesn't have to be related, the test stresses those pretty intentionally.

But it does seem like it could be related, perhaps we somehow end processing
interrupts while issuing IOs, which then again submits IOs or something? I
don't immediately see such code, e.g. pgaio_closing_fd() is careful to iterate
over the list while waiting.

I think pgaio_closing_fd() should get a DEBUG2 (or so) message documenting
that we are waiting for an IO due to closing an FD. pgaio_shutdown() as well,
while at it.


I somewhat suspect that this is the same issue that you reported earlier, that
I also couldn't repro.  That involved a lot of CREATE DATABASEs, which will
trigger a lot of procsignal barriers...  I'll try to come up with a more
dedicated stress test and see whether I can trigger a problem that way.


> I also encountered another failure when running this test:
> t/032_relfilenode_reuse.pl .. 1/? Bailout called.  Further testing stopped:  aborting wait: program died
> 
> Core was generated by `postgres: standby: law postgres [local] SELECT                   '.
> Program terminated with signal SIGABRT, Aborted.
> #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
> 
> (gdb) bt
> #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
> #1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
> #2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
> #3  0x0000700a5a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> #4  0x0000700a5a2288ff in __GI_abort () at ./stdlib/abort.c:79
> #5  0x0000630ff2c5dcea in errfinish (filename=filename@entry=0x630ff2cdd139 "aio.c", lineno=lineno@entry=559,
>     funcname=funcname@entry=0x630ff2e85da8 <__func__.9> "pgaio_io_wait") at elog.c:599
> #6  0x0000630ff2aae79c in pgaio_io_wait (ioh=ioh@entry=0x700a57b59700, ref_generation=3972) at aio.c:559
> #7  0x0000630ff2aaefab in pgaio_io_wait_for_free () at aio.c:771

Do you still have the core file for this? Or at least the generated log
message? It'd be useful to know what state the IO was in when the error was
triggered...

Greetings,

Andres Freund

  191. Re: AIO v2.5

    Alexander Lakhin <exclusion@gmail.com> — 2025-04-24T04:04:04Z

    Hello Andres,

24.04.2025 03:40, Andres Freund wrote:
> Hi,
>
> On 2025-04-20 18:00:00 +0300, Alexander Lakhin wrote:
>
>> 2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for self with 0 pending
> I'd like to extend this debug message with the number of in-flight IOs.  I
> assume nobody will mind me doing that.

I printed that number for debugging and always got 1, because
io_max_concurrency == 1 during that test (because of shared_buffers=1MB).
With shared_buffers increased:
--- a/src/test/recovery/t/032_relfilenode_reuse.pl
+++ b/src/test/recovery/t/032_relfilenode_reuse.pl
@@ -18,7 +18,7 @@ log_connections=on
  # to avoid "repairing" corruption
  full_page_writes=off
  log_min_messages=debug2
-shared_buffers=1MB
+shared_buffers=16MB

I got 100 iterations passed.


>> I could reproduce this error with the following TEMP_CONFIG:
>> log_min_messages = DEBUG4
>>
>> log_connections = on
>> log_disconnections = on
>> log_statement = 'all'
>> log_line_prefix = '%m [%d][%p:%l][%b] %q[%a] '
>>
>> fsync = on
>> io_method=io_uring
>> backtrace_functions = 'pgaio_io_wait_for_free'
> Hm. Several hundred iterations in without triggering the issue even once, with
> that added config. Given your reproducer used fsync, I assume this was on a
> real filesystem not tmpfs? What FS?  I tried on XFS.
>
> very well could just be a timing dependent issue, the time for an fsync will
> differ heavily betweeen devices...

Yeah, it definitely depends on timing.
Using that TEMP_CONFIG changes the duration of the test significantly for me:
Files=1, Tests=14,  2 wallclock secs ( 0.01 usr  0.01 sys +  0.13 cusr  0.32 csys =  0.47 CPU)
->
Files=1, Tests=14, 14 wallclock secs ( 0.00 usr  0.00 sys +  0.14 cusr  0.33 csys =  0.47 CPU)

I reproduced the failure on ext4 (desktop-grade NVME) and on xfs (cheap
SSD), on two different machines.

It's also reproduced on HDD (with ext4), seemingly worse, where the
duration is increased even more:
I 25
# +++ tap check in src/test/recovery +++
t/032_relfilenode_reuse.pl .. ok
All tests successful.
Files=1, Tests=14, 28 wallclock secs ( 0.00 usr  0.00 sys +  0.15 cusr  0.39 csys =  0.54 CPU)
Result: PASS
I 26
# +++ tap check in src/test/recovery +++
t/032_relfilenode_reuse.pl .. 7/? Bailout called.  Further testing stopped:  aborting wait: program timed out


>> When running 032_relfilenode_reuse.pl in a loop, I got failures on
>> iterations 10, 18, 6:
>> 2025-04-20 15:56:01.310 CEST [][1517068:122][startup] DEBUG: updated min recovery point to 0/4296E90 on timeline 1
>> 2025-04-20 15:56:01.310 CEST [][1517068:123][startup] CONTEXT:  WAL redo at
>> 0/4296E48 for Transaction/COMMIT: 2025-04-20 15:56:01.09363+02; inval msgs:
>> catcache 21; sync
>> 2025-04-20 15:56:01.310 CEST [][1517068:124][startup] DEBUG: waiting for all
>> backends to process ProcSignalBarrier generation 5
>> 2025-04-20 15:56:01.310 CEST [][1517068:125][startup] CONTEXT:  WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
>> 2025-04-20 15:56:01.312 CEST [postgres][1517075:144][client backend]
>> [032_relfilenode_reuse.pl] DEBUG:  waiting for self with 0 pending
>> 2025-04-20 15:56:01.312 CEST [postgres][1517075:145][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
>> pgaio_io_wait_for_free at aio.c:703:2
>>   (inlined by) pgaio_io_acquire at aio.c:186:3
>> AsyncReadBuffers at bufmgr.c:1854:9
>> StartReadBuffersImpl at bufmgr.c:1425:18
>>   (inlined by) StartReadBuffers at bufmgr.c:1497:9
>> read_stream_start_pending_read at read_stream.c:333:25
>> read_stream_look_ahead at read_stream.c:475:3
>> read_stream_next_buffer at read_stream.c:840:6
>> pg_prewarm at pg_prewarm.c:214:10
>> ExecInterpExpr at execExprInterp.c:927:7
>> ExecEvalExprNoReturn at executor.h:447:2
>> ...
>> 2025-04-20 15:56:01.312 CEST [][1517068:126][startup] DEBUG: finished
>> waiting for all backends to process ProcSignalBarrier generation 5
>> 2025-04-20 15:56:01.312 CEST [][1517068:127][startup] CONTEXT:  WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
>> 2025-04-20 15:56:01.314 CEST [postgres][1517075:146][client backend]
>> [032_relfilenode_reuse.pl] ERROR:  no free IOs despite no in-flight IOs
> The earlier report also had a ProcSignalBarrier wait just before. That's
> doesn't have to be related, the test stresses those pretty intentionally.
>
> But it does seem like it could be related, perhaps we somehow end processing
> interrupts while issuing IOs, which then again submits IOs or something? I
> don't immediately see such code, e.g. pgaio_closing_fd() is careful to iterate
> over the list while waiting.
>
> I think pgaio_closing_fd() should get a DEBUG2 (or so) message documenting
> that we are waiting for an IO due to closing an FD. pgaio_shutdown() as well,
> while at it.
>
>
> I somewhat suspect that this is the same issue that you reported earlier, that
> I also couldn't repro.  That involved a lot of CREATE DATABASEs, which will
> trigger a lot of procsignal barriers...  I'll try to come up with a more
> dedicated stress test and see whether I can trigger a problem that way.

Maybe it does matter that "no free IOs despite no in-flight IOs" follows
"WAL redo at 0/4296E90 for Database/DROP" here (that's seemingly constant)...

I've tried also:
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -705,6 +705,7 @@ pgaio_io_wait_for_free(void)
         pgaio_debug(DEBUG2, "waiting for self with %d pending",
pgaio_my_backend->num_staged_ios);

+pg_sleep(3000);

and got the test failed on each run with:
t/032_relfilenode_reuse.pl .. Dubious, test returned 29 (wstat 7424, 0x1d00)

032_relfilenode_reuse_primary.log
2025-04-24 05:29:27.103 CEST [3504475] 032_relfilenode_reuse.pl LOG:  statement: UPDATE large SET datab = 1;
2025-04-24 05:29:27.104 CEST [3504475] 032_relfilenode_reuse.pl DEBUG:  waiting for self with 0 pending
2025-04-24 05:29:27.112 CEST [3504369] DEBUG:  releasing pm child slot 3
2025-04-24 05:29:27.112 CEST [3504369] LOG:  client backend (PID 3504475) was terminated by signal 11: Segmentation fault
2025-04-24 05:29:27.112 CEST [3504369] DETAIL:  Failed process was running: UPDATE large SET datab = 1;


>> I also encountered another failure when running this test:
>> t/032_relfilenode_reuse.pl .. 1/? Bailout called.  Further testing stopped:  aborting wait: program died
>>
>> Core was generated by `postgres: standby: law postgres [local] SELECT                   '.
>> Program terminated with signal SIGABRT, Aborted.
>> #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
>>
>> (gdb) bt
>> #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
>> #1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
>> #2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
>> #3  0x0000700a5a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
>> #4  0x0000700a5a2288ff in __GI_abort () at ./stdlib/abort.c:79
>> #5  0x0000630ff2c5dcea in errfinish (filename=filename@entry=0x630ff2cdd139 "aio.c", lineno=lineno@entry=559,
>>      funcname=funcname@entry=0x630ff2e85da8 <__func__.9> "pgaio_io_wait") at elog.c:599
>> #6  0x0000630ff2aae79c in pgaio_io_wait (ioh=ioh@entry=0x700a57b59700, ref_generation=3972) at aio.c:559
>> #7  0x0000630ff2aaefab in pgaio_io_wait_for_free () at aio.c:771
> Do you still have the core file for this? Or at least the generated log
> message? It'd be useful to know what state the IO was in when the error was
> triggered...

I can reproduce it easily, so here is the message:
2025-04-24 06:10:01.924 EEST [postgres][3036378:150][client backend] [032_relfilenode_reuse.pl] PANIC:  waiting for own 
IO in wrong state: 0
2025-04-24 06:10:01.924 EEST [postgres][3036378:151][client backend] [032_relfilenode_reuse.pl] STATEMENT:  SELECT 
SUM(pg_prewarm(oid)) warmed_buffers FROM pg_class WHERE pg_relation_filenode(oid) != 0;

(Sorry for not including it before — I observed it many times, always with
state 0, so it had lost its value for me.)

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

  192. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-25T16:44:14Z

    Hi,

On 2025-04-15 21:00:00 +0300, Alexander Lakhin wrote:
> Please take a look also at the simple reproducer for the crash inside
> pg_get_aios() I mentioned upthread:
> for i in {1..100}; do
>   numjobs=12
>   echo "iteration $i"
>   date
>   for ((j=1;j<=numjobs;j++)); do
>     ( createdb db$j; for k in {1..300}; do
>         echo "CREATE TABLE t (a INT); CREATE INDEX ON t (a); VACUUM t;
>               SELECT COUNT(*) >= 0 AS ok FROM pg_aios; " \
>         | psql -d db$j >/dev/null 2>&1;
>       done; dropdb db$j; ) &
>   done
>   wait
>   psql -c 'SELECT 1' || break;
> done
> 
> it fails for me as follows:
> iteration 20
> Tue Apr 15 07:21:29 PM EEST 2025
> dropdb: error: connection to server on socket "/tmp/.s.PGSQL.55432" failed: No such file or directory
>        Is the server running locally and accepting connections on that socket?
> ...
> 2025-04-15 19:21:30.675 EEST [3111699] LOG:  client backend (PID 3320979) was terminated by signal 11: Segmentation fault
> 2025-04-15 19:21:30.675 EEST [3111699] DETAIL:  Failed process was running: SELECT COUNT(*) >= 0 AS ok FROM pg_aios;
> 2025-04-15 19:21:30.675 EEST [3111699] LOG:  terminating any other active server processes

Thanks for that.  The bug turns out to be pretty stupid - pgaio_io_reclaim()
resets the fields in PgAioHandle *before* updating the generation/state. That
opens up a window in which pg_get_aios() thinks the copied PgAioHandle is
valid, even though it was taken while the fields were being reset.

Once I had figured that out, it was easy to make it more reproducible - put a
pg_usleep() between the fields being reset in pgaio_io_reclaim() and the
generation increase / state update.

The fix is simple, increment generation and state before resetting fields.

Will push the fix for that soon.

Greetings,

Andres Freund

  193. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-04-30T20:00:35Z

    Hi,

After a bit more private back and forth with Alexander I have found the issue
- and it's pretty stupid:

pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch
of other things, finds the oldest in-flight IO and waits for it.

		PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
											   &pgaio_my_backend->in_flight_ios);

		switch (ioh->state)
		{
...
			case PGAIO_HS_COMPLETED_IO:
			case PGAIO_HS_SUBMITTED:
				pgaio_debug_io(DEBUG2, ioh,
							   "waiting for free io with %d in flight",
							   dclist_count(&pgaio_my_backend->in_flight_ios));
...
				pgaio_io_wait(ioh, ioh->generation);
				break;


The problem is that, if the log level is low enough, ereport() (which is
called by pgaio_debug_io()), processes interrupts.  The interrupt processing
may end up execute ProcessBarrierSmgrRelease(), which in turn needs to wait
for all in-flight IOs before the IOs are closed.

Which then leads to the
			elog(PANIC, "waiting for own IO in wrong state: %d",
				 state);

error.

The waiting for in-flight IOs before closing FDs only happens with io-uring,
hence this only triggering with io-uring.


A similar set of steps can lead to the "no free IOs despite no in-flight IOs"
ERROR that Alexander also observed - if pgaio_submit_staged() triggers a debug
ereport that executes ProcessBarrierSmgrRelease() in an interrupt, we might
wait for all in-flight IOs during IO submission, triggering the error.


I'm somewhat amazed that Alexander could somewhat reliably reproduce this - I
haven't been able to do so once using Alexander's recipe.  I did find a
reproducer though:

c=32;pgbench -n -c$c -j$c -P1 -T1000 -f <(echo 'SELECT sum(abalance) FROM pgbench_accounts;')
c=1;pgbench -n -c$c -j$c -P1 -T1000 -f <(echo 'DROP DATABASE IF EXISTS foo;CREATE DATABASE foo;')

trigger both issues quickly if run with
  log_min_messages=debug2
  io_method=io_uring
and not when a non-debug log level is used.


I'm not yet sure how to best fix it - locally I have done so by pgaio_debug()
do a HOLD_INTERRUPTS()/RESUME_INTERRUPTS() around the call to ereport.  But
that doesn't really seem great - otoh requiring various pieces of code to know
that anything emitting debug messages needs to hold interrupts etc makes for
rare and hard to understand bugs.

We could just make the relevant functions hold interrupts, and that might be
the best path forward, but we don't really need to hold all interrupts
(e.g. termination would be fine), so it's a bit coarse grained.  It would need
to happen in a few places, which isn't great either.

Other suggestions?


Thanks again for finding and reporting this Alexander!

Greetings,

Andres Freund

  194. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-05-03T03:05:11Z

    On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
> pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch
> of other things, finds the oldest in-flight IO and waits for it.
> 
> 		PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
> 											   &pgaio_my_backend->in_flight_ios);
> 
> 		switch (ioh->state)
> 		{
> ...
> 			case PGAIO_HS_COMPLETED_IO:
> 			case PGAIO_HS_SUBMITTED:
> 				pgaio_debug_io(DEBUG2, ioh,
> 							   "waiting for free io with %d in flight",
> 							   dclist_count(&pgaio_my_backend->in_flight_ios));
> ...
> 				pgaio_io_wait(ioh, ioh->generation);
> 				break;
> 
> 
> The problem is that, if the log level is low enough, ereport() (which is
> called by pgaio_debug_io()), processes interrupts.  The interrupt processing
> may end up execute ProcessBarrierSmgrRelease(), which in turn needs to wait
> for all in-flight IOs before the IOs are closed.
> 
> Which then leads to the
> 			elog(PANIC, "waiting for own IO in wrong state: %d",
> 				 state);
> 
> error.

Printing state 0 (PGAIO_HS_IDLE), right?  I think the chief problem is that
pgaio_io_wait_for_free() is fetching ioh->state, then possibly processing
interrupts in pgaio_debug_io(), then finally fetching ioh->generation.  If it
fetched ioh->generation to a local variable before pgaio_debug_io, I think
that would resolve this one.  Then the pgaio_io_was_recycled() would prevent
the PANIC:

	if (pgaio_io_was_recycled(ioh, ref_generation, &state))
		return;

	if (am_owner)
	{
		if (state != PGAIO_HS_SUBMITTED
			&& state != PGAIO_HS_COMPLETED_IO
			&& state != PGAIO_HS_COMPLETED_SHARED
			&& state != PGAIO_HS_COMPLETED_LOCAL)
		{
			elog(PANIC, "waiting for own IO in wrong state: %d",
				 state);
		}
	}

Is that right?  If that's the solution, pgaio_closing_fd() and
pgaio_shutdown() would need similar care around fetching the generation before
the pgaio_debug_io.  Maybe there's an opportunity for a common inline
function.  Or at least a comment at the "generation" field on how to safely
time a fetch thereof and any barrier required.

> A similar set of steps can lead to the "no free IOs despite no in-flight IOs"
> ERROR that Alexander also observed - if pgaio_submit_staged() triggers a debug
> ereport that executes ProcessBarrierSmgrRelease() in an interrupt, we might
> wait for all in-flight IOs during IO submission, triggering the error.

That makes sense.

> I'm not yet sure how to best fix it - locally I have done so by pgaio_debug()
> do a HOLD_INTERRUPTS()/RESUME_INTERRUPTS() around the call to ereport.  But
> that doesn't really seem great - otoh requiring various pieces of code to know
> that anything emitting debug messages needs to hold interrupts etc makes for
> rare and hard to understand bugs.
> 
> We could just make the relevant functions hold interrupts, and that might be
> the best path forward, but we don't really need to hold all interrupts
> (e.g. termination would be fine), so it's a bit coarse grained.  It would need
> to happen in a few places, which isn't great either.
> 
> Other suggestions?

For the "no free IOs despite no in-flight IOs" case, I'd replace the
ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
an IO.  Then decide what protection if any, we need against bugs causing an
infinite loop in caller pgaio_io_acquire().  What's the case motivating the
unbounded loop in pgaio_io_acquire(), as opposed to capping at two
pgaio_io_acquire_nb() calls?  If the theory is that pgaio_io_acquire() could
be reentrant, what scenario would reach that reentrancy?

> Thanks again for finding and reporting this Alexander!

+1!

  195. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-05-09T01:06:18Z

    Hi,

On 2025-05-02 20:05:11 -0700, Noah Misch wrote:
> On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
> > pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch
> > of other things, finds the oldest in-flight IO and waits for it.
> > 
> > 		PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
> > 											   &pgaio_my_backend->in_flight_ios);
> > 
> > 		switch (ioh->state)
> > 		{
> > ...
> > 			case PGAIO_HS_COMPLETED_IO:
> > 			case PGAIO_HS_SUBMITTED:
> > 				pgaio_debug_io(DEBUG2, ioh,
> > 							   "waiting for free io with %d in flight",
> > 							   dclist_count(&pgaio_my_backend->in_flight_ios));
> > ...
> > 				pgaio_io_wait(ioh, ioh->generation);
> > 				break;
> > 
> > 
> > The problem is that, if the log level is low enough, ereport() (which is
> > called by pgaio_debug_io()), processes interrupts.  The interrupt processing
> > may end up execute ProcessBarrierSmgrRelease(), which in turn needs to wait
> > for all in-flight IOs before the IOs are closed.
> > 
> > Which then leads to the
> > 			elog(PANIC, "waiting for own IO in wrong state: %d",
> > 				 state);
> > 
> > error.
> 
> Printing state 0 (PGAIO_HS_IDLE), right?

Correct.


>I think the chief problem is that pgaio_io_wait_for_free() is fetching
>ioh->state, then possibly processing interrupts in pgaio_debug_io(), then
>finally fetching ioh->generation.  If it fetched ioh->generation to a local
>variable before pgaio_debug_io, I think that would resolve this one.

That's what I also concluded after playing around with a few different
approaches.


> Then the pgaio_io_was_recycled() would prevent the PANIC:
> 
> 	if (pgaio_io_was_recycled(ioh, ref_generation, &state))
> 		return;
> 
> 	if (am_owner)
> 	{
> 		if (state != PGAIO_HS_SUBMITTED
> 			&& state != PGAIO_HS_COMPLETED_IO
> 			&& state != PGAIO_HS_COMPLETED_SHARED
> 			&& state != PGAIO_HS_COMPLETED_LOCAL)
> 		{
> 			elog(PANIC, "waiting for own IO in wrong state: %d",
> 				 state);
> 		}
> 	}
> 
> Is that right?

Yes, it avoids the issue.


> If that's the solution, pgaio_closing_fd() and pgaio_shutdown() would need
> similar care around fetching the generation before the pgaio_debug_io.
> Maybe there's an opportunity for a common inline function.  Or at least a
> comment at the "generation" field on how to safely time a fetch thereof and
> any barrier required.

I didn't see a good way to move this into an inline function, unfortunately.


We do need to hold interrupts in a few other places, I think - with some debug
infrastructure (things like calling ProcessBarrierSmgrRelease() whenever
interrupts could be processed and calling CFI() in errstart() in its return
false case) it's possible to find state confusions which trigger
assertions. The issue is that pgaio_io_update_state() contains a
pgaio_debug_io() and executing pgaio_closing_fd() in places that call
pgaio_io_update_state() doesn't end well.  There's a similar danger with the
debug message in pgaio_io_reclaim().

In the attached patch I added an assertion to pgaio_io_update_state()
verifying that interrupts are held and added code to hold interrupts in the
relevant places.

> > I'm not yet sure how to best fix it - locally I have done so by pgaio_debug()
> > do a HOLD_INTERRUPTS()/RESUME_INTERRUPTS() around the call to ereport.  But
> > that doesn't really seem great - otoh requiring various pieces of code to know
> > that anything emitting debug messages needs to hold interrupts etc makes for
> > rare and hard to understand bugs.
> > 
> > We could just make the relevant functions hold interrupts, and that might be
> > the best path forward, but we don't really need to hold all interrupts
> > (e.g. termination would be fine), so it's a bit coarse grained.  It would need
> > to happen in a few places, which isn't great either.
> > 
> > Other suggestions?
> 
> For the "no free IOs despite no in-flight IOs" case, I'd replace the
> ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
> an IO.

Hm - it seems better to me to check if there are now free handles and return
if that's the case, but to keep the error check in case there actually is no
free IO? That seems like a not implausible bug...


> Then decide what protection if any, we need against bugs causing an
> infinite loop in caller pgaio_io_acquire().  What's the case motivating the
> unbounded loop in pgaio_io_acquire(), as opposed to capping at two
> pgaio_io_acquire_nb() calls?  If the theory is that pgaio_io_acquire() could
> be reentrant, what scenario would reach that reentrancy?

I do not remember why I wrote this as an endless loop.  If you prefer I could
change that as part of this patch.


It does seem rather dangerous that errstart() processes interrupts for debug
messages, but only if the debug message is actually logged. That's really a
recipe for hard to find bugs.  I wonder if we should, at least in assertion
mode, process interrupts even if not emitting the message.

Greetings,

Andres Freund

  196. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-05-09T01:15:04Z

    Hi,

On 2025-04-23 00:38:01 +0200, Tomas Vondra wrote:
> There's an open item related to this commit (247ce06b883d), based on:

Oops, forgot to reply.


>     For now the default io_method is changed to "worker". We should re-
>     evaluate that around beta1, we might want to be careful and set the
>     default to "sync" for 18.
> 
> FWIW the open item is in the "recheck mid-beta" section, but the commit
> message says "around beta1" which is not that far away (especially if we
> choose to do beta1 on May 8, with the minor releases).

I think we certainly should wait for at least beta1 (not that we have a choice
now), to make sure all of this gets at least some coverage.


> What information we need to gather to make the decision and who's
> expected to do it? I assume someone needs to do run some benchmarks, but
> is anyone working on it already? What benchmarks, what platforms?

I guess I'd say "the community". I think it should depend on how many problems
we find, be it correctness or performance regressions. I don't know how to
make that call without seeing how things are going during beta.

And yes, I think this should involve some benchmarking too - generally a good
thing to do during beta, and if it involves AIO, all the better.


> FWIW I'm asking because of the RMT, but I'm also willing to do some of
> the tests, if needed - but it'd be good to get some guidance.

More testing would be welcome!

Greetings,

Andres Freund

  197. Re: AIO v2.5

    Noah Misch <noah@leadboat.com> — 2025-05-09T02:22:27Z

    On Thu, May 08, 2025 at 09:06:18PM -0400, Andres Freund wrote:
> On 2025-05-02 20:05:11 -0700, Noah Misch wrote:
> > On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:

> We do need to hold interrupts in a few other places, I think - with some debug
> infrastructure (things like calling ProcessBarrierSmgrRelease() whenever
> interrupts could be processed and calling CFI() in errstart() in its return
> false case) it's possible to find state confusions which trigger
> assertions. The issue is that pgaio_io_update_state() contains a
> pgaio_debug_io() and executing pgaio_closing_fd() in places that call
> pgaio_io_update_state() doesn't end well.  There's a similar danger with the
> debug message in pgaio_io_reclaim().
> 
> In the attached patch I added an assertion to pgaio_io_update_state()
> verifying that interrupts are held and added code to hold interrupts in the
> relevant places.

Works for me.

> > For the "no free IOs despite no in-flight IOs" case, I'd replace the
> > ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
> > an IO.
> 
> Hm - it seems better to me to check if there are now free handles and return
> if that's the case, but to keep the error check in case there actually is no
> free IO? That seems like a not implausible bug...

Works for me.

> > Then decide what protection if any, we need against bugs causing an
> > infinite loop in caller pgaio_io_acquire().  What's the case motivating the
> > unbounded loop in pgaio_io_acquire(), as opposed to capping at two
> > pgaio_io_acquire_nb() calls?  If the theory is that pgaio_io_acquire() could
> > be reentrant, what scenario would reach that reentrancy?
> 
> I do not remember why I wrote this as an endless loop.  If you prefer I could
> change that as part of this patch.

I asked because it would be sad to remove the ereport(ERROR) like I proposed
and then have a bug cause a real infinite loop.  Removing the loop was one way
to prove that can't happen.  As you say, another way would be keeping the
ereport(ERROR) and guarding it with a free-handles check, like in your patch
today.  I don't have a strong preference between those.

> It does seem rather dangerous that errstart() processes interrupts for debug
> messages, but only if the debug message is actually logged. That's really a
> recipe for hard to find bugs.  I wonder if we should, at least in assertion
> mode, process interrupts even if not emitting the message.

Yes, that sounds excellent to have.

  198. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-05-20T01:09:55Z

    Hi,

On 2025-05-08 19:22:27 -0700, Noah Misch wrote:
> On Thu, May 08, 2025 at 09:06:18PM -0400, Andres Freund wrote:
> > On 2025-05-02 20:05:11 -0700, Noah Misch wrote:
> > > On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
> 
> > We do need to hold interrupts in a few other places, I think - with some debug
> > infrastructure (things like calling ProcessBarrierSmgrRelease() whenever
> > interrupts could be processed and calling CFI() in errstart() in its return
> > false case) it's possible to find state confusions which trigger
> > assertions. The issue is that pgaio_io_update_state() contains a
> > pgaio_debug_io() and executing pgaio_closing_fd() in places that call
> > pgaio_io_update_state() doesn't end well.  There's a similar danger with the
> > debug message in pgaio_io_reclaim().
> > 
> > In the attached patch I added an assertion to pgaio_io_update_state()
> > verifying that interrupts are held and added code to hold interrupts in the
> > relevant places.
> 
> Works for me.
> 
> > > For the "no free IOs despite no in-flight IOs" case, I'd replace the
> > > ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
> > > an IO.
> > 
> > Hm - it seems better to me to check if there are now free handles and return
> > if that's the case, but to keep the error check in case there actually is no
> > free IO? That seems like a not implausible bug...
> 
> Works for me.
> 
> > > Then decide what protection if any, we need against bugs causing an
> > > infinite loop in caller pgaio_io_acquire().  What's the case motivating the
> > > unbounded loop in pgaio_io_acquire(), as opposed to capping at two
> > > pgaio_io_acquire_nb() calls?  If the theory is that pgaio_io_acquire() could
> > > be reentrant, what scenario would reach that reentrancy?
> > 
> > I do not remember why I wrote this as an endless loop.  If you prefer I could
> > change that as part of this patch.
> 
> I asked because it would be sad to remove the ereport(ERROR) like I proposed
> and then have a bug cause a real infinite loop.  Removing the loop was one way
> to prove that can't happen.  As you say, another way would be keeping the
> ereport(ERROR) and guarding it with a free-handles check, like in your patch
> today.  I don't have a strong preference between those.

I pushed it this way just now.

Thanks again for Alexander for reporting & investigating the issues and for
Noah's review.

Greetings,

Andres Freund

  199. Re: AIO v2.5

    Antonin Houska <ah@cybertec.at> — 2025-06-30T06:58:21Z

    Andres Freund <andres@anarazel.de> wrote:

> On 2025-03-13 11:53:03 +0100, Antonin Houska wrote:
> > Attached are a few proposals for minor comment fixes.
> 
> Thanks, applied.

After reading the code a bit more, I noticed that the 'cb_flags' argument of
PgAioHandleCallbackStage is not really used, at least in the existing
callbacks. Is there an intention to use it in the future (the patches for
async write do not seem to indicate so), or is this only a leftover from
previous versions of the patch?

Besides that, I suggest a minor comment fix.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

  200. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-06-30T14:44:25Z

    Hi,

On 2025-06-30 08:58:21 +0200, Antonin Houska wrote:
> Andres Freund <andres@anarazel.de> wrote:
> 
> > On 2025-03-13 11:53:03 +0100, Antonin Houska wrote:
> > > Attached are a few proposals for minor comment fixes.
> > 
> > Thanks, applied.
> 
> After reading the code a bit more, I noticed that the 'cb_flags' argument of
> PgAioHandleCallbackStage is not really used, at least in the existing
> callbacks. Is there an intention to use it in the future (the patches for
> async write do not seem to indicate so), or is this only a leftover from
> previous versions of the patch?

Thanks for reading through it!

It seemed more symmetric to have it for stage as well as the completion
callback - and I still think that...


> Besides that, I suggest a minor comment fix.

Thanks. Pushed that fix.

Greetings,

Andres Freund

  201. Re: AIO v2.5

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2025-07-09T11:26:09Z

    Hi,

I've been going through the new AIO code as an effort to rebase and
adapt Neon to PG18. In doing so, I found the following
items/curiosities:

1. In aio/README.md, the following code snippet is found:

[...]
pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
[...]

I believe it would be clearer if it took a reference to the buffer:

pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);

The main reason here is that common practice is to have a `Buffer
buffer;` whereas a Buffer * is more commonly plural.


2. In aio.h, PgAioHandleCallbackID is checked to fit in
PGAIO_RESULT_ID_BITS (though the value of PGAIO_HCB_MAX).
However, the check is off by 1:

```
#define PGAIO_HCB_MAX    PGAIO_HCB_LOCAL_BUFFER_READV
StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS), [...])
```

This static assert will not trigger when PGAIO_HCB_MAX is equal to
2^PGAIO_RESULT_ID_BITS, but its value won't fit in those 6 bits and
instead overflow to 0.
To fix this, I suggest the `<=` is replaced with `<` to make that
work, or the definition of PGAIO_HCB_MAX to be updated to
PGAIO_HCB_LOCAL_BUFFER_READV + 1.

Note that this won't have caused bugs yet because we're not nearing
this limit of 64 callback IDs, but it's just a matter of good code
hygiene.


3. I noticed that there is AIO code for writev-related operations
(specifically, pgaio_io_start_writev is exposed, as is
PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
not called from anywhere in the project, and there is no way for
extensions to register the relevant callbacks required to make writev
work well on buffered contents. Is that intentional?

Relatedly, the rationale for using enum PgAioHandleCallbackID rather
than function pointers in the documentation above its definition says
that function pointer instability across backends is one of the 2 main
reasons.
Is there any example of OS or linker behaviour that does not start
PostgreSQL with stable function pointer addresses across backends of
the same PostgreSQL binary? Or is this designed with extensibility
and/or cross-version EXEC_BACKEND in mind, but with extensibility not
yet been implemented due to $constraints?


I've attached a patch that solves the issues of (1) and (2).


Kind regards,

Matthias van de Meent
Databricks

  202. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-07-09T14:59:41Z

    Hi,

On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote:
> I've been going through the new AIO code as an effort to rebase and
> adapt Neon to PG18. In doing so, I found the following
> items/curiosities:
> 
> 1. In aio/README.md, the following code snippet is found:
> 
> [...]
> pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
> [...]
> 
> I believe it would be clearer if it took a reference to the buffer:
> 
> pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
> 
> The main reason here is that common practice is to have a `Buffer
> buffer;` whereas a Buffer * is more commonly plural.

It's also just simply wrong as-is :/. Interpreting the buffer id as a pointer
obviously makes no sense...


> 
> 2. In aio.h, PgAioHandleCallbackID is checked to fit in
> PGAIO_RESULT_ID_BITS (though the value of PGAIO_HCB_MAX).
> However, the check is off by 1:
> 
> ```
> #define PGAIO_HCB_MAX    PGAIO_HCB_LOCAL_BUFFER_READV
> StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS), [...])
> ```
> 
> This static assert will not trigger when PGAIO_HCB_MAX is equal to
> 2^PGAIO_RESULT_ID_BITS, but its value won't fit in those 6 bits and
> instead overflow to 0.
> To fix this, I suggest the `<=` is replaced with `<` to make that
> work, or the definition of PGAIO_HCB_MAX to be updated to
> PGAIO_HCB_LOCAL_BUFFER_READV + 1.

Ooops.



> 3. I noticed that there is AIO code for writev-related operations
> (specifically, pgaio_io_start_writev is exposed, as is
> PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
> not called from anywhere in the project, and there is no way for
> extensions to register the relevant callbacks required to make writev
> work well on buffered contents. Is that intentional?

Yes.  We obviously do want to support writes eventually, and it didn't seem
useful to not have the most basic code for writes in the AIO infrastructure.

You could still use it to e.g. write out temporary file data or such.


> and there is no way for extensions to register the relevant callbacks
> required to make writev work well on buffered contents. Is that intentional?

FWIW, the problem with writev for buffered IO is not so much with the AIO
infrastructure, but with buffer locking and sync.c...




> Relatedly, the rationale for using enum PgAioHandleCallbackID rather
> than function pointers in the documentation above its definition says
> that function pointer instability across backends is one of the 2 main
> reasons.
> Is there any example of OS or linker behaviour that does not start
> PostgreSQL with stable function pointer addresses across backends of
> the same PostgreSQL binary? Or is this designed with extensibility
> and/or cross-version EXEC_BACKEND in mind, but with extensibility not
> yet been implemented due to $constraints?

It's due to EXEC_BACKEND - function pointers aren't stable with EXEC_BACKEND
even *without* cross-version issues. Due to ASLR function/data pointers may be
shifted around.

Even without EXEC_BACKEND we probably would want to have something like
PgAioHandleCallbackID, just to save space. But it'd be a lot easier to make it
extensible, because we could add entries once, rather than having to do so in
every process.


I suspect to make this extensible in the face of EXEC_BACKEND we'd want an
array of dynamically registered callbacks, in shared memory, that has the
library_name/symbol_name for the location of each callback struct, to be
registered in _PG_init(), which then would be entered into the backend-local
table in pgaio_init_backend().

Personally I'm not particularly interested in investing the energy to make
that work, I'd rather wait till we have threading and focus on the
infrastructure work for AIO writes etc. But then I don't work on an extension
these days, so I'm not directly affected by this not being extensible - I'm
not opposed to somebody else doing that work.

Greetings,

Andres Freund

  203. Re: AIO v2.5

    Matthias van de Meent <boekewurm+postgres@gmail.com> — 2025-07-10T19:00:21Z

    On Wed, 9 Jul 2025 at 16:59, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote:
> > I've been going through the new AIO code as an effort to rebase and
> > adapt Neon to PG18. In doing so, I found the following
> > items/curiosities:
> >
> > 1. In aio/README.md, the following code snippet is found:
> >
> > [...]
> > pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
> > [...]
> >
> > I believe it would be clearer if it took a reference to the buffer:
> >
> > pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
> >
> > The main reason here is that common practice is to have a `Buffer
> > buffer;` whereas a Buffer * is more commonly plural.
>
> It's also just simply wrong as-is :/. Interpreting the buffer id as a pointer
> obviously makes no sense...

Given that the snippet didn't contain type indications for buffer upto
that point, technically the buffer variable could've been defined as
`Buffer* buffer;` which would've been type-correct. That would be very
confusing however, hence the suggested change.

After your mail, I also noticed the later snippet which should be updated, too:

```
-smgrstartreadv(ioh, operation->smgr, forknum, blkno,
-               BufferGetBlock(buffer), 1);
+void *page = BufferGetBlock(buffer);
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+               &page, 1);
```

> > 3. I noticed that there is AIO code for writev-related operations
> > (specifically, pgaio_io_start_writev is exposed, as is
> > PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
> > not called from anywhere in the project, and there is no way for
> > extensions to register the relevant callbacks required to make writev
> > work well on buffered contents. Is that intentional?
>
> Yes.  We obviously do want to support writes eventually, and it didn't seem
> useful to not have the most basic code for writes in the AIO infrastructure.
>
> You could still use it to e.g. write out temporary file data or such.

Yes, though IIUC that would require an implementation of at least
PgAioTargetInfo for such a use case (it's definitely not a SMGR
target), which currently isn't available and can't be registered
dynamically by an extension. Or maybe did I miss something?

> > and there is no way for extensions to register the relevant callbacks
> > required to make writev work well on buffered contents. Is that intentional?
>
> FWIW, the problem with writev for buffered IO is not so much with the AIO
> infrastructure, but with buffer locking and sync.c...

Maybe "buffered contents" wasn't the right phrasing, but my main point
remains - I can't seem to find a way to make use of the
pgaio_start_writev API in the 18 branch that doesn't require patching
core code or heavily relying on the behaviour of various pgaio
internals:

The only valid pgaio target is PGAIO_TID_SMGR, which effectively
forces you to use buffered relations as IO target (either shared, or
backend-local/temp). Then, there are no callbacks defined for writev,
so the user will effectively have to do setup and cleanup on their
own, making worker-based IO practically impossible, and requiring the
user to handle all lock acquisition/releases in their own process'
time instead of in the AIO paths - requiring significantly more
involved error handling.


(PS. I'm not quite 100% sure that it is impossible to use, just that
there are rather few handles available for using this part of the new
tool, and it seems completely untested in the PG18 branch)

-----

Something else I've just noticed is the use of int32 in
PgAIOHandle->result. In sync and worker mode, pg_preadv and pg_pwritev
return ssize_t, which most modern systems can't fit in int32 (the
output was int before, then size_t, then ssize_t: [0]). While not
directly an issue in default PG18 due to the use of 1GB relation
segments capping the max IO size for SMGR-managed IOs (and various
other code-level constraints), this may have more issues when an
extension starts bulk-reading data on a system compiled with
RELSEG_SIZE >= 2GB; I can't find any protective checks against
overflows in downcasting the IO result.

I did notice Linux seems to guarantee it won't read more than
0x7FFF_F000 in any one operation, but I can't find any similar
guarantees for e.g. MacOS or Windows (well, our win32 port limits
pg_pread/writev to 1GB, which seems sufficient for this exact case).

If we're keeping int32, then it would be great if the considerations
for why it's OK to use 32 bits to store the ssize_t vlaue would be
included in the docs/comments.


Kind regards,

Matthias van de Meent
Databricks

[0] https://postgr.es/m/flat/1672202.1703441340%40sss.pgh.pa.us

  204. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-07-10T19:29:33Z

    Hi,

On 2025-07-10 21:00:21 +0200, Matthias van de Meent wrote:
> On Wed, 9 Jul 2025 at 16:59, Andres Freund <andres@anarazel.de> wrote:
> > > 3. I noticed that there is AIO code for writev-related operations
> > > (specifically, pgaio_io_start_writev is exposed, as is
> > > PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
> > > not called from anywhere in the project, and there is no way for
> > > extensions to register the relevant callbacks required to make writev
> > > work well on buffered contents. Is that intentional?
> >
> > Yes.  We obviously do want to support writes eventually, and it didn't seem
> > useful to not have the most basic code for writes in the AIO infrastructure.
> >
> > You could still use it to e.g. write out temporary file data or such.
>
> Yes, though IIUC that would require an implementation of at least
> PgAioTargetInfo for such a use case (it's definitely not a SMGR
> target), which currently isn't available and can't be registered
> dynamically by an extension. Or maybe did I miss something?

I can see some hacky ways around that, but they're just that, hacky...



> (PS. I'm not quite 100% sure that it is impossible to use, just that
> there are rather few handles available for using this part of the new
> tool, and it seems completely untested in the PG18 branch)

I'm not saying it's 100% ready to use without modifying core code, but for
something that's like 30 lines of code, as part of a considerably larger
subystem, I just don't see a problem with writev not yet being covered.  It's
just incremental development.


> -----
>
> Something else I've just noticed is the use of int32 in
> PgAIOHandle->result. In sync and worker mode, pg_preadv and pg_pwritev
> return ssize_t, which most modern systems can't fit in int32 (the
> output was int before, then size_t, then ssize_t: [0]).

I don't think there's anything that can actually do IO that's large enough to
be problematic. What's the potential scenario where you'd want to read/write
more than 3GB of data within one syscall? That just doesn't seem to make
sense.


> While not directly an issue in default PG18 due to the use of 1GB relation
> segments capping the max IO size for SMGR-managed IOs (and various other
> code-level constraints), this may have more issues when an extension starts
> bulk-reading data on a system compiled with RELSEG_SIZE >= 2GB; I can't find
> any protective checks against overflows in downcasting the IO result.

I don't think the relation size is relevant piece here, it's just that it
doesn't make sense (and likely isn't possible) to read that much data at once.


Greetings,

Andres Freund

  205. Re: AIO v2.5

    Tomas Vondra <tomas@vondra.me> — 2025-07-11T21:03:53Z

    Hi,

I've been running some benchmarks comparing the io_methods, to help with
resolving this PG18 open item. So here are some results, and my brief
analysis of it.

I was hoping to get this out sooner before beta2 :-( and some tests are
still running, but I don't think it'll affect the conclusions.


The TL;DR version
-----------------

* The "worker" method seems good, and I think we should keep it as a
default. We should probably think about increasing the number of workers
a bit, the current io_workers=3 seems to be too low and regresses in a
couple tests.

* The "sync" seems OK too, but it's more of a conservative choice, i.e.
more weight for keeping the PG17 behavior / not causing regressions. But
I haven't seen that (with enough workers). And there are cases when the
"worker" is much faster. It'd be a shame to throw away that benefit.

* There might be bugs in "worker", simply because it has to deal with
multiple concurrent processes etc. But I guess we'll fix those just like
other bugs. I don't think it's a good argument against "worker" default.

* All my tests were done on Linux and NVMe drives. It'd be good to do
similar testing on other platforms (e.g. FreeBSD) and/or storage. I plan
to do some of that, but it'd be great to cover more cases. I can help
with getting my script running, a run takes ~1-2 days.


A more detailed version ...
---------------------------

The benchmark I did works like this:

1) It generates datasets with different data distributions (patterns in
the data - uniform, linear, cyclic, ...). Each table is ~4.3GB of data.

2) It then runs queries on that, with a BETWEEN clause with a certain
selectivity (matching % of the rows), forcing a particular scan type
(indexscan, bitmapscan, seqscan).

3) For each query it measures duration of "cold" and "warm" runs. Cold
means "nothing in page cache / shared  buffers", while "warm" means
everything is cached somewhere (by the first "cold" run).

There's a couple relevant parameters varied between the runs:

* effective_io_concurrency = [0, 1, 16, 32]
* shared_buffers = [4GB, 32GB]

Parallel query was disabled for these tests.

The test also included PG17 for comparison, but I forgot PG18 enabled
checksums by default. So PG17 results are with checksums off, which in
some cases means PG17 seems a little bit faster. I'm re-running it with
checksums enabled on PG17, and that seems to eliminate the differences
(as expected).


Scripts
-------

The benchmark scripts/results/charts are available here:

  https://github.com/tvondra/iomethod-tests

The SQL data generator and the script driving the benchmark:

* https://github.com/tvondra/iomethod-tests/blob/master/create2.sql

* https://github.com/tvondra/iomethod-tests/blob/master/test-full-cost-2.sh

The script is fairly simple. It first generates the combinations of
parameters to test, randomizes them, and then tests each of them.
Results are recorded in a CSV file "results.csv".


Test machines
-------------

As usual, I did this on two machines I have at home:

* ryzen (~2024)
  * Ryzen 9 9900X (12 cores)
  * 64GB RAM
  * 4x NVMe SSD (Samsung 990 PRO 1TB) in RAID0

* xeon (~2016)
  * 2x Xeon 2699v4 (44 cores)
  * 64GB RAM
  * 1x NVMe SSD WDC Ultrastar DC SN640 960GB

The ryzen is much beefier in terms of I/O, it can do ~20GB/s in
sequential read. The xeon is much more modest (and generally older).


Charts
------

Most of the repository is PDF charts generated from the CSV results.
There's a README explaining the naming convention of the charts, and
some other details.

But in general there are two "types" of charts. The first one fixes all
parameters except for "dataset", and then shows comparison of results
for all datasets.

For example the attached "ryzen-rows-cold-32GB-16-unscaled.pdf" shows
results for:

 * ryzen machine
 * selectivity calculated as "% of rows" (not pages)
 * cold runs, i.e. data not cached
 * shared_buffers=32GB
 * effective_io_concurrency=16
 * unscaled - each plot has custom y-range

There are "scaled" plots too, with all the plots scaled to the same
y-range. This makes it easier to confirm plots from the same PDF (e.g.
how the different scans perform). But that's irrelevant for picking the
io_method default. and it often hides details in some plots.

The other type fixes "dataset" and varies effective_io_concurrency, so
see how it affects duration (e.g. due to prefetching). For example the
attached ryzen-rows-cold-32GB-uniform-unscaled.pdf shows that for the
"uniform" data set.

The charts show behavior for the whole selectivity range, from 0% to
100%, for each scan type. Of course, the scan type may not be the right
choice for that selectivity point. For example, we'd probably not pick
an index scan for 50% selectivity, or a seqscan for 0.01%. But I think
it makes sense to still consider the whole range, for robustness. We
make estimation/planning mistakes fairly regularly, so it's important to
care about what happens in those cases.


Findings
--------

I'm attaching only three PDFs with charts from the cold runs, to keep
the e-mail small (each PDF is ~100-200kB). Feel free to check the other
PDFs in the git repository, but it's all very similar and the attached
PDFs are quite representative.

Some basic observations:

a) index scans

There's almost no difference for indexscans, i.e. the middle column in
the PDFs. There's a bit of variation on some of the cyclic/linear data
sets, but it seems more like random noise than a systemic difference.

Which is not all that surprising, considering index scans don't really
use read_stream yet, so there's no prefetching etc.

b) bitmapscans (ryzen-bitmapscan-uniform.png)

Bitmapscans are much more affected, with a lot of differences. See the
attached PNG image, for example. That shows that at ~10% you get 2x
faster query with io_method=worker than "sync".

And it also shows that PG18 with io_method=sync is much faster than
PG17. Which is a bit surprising, TBH. At first I thought it's due to the
"checksums=off" on PG17 in this run, but that doesn't seem to be the
case - results from PG17 with checksums=on show the same thing. In fact,
the difference increases, which is rather bizarre. (There's a branch
"run2-17-checksums-on" with results from that run.)

I believe this is likely due to the bitmapscan prefetch fixes, which we
decided to not backpatch (IIRC), because the results with e_i_c=0 show
no difference between PG17 and PG18/sync.

The "ryzen" results however demonstrate that 3 workers may be too low.
The timing spikes to ~3000ms (at ~1% selectivity), before quickly
dropping back to ~1000ms. The other datasets show similar difference.
With 12 workers, there's no such problem.

On "xeon" the differences are much smaller, or not visible at all. My
guess it's due to the hardware differences (slower CPU / single NVMe for
storage).

But I'm also wondering what would happen if there are multiple queries
doing bitmapscans. Consider the benchmark is only a backend running
queries, and parallel query is disabled.

The problems with too few workers are much better visible in the
log-scale charts (just add "-log" at the end). See the attached example
(ryzen-bitmapscan-uniform-log.png). It shows how much slower it's
compared to all other methods (effectively regression compared to PG17)
for a huge chunk of the selectivity. I'm sure we'd pick bitmapscan for
some of those queries.


c) seqscan (ryzen-seqscan-uniform.png)

This shows a lot of significant differences between different io_method
options. And by significant I mean ~2x difference, on both machines.

On "ryzen" the "worker" takes ~800ms, while "sync" is ~1700ms. PG17
seems a bit faster, but that's due to missing checksums, and with
checksums it's almost exactly the same as PG18/sync. io_uring is in
between at ~1200ms.

On "xeon" it's very similar, but io_uring does better and worker/3 a bit
worse (which I think is another reason to increase io_workers default).

I believe these differences can be explained by where the work happens
for different io_method cases. And by work I mean memcpy and checksum
verification. With "sync/io_uring" it's in the backend itself, while
with "worker" it's offloaded to the worker processes. So single-process
vs. multi-process memory bandwidth.

Which is nice, but we need to have enough of them ;-)


d) warm runs

The "warm" runs are very uninteresting. There's literally no difference
between the various io_methods. On "ryzen" it's very smooth, while on
"xeon" the "seqscan" is very noisy. I believe this is due to NUMA
effects (two sockets etc.) but I need to verify that. It affects all
io_methods equally, so it's irrelevant for this.


e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)

There's an interesting difference difference I noticed in the run with
checksums on PG17. The full PDF is available here:

https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled.pdf

The interesting thing is that PG17 indexscans on uniform dataset got a
little bit faster. In the attached PDF it's exactly on par with PG18,
but here it got a bit faster. Which makes no sense, if it has to also
verify checksums. I haven't had time to investigate this yet.


Conclusion
----------

That's all I have at the moment. I still think it makes sense to keep
io_method=worker, but bump up the io_workers a bit higher. Could we also
add some suggestions how to pick a good value to the docs?

You can probably find more interesting results in the other PDFs in the
repository. I'm certainly going to keep looking.

You might also run the benchmark on different hardware, and either
build/publish the plots somewhere, or just give me the CSV and I'll do
that. Better to find strange stuff / regressions now.

The repository also has branches with plots showing results with WIP
indexscan prefetching. (It's excluded from the PDFs I presented here).
The conclusions are similar to what we found here - "worker" is good
with enough workers, io_uring is good too. Sync has issues for some of
the data sets, but still helps a lot.


regards

-- 
Tomas Vondra

  206. Re: AIO v2.5

    Tomas Vondra <tomas@vondra.me> — 2025-07-13T18:04:51Z

    On 7/11/25 23:03, Tomas Vondra wrote:
> ...
>
> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
> 
> There's an interesting difference difference I noticed in the run with
> checksums on PG17. The full PDF is available here:
> 
> https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled.pdf
> 
> The interesting thing is that PG17 indexscans on uniform dataset got a
> little bit faster. In the attached PDF it's exactly on par with PG18,
> but here it got a bit faster. Which makes no sense, if it has to also
> verify checksums. I haven't had time to investigate this yet.

I was intrigued by this, so I looked into this today.

TL;DR I believe it was caused by something in the filesystem or even the
storage devices, making the "PG17" data directory (or maybe even just
the "uniform" table) a bit faster.

I started by reproducing the behavior with an indexscan matching 10% of
the rows, and it was very easy to reproduce the difference shows on the
chart (all timings in milliseconds):

PG17: 14112.800 ms
PG18: 21612.090 ms

This was perfectly reproducible, affecting the whole table (not just one
part of it), etc. At some point I recalled that I might have initialized
the databases in slightly different ways - one by running the SQL, the
other one by pg_dump/pg_restore (likely with multiple jobs).

I couldn't think of any other difference between the data directories,
so I simply reloaded them by pg_restore (from the same dump). Which
however made them both slow :O

And it didn't matter how many jobs are used, or anything else I tried.
But every now and then an instance (17 or 18) happened to be fast
(~14000 ms). Consistently, for all queries on the table, not randomly.

In the end I recreated the (ext4) filesystem, loaded the databases and
now both instances are fast. I have no idea what the root cause was, and
I assume recreating the filesystem destroyed all the evidence.

I'll rerun the tests - will take a couple days. I don't think it's
likely to change the conclusions, though. It should only affect how PG17
compares to PG18, not how the io_methods compare to each other. Also, I
don't think the "xeon" resuls are affected.


regards

-- 
Tomas Vondra

  207. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-07-14T18:36:49Z

    Hi,

On 2025-07-11 23:03:53 +0200, Tomas Vondra wrote:
> I've been running some benchmarks comparing the io_methods, to help with
> resolving this PG18 open item. So here are some results, and my brief
> analysis of it.

Thanks for doing that!



> The TL;DR version
> -----------------
> 
> * The "worker" method seems good, and I think we should keep it as a
> default. We should probably think about increasing the number of workers
> a bit, the current io_workers=3 seems to be too low and regresses in a
> couple tests.
> 
> * The "sync" seems OK too, but it's more of a conservative choice, i.e.
> more weight for keeping the PG17 behavior / not causing regressions. But
> I haven't seen that (with enough workers). And there are cases when the
> "worker" is much faster. It'd be a shame to throw away that benefit.
> 
> * There might be bugs in "worker", simply because it has to deal with
> multiple concurrent processes etc. But I guess we'll fix those just like
> other bugs. I don't think it's a good argument against "worker" default.
> 
> * All my tests were done on Linux and NVMe drives. It'd be good to do
> similar testing on other platforms (e.g. FreeBSD) and/or storage. I plan
> to do some of that, but it'd be great to cover more cases. I can help
> with getting my script running, a run takes ~1-2 days.

FWIW, in my very limited tests on windows, the benefit of worker was
considerably bigger there, due to having much more minimal readahead not
having posix_fadvise...


> The test also included PG17 for comparison, but I forgot PG18 enabled
> checksums by default. So PG17 results are with checksums off, which in
> some cases means PG17 seems a little bit faster. I'm re-running it with
> checksums enabled on PG17, and that seems to eliminate the differences
> (as expected).

My sneaking suspicion is that, independent of AIO, we're not really ready to
default to checksums defaulting to on.


> Findings
> --------
> 
> I'm attaching only three PDFs with charts from the cold runs, to keep
> the e-mail small (each PDF is ~100-200kB). Feel free to check the other
> PDFs in the git repository, but it's all very similar and the attached
> PDFs are quite representative.
> 
> Some basic observations:
> 
> a) index scans
> 
> There's almost no difference for indexscans, i.e. the middle column in
> the PDFs. There's a bit of variation on some of the cyclic/linear data
> sets, but it seems more like random noise than a systemic difference.
>
> Which is not all that surprising, considering index scans don't really
> use read_stream yet, so there's no prefetching etc.

Indeed.


> The "ryzen" results however demonstrate that 3 workers may be too low.
> The timing spikes to ~3000ms (at ~1% selectivity), before quickly
> dropping back to ~1000ms. The other datasets show similar difference.
> With 12 workers, there's no such problem.

I don't really know what to do about that - for now we don't have dynamic
#workers, and starting 12 workers on a tiny database doesn't really make
sense...  I suspect that on most hardware & queries it won't matter that much,
but clearly, if you have high iops hardware it might.  I can perhaps see
increasing the default to 5 or so, but after that...  I guess we could try
some autoconf formula based on the size of s_b or such? But that seems
somewhat awkward too.



> 
> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
> 
> There's an interesting difference difference I noticed in the run with
> checksums on PG17. The full PDF is available here:

(there's a subsequent email about this, will reply there)


> Conclusion
> ----------
> 
> That's all I have at the moment. I still think it makes sense to keep
> io_method=worker, but bump up the io_workers a bit higher. Could we also
> add some suggestions how to pick a good value to the docs?

.oO(/me ponders a troll patch to re-add a reference the number of spindles in
that tuning advice)

I'm not sure what advice to give here.  Maybe just to set it to a considerably
larger number once not running on a tiny system? The incremental overhead of
having an idle worker is rather small unless you're on a really tiny system...


> You might also run the benchmark on different hardware, and either
> build/publish the plots somewhere, or just give me the CSV and I'll do
> that. Better to find strange stuff / regressions now.

Probably the most interesting thing would be some runs with cloud-ish storage
(relatively high iops, very high latency)...


> The repository also has branches with plots showing results with WIP
> indexscan prefetching. (It's excluded from the PDFs I presented here).

Hm, I looked for those, but I couldn't quickly find any plots that include
them.  Would I have to generate the plots from a checkout of the repo?


> The conclusions are similar to what we found here - "worker" is good
> with enough workers, io_uring is good too. Sync has issues for some of
> the data sets, but still helps a lot.

Nice.

Greetings,

Andres Freund

  208. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-07-14T18:44:21Z

    On 2025-07-13 20:04:51 +0200, Tomas Vondra wrote:
> On 7/11/25 23:03, Tomas Vondra wrote:
> > ...
> >
> > e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
> >
> > There's an interesting difference difference I noticed in the run with
> > checksums on PG17. The full PDF is available here:
> >
> > https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled.pdf
> >
> > The interesting thing is that PG17 indexscans on uniform dataset got a
> > little bit faster. In the attached PDF it's exactly on par with PG18,
> > but here it got a bit faster. Which makes no sense, if it has to also
> > verify checksums. I haven't had time to investigate this yet.
>
> I was intrigued by this, so I looked into this today.
>
> TL;DR I believe it was caused by something in the filesystem or even the
> storage devices, making the "PG17" data directory (or maybe even just
> the "uniform" table) a bit faster.
>
> I started by reproducing the behavior with an indexscan matching 10% of
> the rows, and it was very easy to reproduce the difference shows on the
> chart (all timings in milliseconds):
>
> PG17: 14112.800 ms
> PG18: 21612.090 ms

That's a decidedly nontrivial difference.

Did you keep any metrics from those runs? E.g. whether there were larger IOs
or such?



> This was perfectly reproducible, affecting the whole table (not just one
> part of it), etc. At some point I recalled that I might have initialized
> the databases in slightly different ways - one by running the SQL, the
> other one by pg_dump/pg_restore (likely with multiple jobs).

I guess that's an INSERT ...  SELECT vs COPY?

Which one was the faster one?

If this ever re-occurs, it might be interesting to look at the fragmentation
of the underlying files with filefrag.


> I couldn't think of any other difference between the data directories,
> so I simply reloaded them by pg_restore (from the same dump). Which
> however made them both slow :O

So that suggests that COPY is the slow case, interesting.

One potentially relevant factor could be that parallel COPY into logged tables
currently leads to really sub-optimal write patterns, due to us writing back
buffers one-by-one, interspersed by WAL writes and file extensions. I know how
that affects write speed, but it's not entirely obvious how it would affect
read speed...


> And it didn't matter how many jobs are used, or anything else I tried.
> But every now and then an instance (17 or 18) happened to be fast
> (~14000 ms). Consistently, for all queries on the table, not randomly.
>
> In the end I recreated the (ext4) filesystem, loaded the databases and
> now both instances are fast. I have no idea what the root cause was, and
> I assume recreating the filesystem destroyed all the evidence.

Besides differences in filesystem level fragmentation, another potential
theory is that the SSDs were internally more fragmented. Occasionally
dumping/restoring the data could allow the drive to do internal wear
leveling before the new data is loaded, leading to a better layout.

I found that I get more consistent benchmark performance if I delete as much
of the data as possible, run fstrim -v -a and then load the data. And do
another round of fstrim.

Greetings,

Andres Freund

  209. Re: AIO v2.5

    Tomas Vondra <tomas@vondra.me> — 2025-07-15T00:13:03Z

    On 7/14/25 20:44, Andres Freund wrote:
> On 2025-07-13 20:04:51 +0200, Tomas Vondra wrote:
>> On 7/11/25 23:03, Tomas Vondra wrote:
>>> ...
>>>
>>> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
>>>
>>> There's an interesting difference difference I noticed in the run with
>>> checksums on PG17. The full PDF is available here:
>>>
>>> https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled.pdf
>>>
>>> The interesting thing is that PG17 indexscans on uniform dataset got a
>>> little bit faster. In the attached PDF it's exactly on par with PG18,
>>> but here it got a bit faster. Which makes no sense, if it has to also
>>> verify checksums. I haven't had time to investigate this yet.
>>
>> I was intrigued by this, so I looked into this today.
>>
>> TL;DR I believe it was caused by something in the filesystem or even the
>> storage devices, making the "PG17" data directory (or maybe even just
>> the "uniform" table) a bit faster.
>>
>> I started by reproducing the behavior with an indexscan matching 10% of
>> the rows, and it was very easy to reproduce the difference shows on the
>> chart (all timings in milliseconds):
>>
>> PG17: 14112.800 ms
>> PG18: 21612.090 ms
> 
> That's a decidedly nontrivial difference.
> 

It is. It surprised me.

> Did you keep any metrics from those runs? E.g. whether there were larger IOs
> or such?
> 

Unfortunately no. I should have investigated more before rebuilding the
filesystem. But I suspect I might be able to reproduce it, if I do the
loads in a loop or something like that.

> 
>> This was perfectly reproducible, affecting the whole table (not just one
>> part of it), etc. At some point I recalled that I might have initialized
>> the databases in slightly different ways - one by running the SQL, the
>> other one by pg_dump/pg_restore (likely with multiple jobs).
> 
> I guess that's an INSERT ...  SELECT vs COPY?
> 
> Which one was the faster one?
> 

I suspect I initialized one instance by the SQL script that generates
data by INSERT ... SELECT for tables one by one (and then builds
indexes, also one by one). And then I might have initialized the other
database by pg_dump/pg_restore, so that'd be COPY.

FWIW I might have used pg_restore with parallelism, and I suspect that
might be more important than INSERT vs. COPY.

But maybe I'm misremembering things, I rebuilt the benchmark databases
so many times over the past couple days ... not sure.


> If this ever re-occurs, it might be interesting to look at the fragmentation
> of the underlying files with filefrag.
> 

Yeah, I'll keep that in mind. And I'll try to reproduce this, once I'm
done with those benchmarks.

> 
>> I couldn't think of any other difference between the data directories,
>> so I simply reloaded them by pg_restore (from the same dump). Which
>> however made them both slow :O
> 
> So that suggests that COPY is the slow case, interesting.
> 
> One potentially relevant factor could be that parallel COPY into logged tables
> currently leads to really sub-optimal write patterns, due to us writing back
> buffers one-by-one, interspersed by WAL writes and file extensions. I know how
> that affects write speed, but it's not entirely obvious how it would affect
> read speed...
> 

Maybe. I don't remember which database I loaded first. But this time I
simply copied a backup of the data directory from a different device,
which would "cleanup" the write pattern.

I'd assume filesystems allocate space in larger chunks, e.g. ext4 does
delayed allocation (and I don't see why that wouldn't work here).

> 
>> And it didn't matter how many jobs are used, or anything else I tried.
>> But every now and then an instance (17 or 18) happened to be fast
>> (~14000 ms). Consistently, for all queries on the table, not randomly.
>>
>> In the end I recreated the (ext4) filesystem, loaded the databases and
>> now both instances are fast. I have no idea what the root cause was, and
>> I assume recreating the filesystem destroyed all the evidence.
> 
> Besides differences in filesystem level fragmentation, another potential
> theory is that the SSDs were internally more fragmented. Occasionally
> dumping/restoring the data could allow the drive to do internal wear
> leveling before the new data is loaded, leading to a better layout.
> 
> I found that I get more consistent benchmark performance if I delete as much
> of the data as possible, run fstrim -v -a and then load the data. And do
> another round of fstrim.
> 

Maybe, but the filesystem is far from full, which I think helps wear
leveling. The RAID has ~4TB and only ~500GB was ever used. But who
knows, and if that's the cause, we'll never really know I'm afraid. It's
hard to confirm what happens inside a SSD :-(


regards

-- 
Tomas Vondra

  210. Re: AIO v2.5

    Tomas Vondra <tomas@vondra.me> — 2025-07-15T00:30:10Z

    
On 7/14/25 20:36, Andres Freund wrote:
> Hi,
> 
> On 2025-07-11 23:03:53 +0200, Tomas Vondra wrote:
>> I've been running some benchmarks comparing the io_methods, to help with
>> resolving this PG18 open item. So here are some results, and my brief
>> analysis of it.
> 
> Thanks for doing that!
> 
> 
> 
>> The TL;DR version
>> -----------------
>>
>> * The "worker" method seems good, and I think we should keep it as a
>> default. We should probably think about increasing the number of workers
>> a bit, the current io_workers=3 seems to be too low and regresses in a
>> couple tests.
>>
>> * The "sync" seems OK too, but it's more of a conservative choice, i.e.
>> more weight for keeping the PG17 behavior / not causing regressions. But
>> I haven't seen that (with enough workers). And there are cases when the
>> "worker" is much faster. It'd be a shame to throw away that benefit.
>>
>> * There might be bugs in "worker", simply because it has to deal with
>> multiple concurrent processes etc. But I guess we'll fix those just like
>> other bugs. I don't think it's a good argument against "worker" default.
>>
>> * All my tests were done on Linux and NVMe drives. It'd be good to do
>> similar testing on other platforms (e.g. FreeBSD) and/or storage. I plan
>> to do some of that, but it'd be great to cover more cases. I can help
>> with getting my script running, a run takes ~1-2 days.
> 
> FWIW, in my very limited tests on windows, the benefit of worker was
> considerably bigger there, due to having much more minimal readahead not
> having posix_fadvise...
> 
> 
>> The test also included PG17 for comparison, but I forgot PG18 enabled
>> checksums by default. So PG17 results are with checksums off, which in
>> some cases means PG17 seems a little bit faster. I'm re-running it with
>> checksums enabled on PG17, and that seems to eliminate the differences
>> (as expected).
> 
> My sneaking suspicion is that, independent of AIO, we're not really ready to
> default to checksums defaulting to on.
> 
> 
>> Findings
>> --------
>>
>> I'm attaching only three PDFs with charts from the cold runs, to keep
>> the e-mail small (each PDF is ~100-200kB). Feel free to check the other
>> PDFs in the git repository, but it's all very similar and the attached
>> PDFs are quite representative.
>>
>> Some basic observations:
>>
>> a) index scans
>>
>> There's almost no difference for indexscans, i.e. the middle column in
>> the PDFs. There's a bit of variation on some of the cyclic/linear data
>> sets, but it seems more like random noise than a systemic difference.
>>
>> Which is not all that surprising, considering index scans don't really
>> use read_stream yet, so there's no prefetching etc.
> 
> Indeed.
> 
> 
>> The "ryzen" results however demonstrate that 3 workers may be too low.
>> The timing spikes to ~3000ms (at ~1% selectivity), before quickly
>> dropping back to ~1000ms. The other datasets show similar difference.
>> With 12 workers, there's no such problem.
> 
> I don't really know what to do about that - for now we don't have dynamic
> #workers, and starting 12 workers on a tiny database doesn't really make
> sense...  I suspect that on most hardware & queries it won't matter that much,
> but clearly, if you have high iops hardware it might.  I can perhaps see
> increasing the default to 5 or so, but after that...  I guess we could try
> some autoconf formula based on the size of s_b or such? But that seems
> somewhat awkward too.
> 

True. I don't have a great idea either. FWIW most of our defaults are
very conservative/low, and you have to bump them up on bigger machines
anyway. So having to bump one more GUC is not a big deal, and I don't
think we have to invent some magic formula for this one.

Also, autoconf wouldn't even know about s_b size, it'd have to be
something at startup. We could do some automated sizing if set to -1,
perhaps. But is s_b even a good value to tie this to? I doubt that.

> 
> 
>>
>> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
>>
>> There's an interesting difference difference I noticed in the run with
>> checksums on PG17. The full PDF is available here:
> 
> (there's a subsequent email about this, will reply there)
> 
> 
>> Conclusion
>> ----------
>>
>> That's all I have at the moment. I still think it makes sense to keep
>> io_method=worker, but bump up the io_workers a bit higher. Could we also
>> add some suggestions how to pick a good value to the docs?
> 
> .oO(/me ponders a troll patch to re-add a reference the number of spindles in
> that tuning advice)
>

;-)

> I'm not sure what advice to give here.  Maybe just to set it to a considerably
> larger number once not running on a tiny system? The incremental overhead of
> having an idle worker is rather small unless you're on a really tiny system...
> 

Too bad the patch doesn't collect any stats about how utilized the
workers are :-( That'd make it a bit easier, we could even print
something into the log if the queues overflow "too often", similarly to
max_wal_size when checkpoints happen too often.

> 
>> You might also run the benchmark on different hardware, and either
>> build/publish the plots somewhere, or just give me the CSV and I'll do
>> that. Better to find strange stuff / regressions now.
> 
> Probably the most interesting thing would be some runs with cloud-ish storage
> (relatively high iops, very high latency)...
> 

Yeah, I've started a test on a cloud VM. Will see in a day or two. And
another on FreeBSD, for good measure.

> 
>> The repository also has branches with plots showing results with WIP
>> indexscan prefetching. (It's excluded from the PDFs I presented here).
> 
> Hm, I looked for those, but I couldn't quickly find any plots that include
> them.  Would I have to generate the plots from a checkout of the repo?
> 

No, the charts are there, you don't need to generate them.

Look into the "with-indexscan-prefetch-run2-17-checksums" branch. E.g.
this is the same "ryzen" plot I shared earlier, but with "indexscan
prefetch" column:

https://github.com/tvondra/iomethod-tests/blob/with-indexscan-prefetch-run2-17-checksums/ryzen-rows-cold-32GB-16-unscaled.pdf

It might be better to look at the "scaled" charts, which makes it easier
to compare the different scans (and the benefit of prefetching).

https://github.com/tvondra/iomethod-tests/blob/with-indexscan-prefetch-run2-17-checksums/ryzen-rows-cold-32GB-16-scaled.pdf

> 
>> The conclusions are similar to what we found here - "worker" is good
>> with enough workers, io_uring is good too. Sync has issues for some of
>> the data sets, but still helps a lot.
> 
> Nice.
> 
> Greetings,
> 
> Andres Freund


regards

-- 
Tomas Vondra

  211. Re: AIO v2.5

    Andres Freund <andres@anarazel.de> — 2025-07-22T13:04:10Z

    Hi,

On 2025-07-10 21:00:21 +0200, Matthias van de Meent wrote:
> On Wed, 9 Jul 2025 at 16:59, Andres Freund <andres@anarazel.de> wrote:
> > On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote:
> > > I've been going through the new AIO code as an effort to rebase and
> > > adapt Neon to PG18. In doing so, I found the following
> > > items/curiosities:
> > >
> > > 1. In aio/README.md, the following code snippet is found:
> > >
> > > [...]
> > > pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
> > > [...]
> > >
> > > I believe it would be clearer if it took a reference to the buffer:
> > >
> > > pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
> > >
> > > The main reason here is that common practice is to have a `Buffer
> > > buffer;` whereas a Buffer * is more commonly plural.
> >
> > It's also just simply wrong as-is :/. Interpreting the buffer id as a pointer
> > obviously makes no sense...
> 
> Given that the snippet didn't contain type indications for buffer upto
> that point, technically the buffer variable could've been defined as
> `Buffer* buffer;` which would've been type-correct. That would be very
> confusing however, hence the suggested change.
> 
> After your mail, I also noticed the later snippet which should be updated, too:
> 
> ```
> -smgrstartreadv(ioh, operation->smgr, forknum, blkno,
> -               BufferGetBlock(buffer), 1);
> +void *page = BufferGetBlock(buffer);
> +smgrstartreadv(ioh, operation->smgr, forknum, blkno,
> +               &page, 1);
> ```

I now pushed your change, with that squashed in.

Thanks!

Andres

  212. Re: AIO v2.5

    Tomas Vondra <tomas@vondra.me> — 2025-07-22T13:51:12Z

    Hi,

I kept running the tests on additional storage systems, both "local"
(e.g. RAID on SATA SSDs) and "cloud" (Azure Premium SSD volumes). And
also on FreeBSD/ZFS, to see what non-Linux systems do. I pushed all the
results to the same github repository, into separate branches.

I'm attaching three example charts, from these "new" systems.

In general, it doesn't change the earlier conclusions at all. Most of
the earlier observations apply to these results from these machines with
reasonably different types of storage. So I still think we should stick
to "worker", but consider increasing the default number of workers.

It's possible there's machine with storage/hardware that would work
better with some other io_method value, but that's why it's a GUC. The
default should work for most systems.

I plan to let the current tests complete, but that's about it. I don't
plan to test additional systems for this thread. I'll keep running tests
for the index-prefetch thread, but that's focused on other questions.


regards

-- 
Tomas Vondra

  213. Re: AIO v2.5

    Nathan Bossart <nathandbossart@gmail.com> — 2025-07-29T16:41:01Z

    [RMT hat]

On Tue, Jul 22, 2025 at 03:51:12PM +0200, Tomas Vondra wrote:
> In general, it doesn't change the earlier conclusions at all. Most of
> the earlier observations apply to these results from these machines with
> reasonably different types of storage. So I still think we should stick
> to "worker", but consider increasing the default number of workers.
> 
> It's possible there's machine with storage/hardware that would work
> better with some other io_method value, but that's why it's a GUC. The
> default should work for most systems.

Given there have been no further proposals or discussion, we have marked
the related open item as "Won't Fix".  This means we plan to ship v18 with
io_method=worker and io_workers=3 by default.

-- 
nathan