Re: Non-reproducible AIO failure
Konstantin Knizhnik <knizhnik@garret.ru>
From: Konstantin Knizhnik <knizhnik@garret.ru>
To: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2025-06-12T13:30:54Z
Lists: pgsql-hackers
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
aio: Stop using enum bitfields due to bad code generation
- ce161b194e84 18.0 landed
- 5865150b6d53 19 (unreleased) landed
-
amcheck: Fix posting tree checks in gin_index_check()
- 0cf205e122ae 18.0 cited
-
aio: Add missing memory barrier when waiting for IO handle
- e9a3615a5224 18.0 landed
On 12/06/2025 4:13 pm, Andres Freund wrote: > Hi, > > On 2025-06-12 15:12:00 +0300, Konstantin Knizhnik wrote: >> Reproduced it once again with with write-protected io handle. >> But once again - no access violation, just assert failure. >> >> Previously "op" field was overwritten somewhere between `pgaio_io_reclaim` >> and `AsyncReadBuffers`: >> >> !!!pgaio_io_reclaim [20376]| ioh: 0x1019bc000, ioh->op: 0, ioh->generation: >> 19346 >> !!!AsyncReadBuffers [20376] (1)| blocknum: 21, ioh: 0x1019bc000, ioh->op: 1, >> ioh->state: 1, ioh->result: 0, ioh->num_callbacks: 0, ioh->generation: 19346 >> >> Now it is overwritten after print in AsyncReadBuffers: >> >> !!!pgaio_io_reclaim [88932]| ioh: 0x105a5c000, ioh->op: 0, ioh->generation: >> 42848 >> !!!pgaio_io_acquire_nb[88932]| ioh: 0x105a5c000, ioh->op: 0, >> ioh->generation: 42848 >> !!!AsyncReadBuffers [88932] (1)| blocknum: 10, ioh: 0x105a5c000, ioh->op: 0, >> ioh->state: 1, ioh->result: 0, ioh->num_callbacks: 0, ioh->generation: 42848 >> !!!pgaio_io_before_start| ioh: 0x105a5c000, ioh->op: 1, ioh->state: 1, >> ioh->result: 0, ioh->num_callbacks: 2, ioh->generation: 42848 >> >> In this run I prohibit writes to io handle in `pgaio_io_acquire_nb` and >> reenable them in `AsyncReadBuffer`. > I'm reasonably certain I found the issue, I think it's a missing memory > barrier on the read side. The CPU is reordering the read (or just using a > cached value) of ->distilled_result to be before the load of ->state. > > But it'll take a bit to verify that that's the issue... It is great! But I wonder how it correlates with your previous statement: > There shouldn't be any concurrent accesses here, so I don't really see how the > above would explain the problem (the IO can only ever be modified by one > backend, initially the "owning backend", then, when submitted, by the IO > worker, and then again by the backend). This is what I am observing myself: "op" field is modified and fetched by the same process. Certainly process can be rescheduled to some other CPU. But if such reschedule can cause loose of stored value, then nothing will work, will it? So assume that there is some variable "x" which is updated by process "x=1" executed at CPU1, then process is rescheduled to CPU2 which does "x=2", then process is once again rescheduled to CPU1 and we found out that "x==1". And to prevent it we need to explicitly enforce some memory barrier. Unless I missing something , nothing can work with such memory model.