Horton

Re: Non-reproducible AIO failure

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Konstantin Knizhnik <knizhnik@garret.ru>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2025-06-12T13:13:04Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. aio: Stop using enum bitfields due to bad code generation

  2. amcheck: Fix posting tree checks in gin_index_check()

  3. aio: Add missing memory barrier when waiting for IO handle

Hi,

On 2025-06-12 15:12:00 +0300, Konstantin Knizhnik wrote:
> Reproduced it once again with with write-protected io handle.
> But once again - no access violation, just assert failure.
> 
> Previously "op" field was overwritten somewhere between `pgaio_io_reclaim`
> and `AsyncReadBuffers`:
> 
> !!!pgaio_io_reclaim [20376]| ioh: 0x1019bc000, ioh->op: 0, ioh->generation:
> 19346
> !!!AsyncReadBuffers [20376] (1)| blocknum: 21, ioh: 0x1019bc000, ioh->op: 1,
> ioh->state: 1, ioh->result: 0, ioh->num_callbacks: 0, ioh->generation: 19346
> 
> Now it is overwritten after print in AsyncReadBuffers:
> 
> !!!pgaio_io_reclaim [88932]| ioh: 0x105a5c000, ioh->op: 0, ioh->generation:
> 42848
> !!!pgaio_io_acquire_nb[88932]| ioh: 0x105a5c000, ioh->op: 0,
> ioh->generation: 42848
> !!!AsyncReadBuffers [88932] (1)| blocknum: 10, ioh: 0x105a5c000, ioh->op: 0,
> ioh->state: 1, ioh->result: 0, ioh->num_callbacks: 0, ioh->generation: 42848
> !!!pgaio_io_before_start| ioh: 0x105a5c000, ioh->op: 1, ioh->state: 1,
> ioh->result: 0, ioh->num_callbacks: 2, ioh->generation: 42848
> 
> In this run I prohibit writes to io handle in `pgaio_io_acquire_nb` and
> reenable them in `AsyncReadBuffer`.

I'm reasonably certain I found the issue, I think it's a missing memory
barrier on the read side. The CPU is reordering the read (or just using a
cached value) of ->distilled_result to be before the load of ->state.

But it'll take a bit to verify that that's the issue...

Greetings,

Andres Freund