Thread
-
Add per-backend AIO statistics
Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2026-07-07T11:02:03Z
Hi hackers, Currently to monitor AIO we can use: 1/ pg_aios that lists all AIO handles that are currently in use. That shows what's happening right now, but not what has happened. 2/ pg_stat_get_backend_io() that shows how much IO was done, but not how it was done. There's no way to see whether IOs ran synchronously or asynchronously, whether a backend was stalling on handle exhaustion, or how completions are distributed across backends. This patch helps answering those questions by exposing cumulative per-backend AIO counters: - started: total AIO operations initiated - executed_sync: IOs executed synchronously (fallback path) - executed_async: IOs submitted asynchronously - completed_self: IO completions processed by the issuing backend - completed_other: IO completions processed on behalf of another backend - handle_waits: times waited for a free AIO handle - submitted: number of submit calls to the IO method These counters are useful for understanding and tuning AIO behavior: - executed_async / started. A ratio near zero means the backend is falling back to synchronous execution (TOAST chunk fetches, temp buffers, ...). - a non-zero handle_waits means the backend exhausted all its AIO handles. That could mean that io_max_concurrency is too low. - completed_self vs completed_other reveals cross-backend completion patterns. That helps see how IO completion work is distributed and could help interpret per backend IO statistics values. - executed_async / submitted gives the average batch size per submit call. As far as the technical implementation: This data can be retrieved with a new system function called pg_stat_get_backend_aio(), that returns one row based on the PID provided in input. pgstat_flush_backend() gains a new flag value, able to control the flush of the AIO stats. This patch relies mostly on the infrastructure provided by 9aea73fc61d4, that has introduced backend statistics. The overhead (4 functions calls and counters increments) kind of follow the same patterns as pgstat_count_backend_io_op() and I did not observe measurable regression (I did not expect to). Also that does not add that much memory per-backend: PgStat_AioCounters is 56 bytes. There is no "double" counting as a global view to show those counters does not exist. I think that's better to start with the per-backend side of it and see if we want to also add a global view. For example, completed_other identifies which backends did IOs for other backends. Also this allows correlating with pg_stat_activity and pg_stat_get_backend_io(). Examples based on Franck's blog post [1]: 1/ query the smalldocs table: postgres=# select count(*),avg(length(data)) from smalldocs; count | avg ---------+----------------------- 1024000 | 1024.0000000000000000 (1 row) postgres=# SELECT * FROM pg_stat_get_backend_aio(pg_backend_pid()); started | executed_sync | executed_async | completed_self | completed_other | handle_waits | submitted | stats_reset ---------+---------------+----------------+----------------+-----------------+--------------+-----------+------------------------------- 3125 | 46 | 3079 | 46 | 0 | 0 | 3078 | 2026-07-07 09:28:27.412136+00 We can see that the sequential scan fully benefits from AIO. 2/ query the largedocs table: postgres=# select count(*),avg(length(data)) from largedocs; count | avg -------+---------------------- 1000 | 1048576.000000000000 (1 row) postgres=# SELECT * FROM pg_stat_get_backend_aio(pg_backend_pid()); started | executed_sync | executed_async | completed_self | completed_other | handle_waits | submitted | stats_reset ---------+---------------+----------------+----------------+-----------------+--------------+-----------+------------------------------- 121154 | 121150 | 4 | 121150 | 0 | 0 | 4 | 2026-07-07 09:35:00.504872+00 We can see that the sequential scan bypasses AIO. Looking forward to your feedback. [1]: https://dev.to/franckpachot/iouring-buffered-reads-in-postgresql-19-iouring-mcn Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com -
Re: Add per-backend AIO statistics
Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2026-07-08T06:00:22Z
Hi, On Tue, Jul 07, 2026 at 11:02:03AM +0000, Bertrand Drouvot wrote: > postgres=# select count(*),avg(length(data)) from smalldocs; > count | avg > ---------+----------------------- > 1024000 | 1024.0000000000000000 > (1 row) > > postgres=# SELECT * FROM pg_stat_get_backend_aio(pg_backend_pid()); > started | executed_sync | executed_async | completed_self | completed_other | handle_waits | submitted | stats_reset > ---------+---------------+----------------+----------------+-----------------+--------------+-----------+------------------------------- > 3125 | 46 | 3079 | 46 | 0 | 0 | 3078 | 2026-07-07 09:28:27.412136+00 > > We can see that the sequential scan fully benefits from AIO. > > 2/ query the largedocs table: > > postgres=# select count(*),avg(length(data)) from largedocs; > count | avg > -------+---------------------- > 1000 | 1048576.000000000000 > (1 row) > > postgres=# SELECT * FROM pg_stat_get_backend_aio(pg_backend_pid()); > started | executed_sync | executed_async | completed_self | completed_other | handle_waits | submitted | stats_reset > ---------+---------------+----------------+----------------+-----------------+--------------+-----------+------------------------------- > 121154 | 121150 | 4 | 121150 | 0 | 0 | 4 | 2026-07-07 09:35:00.504872+00 > > We can see that the sequential scan bypasses AIO. I was just doing some AIO experiments and was using the new pg_stat_get_backend_aio() function. So, while at it, sharing more examples here: 3/ pg_stat_get_backend_aio() and pg_stat_get_backend_io() correlation postgres=# SELECT executed_sync, executed_async FROM pg_stat_get_backend_aio(pg_backend_pid()); executed_sync | executed_async ---------------+---------------- 46 | 3088 (1 row) postgres=# SELECT object, context, reads, read_bytes FROM pg_stat_get_backend_io(pg_backend_pid()); object | context | reads | read_bytes ---------------+-----------+-------+------------ relation | bulkread | 3088 | 401580032 relation | bulkwrite | 0 | 0 relation | init | 0 | 0 relation | normal | 46 | 376832 relation | vacuum | 0 | 0 temp relation | normal | 0 | 0 wal | init | | wal | normal | 0 | 0 (8 rows) We can see that the "executed_sync" matches the reads "normal" context and that the "executed_async" matches the reads "bulkread" context. 4/ io_uring and multiple backends postgres=# SELECT a.pid, (pg_stat_get_backend_aio(a.pid)).completed_other FROM pg_stat_activity a WHERE a.backend_type = 'client backend'; pid | completed_other ---------+----------------- 1911889 | 245 1911892 | 511 1911912 | 147 1911933 | 161 (4 rows) We can see that the backends completed AIO on behalf of other backends, which makes fully sense in io_uring mode. 5/ io_max_concurrency = 4 postgres=# SELECT started, handle_waits FROM pg_stat_get_backend_aio(pg_backend_pid()); started | handle_waits ---------+-------------- 3139 | 3026 (1 row) We can see that the backend had to wait for free AIO handles on 96% of its IOs. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com -
Re: Add per-backend AIO statistics
Michael Paquier <michael@paquier.xyz> — 2026-07-08T06:52:20Z
On Tue, Jul 07, 2026 at 11:02:03AM +0000, Bertrand Drouvot wrote: > 1/ pg_aios that lists all AIO handles that are currently in use. That shows > what's happening right now, but not what has happened. > > 2/ pg_stat_get_backend_io() that shows how much IO was done, but not how it > was done. There's no way to see whether IOs ran synchronously or > asynchronously, whether a backend was stalling on handle exhaustion, or how > completions are distributed across backends. While the information may be useful, one thing that sounds very important to me is how this impacts workloads by default. Andres is usually able to catch bottlenecks that everybody else is unable to see, so perhaps checking with him the location of these extra function calls would be a good first step. Your proposal goes down to pgaio_io_stage(), pgaio_io_process_completion() and pgaio_submit_staged() to track these counter increments. -- Michael
-
Re: Add per-backend AIO statistics
Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2026-07-08T08:15:26Z
Hi, On Wed, Jul 08, 2026 at 03:52:20PM +0900, Michael Paquier wrote: > On Tue, Jul 07, 2026 at 11:02:03AM +0000, Bertrand Drouvot wrote: > > 1/ pg_aios that lists all AIO handles that are currently in use. That shows > > what's happening right now, but not what has happened. > > > > 2/ pg_stat_get_backend_io() that shows how much IO was done, but not how it > > was done. There's no way to see whether IOs ran synchronously or > > asynchronously, whether a backend was stalling on handle exhaustion, or how > > completions are distributed across backends. > > While the information may be useful, Thanks for looking at it! > Andres is usually able to catch bottlenecks that everybody else is > unable to see, so perhaps checking with him the location of these > extra function calls would be a good first step. Your proposal goes > down to pgaio_io_stage(), pgaio_io_process_completion() and > pgaio_submit_staged() to track these counter increments. yeah, and also to 1/ confirm that I did understand this area of the AIO code correctly and 2/ see if other counters could make sense. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com