Thread
-
BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
PG Bug reporting form <noreply@postgresql.org> — 2026-01-02T22:46:17Z
The following bug has been logged on the website: Bug reference: 19369 Logged by: Evan Si Email address: evsi@amazon.com PostgreSQL version: 18.1 Operating system: Amazon Linux 2 Description: Hello, The documentation doesn't explicitly state a minimum kernel version for io_uring, but in src/backend/storage/aio/README.md, it claims that: > io_method=io_uring is available on Linux 5.1+. Attempting to set io_method to io_uring on versions without io_uring at all (below 5.1) causes the server to graciously report that "[the k]ernel does not support io_uring". But setting io_uring on kernel versions between 5.1 and 5.6 sees the server start but connections failing with EINVAL. ``` [ec2-user@ip-172-31-56-49 postgres]$ psql postgres psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument ``` It seems like the implementation in postgres leverages non-vectored I/O (i.e. io_uring_prep_read and io_uring_prep_write) in src/backend/storage/aio/method_io_uring.c. But, these opcodes aren't actually available until kernel 5.6 (https://github.com/axboe/liburing/issues/44), (https://man7.org/linux/man-pages/man2/io_uring_enter.2.html - under IORING_OP_READ, IORING_OP_WRITE). It seems kernel 5.4 LTS ended last month, but this seems worth clarifying, I think. -- Precise reproduction example: OS details ``` [ec2-user@ip-172-31-56-49 postgres]$ uname -a Linux ip-172-31-56-49.us-west-2.compute.internal 5.4.301-221.450.amzn2.x86_64 #1 SMP Tue Nov 18 16:40:04 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux ``` Build postgres with liburing 2.8, set postgresql.conf with io_method=io_uring, log_min_messages = debug3. Postgres starts without complaint, but trying to connect will fail: ``` [ec2-user@ip-172-31-56-49 postgres]$ psql postgres psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument ``` Inside logfile: ``` 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 3, ref_gen: 3, cycle 0 2026-01-02 22:35:43.863 UTC [8517] LOG: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-02 22:35:43.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state IDLE : wait_one io_gen: 4, ref_gen: 3, cycle 0 2026-01-02 22:35:43.863 UTC [8517] DEBUG: wait_one with 0 sleeps 2026-01-02 22:35:43.863 UTC [8517] ERROR: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-02 22:35:44.057 UTC [8512] DEBUG: assigned pm child slot 1 for backend 2026-01-02 22:35:44.058 UTC [8512] DEBUG: forked new client backend, pid=8521 socket=151 2026-01-02 22:35:44.058 UTC [8521] DEBUG: InitPostgres 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 1, ref_gen: 1, cycle 0 2026-01-02 22:35:44.059 UTC [8521] LOG: could not read blocks 0..0 in file "global/1260": Invalid argument 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-02 22:35:44.059 UTC [8521] DEBUG: drained 1/1, now expecting 0 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target invalid|state IDLE : wait_one io_gen: 2, ref_gen: 1, cycle 0 2026-01-02 22:35:44.059 UTC [8521] DEBUG: wait_one with 0 sleeps 2026-01-02 22:35:44.059 UTC [8521] FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 5 before_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 7 on_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(1): 2 callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: exit(1) 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(-1): 0 callbacks to make 2026-01-02 22:35:44.060 UTC [8512] DEBUG: releasing pm child slot 1 2026-01-02 22:35:44.060 UTC [8512] DEBUG: client backend (PID 8521) exited with exit code 1 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 4, ref_gen: 4, cycle 0 2026-01-02 22:35:44.863 UTC [8517] LOG: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-02 22:35:44.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state IDLE : wait_one io_gen: 5, ref_gen: 4, cycle 0 2026-01-02 22:35:44.863 UTC [8517] DEBUG: wait_one with 0 sleeps 2026-01-02 22:35:44.863 UTC [8517] ERROR: could not read blocks 0..0 in file "global/1262": Invalid argument ```
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
surya poondla <suryapoondla4@gmail.com> — 2026-01-08T04:15:33Z
Hi Evsi, I created a new Ubuntu server and am able to reproduce the same behavior you reported for Postgres 18.1 on aarch64 architecture. surya@surya:~/pgdata18/log$ uname -r 5.4.0-216-generic surya@surya:~/pgdata18/log$ uname -a Linux surya 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:55:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux I did some testing and see the below In the postgresql.conf if I put io_method = 'worker' or io_method = 'sync', postgres is able to restart with no errors. But if io_method = 'io_uring' then I see the below on my console surya@surya:~/postgresql-18.1$ vim ~/pgdata18/postgresql.conf surya@surya:~/postgresql-18.1$ ~/pg18/bin/pg_ctl -D ~/pgdata18 restart waiting for server to shut down....2026-01-08 04:06:07.081 UTC [44973] DEBUG: logger shutting down 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(0): 0 before_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(0): 0 on_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: proc_exit(0): 0 callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: exit(0) 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: proc_exit(-1): 0 callbacks to make done server stopped waiting for server to start....2026-01-08 04:06:07.172 GMT [44988] LOG: invalid value for parameter "io_method": "io_uring" 2026-01-08 04:06:07.172 GMT [44988] HINT: Available values: sync, worker. 2026-01-08 04:06:07.172 UTC [44988] FATAL: configuration file "/home/surya/pgdata18/postgresql.conf" contains errors 2026-01-08 04:06:07.172 UTC [44988] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-01-08 04:06:07.172 UTC [44988] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-01-08 04:06:07.172 UTC [44988] DEBUG: proc_exit(1): 0 callbacks to make 2026-01-08 04:06:07.172 UTC [44988] DEBUG: exit(1) stopped waiting pg_ctl: could not start server Examine the log output. From the above we can see that the stderr says that "*invalid value for parameter "io_method": "io_uring"*" And in the logs I see similar thing like you reported 2026-01-08 04:05:31.438 UTC [44972] DEBUG: starting background worker process "logical replication launcher" 2026-01-08 04:05:31.438 UTC [44981] DEBUG: autovacuum launcher started 2026-01-08 04:05:31.438 UTC [44981] DEBUG: InitPostgres 2026-01-08 04:05:31.438 UTC [44982] DEBUG: logical replication launcher started 2026-01-08 04:05:31.438 UTC [44982] DEBUG: InitPostgres 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-08 04:05:31.439 UTC [44974] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 2026-01-08 04:05:31.439 UTC [44981] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 I am working on a patch and will fix the inconsistencies in documentation about io_uring. Best, Surya Poondla On Wed, Jan 7, 2026 at 7:54 PM PG Bug reporting form <noreply@postgresql.org> wrote: > The following bug has been logged on the website: > > Bug reference: 19369 > Logged by: Evan Si > Email address: evsi@amazon.com > PostgreSQL version: 18.1 > Operating system: Amazon Linux 2 > Description: > > Hello, > > The documentation doesn't explicitly state a minimum kernel version for > io_uring, but in src/backend/storage/aio/README.md, it claims that: > > > io_method=io_uring is available on Linux 5.1+. > > Attempting to set io_method to io_uring on versions without io_uring at all > (below 5.1) causes the server to graciously report that "[the k]ernel does > not support io_uring". > But setting io_uring on kernel versions between 5.1 and 5.6 sees the server > start but connections failing with EINVAL. > > ``` > [ec2-user@ip-172-31-56-49 postgres]$ psql postgres > psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: > FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument > ``` > > It seems like the implementation in postgres leverages non-vectored I/O > (i.e. io_uring_prep_read and io_uring_prep_write) in > src/backend/storage/aio/method_io_uring.c. > > But, these opcodes aren't actually available until kernel 5.6 > (https://github.com/axboe/liburing/issues/44), > (https://man7.org/linux/man-pages/man2/io_uring_enter.2.html - under > IORING_OP_READ, IORING_OP_WRITE). > > It seems kernel 5.4 LTS ended last month, but this seems worth clarifying, > I > think. > > -- > > Precise reproduction example: > > OS details > ``` > [ec2-user@ip-172-31-56-49 postgres]$ uname -a > Linux ip-172-31-56-49.us-west-2.compute.internal > 5.4.301-221.450.amzn2.x86_64 #1 SMP Tue Nov 18 16:40:04 UTC 2025 x86_64 > x86_64 x86_64 GNU/Linux > ``` > > Build postgres with liburing 2.8, set postgresql.conf with > io_method=io_uring, log_min_messages = debug3. > > Postgres starts without complaint, but trying to connect will fail: > > ``` > [ec2-user@ip-172-31-56-49 postgres]$ psql postgres > psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: > FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument > ``` > Inside logfile: > ``` > 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target > invalid|state HANDED_OUT : adding cb #1, id > 2/aio_shared_buffer_readv_cb > 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target > smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state DEFINED : calling cb #1 > 2/aio_shared_buffer_readv_cb->stage(0) > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state STAGED : staged (synchronous: 0, in_batch: 1) > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state SUBMITTED : wait_one io_gen: 3, ref_gen: 3, cycle 0 > 2026-01-02 22:35:43.863 UTC [8517] LOG: could not read blocks 0..0 in file > "global/1262": Invalid argument > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state COMPLETED_IO : after shared completion: distilled result: > (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, > id 1, error_data 22, result 0), raw_result: -22 > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op invalid|target > invalid|state IDLE : wait_one io_gen: 4, ref_gen: 3, cycle 0 > 2026-01-02 22:35:43.863 UTC [8517] DEBUG: wait_one with 0 sleeps > 2026-01-02 22:35:43.863 UTC [8517] ERROR: could not read blocks 0..0 in > file "global/1262": Invalid argument > 2026-01-02 22:35:44.057 UTC [8512] DEBUG: assigned pm child slot 1 for > backend > 2026-01-02 22:35:44.058 UTC [8512] DEBUG: forked new client backend, > pid=8521 socket=151 > 2026-01-02 22:35:44.058 UTC [8521] DEBUG: InitPostgres > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target > invalid|state HANDED_OUT : adding cb #1, id > 2/aio_shared_buffer_readv_cb > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target > smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target > smgr|state DEFINED : calling cb #1 > 2/aio_shared_buffer_readv_cb->stage(0) > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target > smgr|state STAGED : staged (synchronous: 0, in_batch: 1) > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target > smgr|state SUBMITTED : wait_one io_gen: 1, ref_gen: 1, cycle 0 > 2026-01-02 22:35:44.059 UTC [8521] LOG: could not read blocks 0..0 in file > "global/1260": Invalid argument > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target > smgr|state COMPLETED_IO : after shared completion: distilled result: > (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target > smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, > id 1, error_data 22, result 0), raw_result: -22 > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: drained 1/1, now expecting 0 > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target > invalid|state IDLE : wait_one io_gen: 2, ref_gen: 1, cycle 0 > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: wait_one with 0 sleeps > 2026-01-02 22:35:44.059 UTC [8521] FATAL: could not read blocks 0..0 in > file "global/1260": Invalid argument > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 5 > before_shmem_exit callbacks to make > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 7 on_shmem_exit > callbacks to make > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(1): 2 callbacks to > make > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: exit(1) > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 > before_shmem_exit callbacks to make > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 on_shmem_exit > callbacks to make > 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(-1): 0 callbacks to > make > 2026-01-02 22:35:44.060 UTC [8512] DEBUG: releasing pm child slot 1 > 2026-01-02 22:35:44.060 UTC [8512] DEBUG: client backend (PID 8521) exited > with exit code 1 > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target > invalid|state HANDED_OUT : adding cb #1, id > 2/aio_shared_buffer_readv_cb > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target > smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state DEFINED : calling cb #1 > 2/aio_shared_buffer_readv_cb->stage(0) > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state STAGED : staged (synchronous: 0, in_batch: 1) > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state SUBMITTED : wait_one io_gen: 4, ref_gen: 4, cycle 0 > 2026-01-02 22:35:44.863 UTC [8517] LOG: could not read blocks 0..0 in file > "global/1262": Invalid argument > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state COMPLETED_IO : after shared completion: distilled result: > (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target > smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, > id 1, error_data 22, result 0), raw_result: -22 > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target > invalid|state IDLE : wait_one io_gen: 5, ref_gen: 4, cycle 0 > 2026-01-02 22:35:44.863 UTC [8517] DEBUG: wait_one with 0 sleeps > 2026-01-02 22:35:44.863 UTC [8517] ERROR: could not read blocks 0..0 in > file "global/1262": Invalid argument > ``` > >
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
surya poondla <suryapoondla4@gmail.com> — 2026-01-08T06:52:10Z
Hi Evan, In my previous email/findings, I didn't compile postgres with the liburing option. I did it this time and see the same symptoms as you saw. surya@surya:~/pgdata18$ uname -r 5.4.0-216-generic surya@surya:~/pgdata18$ uname -a Linux surya 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:55:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux surya@surya:~/pgdata18$ surya@surya:~/pgdata18$ ~/pg18/bin/pg_ctl -D ~/pgdata18 restart waiting for server to shut down.... done server stopped waiting for server to start....2026-01-08 06:20:36.301 UTC [17042] LOG: redirecting log output to logging collector process 2026-01-08 06:20:36.301 UTC [17042] HINT: Future log output will appear in directory "log". done server started surya@surya:~/pgdata18$ ls log/ postgresql-2026-01-08_061910.log postgresql-2026-01-08_062036.log postgresql-2026-01-08_061955.log surya@surya:~/pgdata18$ ~/pg18/bin/psql -d postgres psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument In the logs, I see 026-01-08 06:24:29.858 UTC [17126] DEBUG: InitPostgres 2026-01-08 06:24:29.858 UTC [17127] DEBUG: logical replication launcher started 2026-01-08 06:24:29.858 UTC [17127] DEBUG: InitPostgres 2026-01-08 06:24:29.858 UTC [17126] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-08 06:24:29.858 UTC [17126] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-08 06:24:29.858 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 1, ref_gen: 1, cycle 0 2026-01-08 06:24:29.859 UTC [17126] LOG: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-08 06:24:29.859 UTC [17126] DEBUG: drained 1/1, now expecting 0 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op invalid|target invalid|state IDLE : wait_one io_gen: 2, ref_gen: 1, cycle 0 2026-01-08 06:24:29.859 UTC [17126] DEBUG: wait_one with 0 sleeps 2026-01-08 06:24:29.859 UTC [17126] ERROR: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-08 06:24:30.859 UTC [17126] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cbhe logs I will do some more testing on this and update my findings. -Surya Poondla On Wed, Jan 7, 2026 at 8:15 PM surya poondla <suryapoondla4@gmail.com> wrote: > Hi Evsi, > > I created a new Ubuntu server and am able to reproduce the same behavior > you reported for Postgres 18.1 on aarch64 architecture. > > surya@surya:~/pgdata18/log$ uname -r > 5.4.0-216-generic > surya@surya:~/pgdata18/log$ uname -a > Linux surya 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:55:34 UTC 2025 > aarch64 aarch64 aarch64 GNU/Linux > > I did some testing and see the below > In the postgresql.conf if I put > io_method = 'worker' or io_method = 'sync', postgres is able to restart > with no errors. > But if io_method = 'io_uring' then I see the below on my console > > surya@surya:~/postgresql-18.1$ vim ~/pgdata18/postgresql.conf > surya@surya:~/postgresql-18.1$ ~/pg18/bin/pg_ctl -D ~/pgdata18 restart > waiting for server to shut down....2026-01-08 04:06:07.081 UTC [44973] > DEBUG: logger shutting down > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(0): 0 > before_shmem_exit callbacks to make > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(0): 0 on_shmem_exit > callbacks to make > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: proc_exit(0): 0 callbacks to > make > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: exit(0) > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(-1): 0 > before_shmem_exit callbacks to make > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(-1): 0 > on_shmem_exit callbacks to make > 2026-01-08 04:06:07.081 UTC [44973] DEBUG: proc_exit(-1): 0 callbacks to > make > done > server stopped > waiting for server to start....2026-01-08 04:06:07.172 GMT [44988] LOG: > invalid value for parameter "io_method": "io_uring" > 2026-01-08 04:06:07.172 GMT [44988] HINT: Available values: sync, worker. > 2026-01-08 04:06:07.172 UTC [44988] FATAL: configuration file > "/home/surya/pgdata18/postgresql.conf" contains errors > 2026-01-08 04:06:07.172 UTC [44988] DEBUG: shmem_exit(1): 0 > before_shmem_exit callbacks to make > 2026-01-08 04:06:07.172 UTC [44988] DEBUG: shmem_exit(1): 0 on_shmem_exit > callbacks to make > 2026-01-08 04:06:07.172 UTC [44988] DEBUG: proc_exit(1): 0 callbacks to > make > 2026-01-08 04:06:07.172 UTC [44988] DEBUG: exit(1) > stopped waiting > pg_ctl: could not start server > Examine the log output. > > > From the above we can see that the stderr says that "*invalid value for > parameter "io_method": "io_uring"*" > > And in the logs I see similar thing like you reported > 2026-01-08 04:05:31.438 UTC [44972] DEBUG: starting background worker > process "logical replication launcher" > 2026-01-08 04:05:31.438 UTC [44981] DEBUG: autovacuum launcher started > 2026-01-08 04:05:31.438 UTC [44981] DEBUG: InitPostgres > 2026-01-08 04:05:31.438 UTC [44982] DEBUG: logical replication launcher > started > 2026-01-08 04:05:31.438 UTC [44982] DEBUG: InitPostgres > 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op > invalid|target invalid|state HANDED_OUT : adding cb #1, id > 2/aio_shared_buffer_readv_cb > 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op > invalid|target smgr|state HANDED_OUT : adding cb #2, id > 1/aio_md_readv_cb > 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op readv|target > smgr|state DEFINED : calling cb #1 > 2/aio_shared_buffer_readv_cb->stage(0) > 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op readv|target > smgr|state STAGED : staged (synchronous: 0, in_batch: 1) > 2026-01-08 04:05:31.439 UTC [44974] DEBUG: io 6400 |op readv|target > smgr|state COMPLETED_IO : after shared completion: distilled result: > (status OK, id 0, error_data: 0, result 1), raw_result: 8192 > 2026-01-08 04:05:31.439 UTC [44981] DEBUG: io 6400 |op readv|target > smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id > 0, error_data 0, result 1), raw_result: 8192 > > I am working on a patch and will fix the inconsistencies in documentation > about io_uring. > > Best, > Surya Poondla > > On Wed, Jan 7, 2026 at 7:54 PM PG Bug reporting form < > noreply@postgresql.org> wrote: > >> The following bug has been logged on the website: >> >> Bug reference: 19369 >> Logged by: Evan Si >> Email address: evsi@amazon.com >> PostgreSQL version: 18.1 >> Operating system: Amazon Linux 2 >> Description: >> >> Hello, >> >> The documentation doesn't explicitly state a minimum kernel version for >> io_uring, but in src/backend/storage/aio/README.md, it claims that: >> >> > io_method=io_uring is available on Linux 5.1+. >> >> Attempting to set io_method to io_uring on versions without io_uring at >> all >> (below 5.1) causes the server to graciously report that "[the k]ernel does >> not support io_uring". >> But setting io_uring on kernel versions between 5.1 and 5.6 sees the >> server >> start but connections failing with EINVAL. >> >> ``` >> [ec2-user@ip-172-31-56-49 postgres]$ psql postgres >> psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: >> FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument >> ``` >> >> It seems like the implementation in postgres leverages non-vectored I/O >> (i.e. io_uring_prep_read and io_uring_prep_write) in >> src/backend/storage/aio/method_io_uring.c. >> >> But, these opcodes aren't actually available until kernel 5.6 >> (https://github.com/axboe/liburing/issues/44), >> (https://man7.org/linux/man-pages/man2/io_uring_enter.2.html - under >> IORING_OP_READ, IORING_OP_WRITE). >> >> It seems kernel 5.4 LTS ended last month, but this seems worth >> clarifying, I >> think. >> >> -- >> >> Precise reproduction example: >> >> OS details >> ``` >> [ec2-user@ip-172-31-56-49 postgres]$ uname -a >> Linux ip-172-31-56-49.us-west-2.compute.internal >> 5.4.301-221.450.amzn2.x86_64 #1 SMP Tue Nov 18 16:40:04 UTC 2025 x86_64 >> x86_64 x86_64 GNU/Linux >> ``` >> >> Build postgres with liburing 2.8, set postgresql.conf with >> io_method=io_uring, log_min_messages = debug3. >> >> Postgres starts without complaint, but trying to connect will fail: >> >> ``` >> [ec2-user@ip-172-31-56-49 postgres]$ psql postgres >> psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: >> FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument >> ``` >> Inside logfile: >> ``` >> 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target >> invalid|state HANDED_OUT : adding cb #1, id >> 2/aio_shared_buffer_readv_cb >> 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target >> smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state DEFINED : calling cb #1 >> 2/aio_shared_buffer_readv_cb->stage(0) >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state STAGED : staged (synchronous: 0, in_batch: 1) >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state SUBMITTED : wait_one io_gen: 3, ref_gen: 3, cycle 0 >> 2026-01-02 22:35:43.863 UTC [8517] LOG: could not read blocks 0..0 in >> file >> "global/1262": Invalid argument >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state COMPLETED_IO : after shared completion: distilled result: >> (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state COMPLETED_SHARED: after local completion: result: (status >> ERROR, >> id 1, error_data 22, result 0), raw_result: -22 >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op invalid|target >> invalid|state IDLE : wait_one io_gen: 4, ref_gen: 3, cycle 0 >> 2026-01-02 22:35:43.863 UTC [8517] DEBUG: wait_one with 0 sleeps >> 2026-01-02 22:35:43.863 UTC [8517] ERROR: could not read blocks 0..0 in >> file "global/1262": Invalid argument >> 2026-01-02 22:35:44.057 UTC [8512] DEBUG: assigned pm child slot 1 for >> backend >> 2026-01-02 22:35:44.058 UTC [8512] DEBUG: forked new client backend, >> pid=8521 socket=151 >> 2026-01-02 22:35:44.058 UTC [8521] DEBUG: InitPostgres >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target >> invalid|state HANDED_OUT : adding cb #1, id >> 2/aio_shared_buffer_readv_cb >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target >> smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target >> smgr|state DEFINED : calling cb #1 >> 2/aio_shared_buffer_readv_cb->stage(0) >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target >> smgr|state STAGED : staged (synchronous: 0, in_batch: 1) >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target >> smgr|state SUBMITTED : wait_one io_gen: 1, ref_gen: 1, cycle 0 >> 2026-01-02 22:35:44.059 UTC [8521] LOG: could not read blocks 0..0 in >> file >> "global/1260": Invalid argument >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target >> smgr|state COMPLETED_IO : after shared completion: distilled result: >> (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target >> smgr|state COMPLETED_SHARED: after local completion: result: (status >> ERROR, >> id 1, error_data 22, result 0), raw_result: -22 >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: drained 1/1, now expecting 0 >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target >> invalid|state IDLE : wait_one io_gen: 2, ref_gen: 1, cycle 0 >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: wait_one with 0 sleeps >> 2026-01-02 22:35:44.059 UTC [8521] FATAL: could not read blocks 0..0 in >> file "global/1260": Invalid argument >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 5 >> before_shmem_exit callbacks to make >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 7 on_shmem_exit >> callbacks to make >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(1): 2 callbacks to >> make >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: exit(1) >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 >> before_shmem_exit callbacks to make >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 on_shmem_exit >> callbacks to make >> 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(-1): 0 callbacks to >> make >> 2026-01-02 22:35:44.060 UTC [8512] DEBUG: releasing pm child slot 1 >> 2026-01-02 22:35:44.060 UTC [8512] DEBUG: client backend (PID 8521) >> exited >> with exit code 1 >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target >> invalid|state HANDED_OUT : adding cb #1, id >> 2/aio_shared_buffer_readv_cb >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target >> smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state DEFINED : calling cb #1 >> 2/aio_shared_buffer_readv_cb->stage(0) >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state STAGED : staged (synchronous: 0, in_batch: 1) >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state SUBMITTED : wait_one io_gen: 4, ref_gen: 4, cycle 0 >> 2026-01-02 22:35:44.863 UTC [8517] LOG: could not read blocks 0..0 in >> file >> "global/1262": Invalid argument >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state COMPLETED_IO : after shared completion: distilled result: >> (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target >> smgr|state COMPLETED_SHARED: after local completion: result: (status >> ERROR, >> id 1, error_data 22, result 0), raw_result: -22 >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target >> invalid|state IDLE : wait_one io_gen: 5, ref_gen: 4, cycle 0 >> 2026-01-02 22:35:44.863 UTC [8517] DEBUG: wait_one with 0 sleeps >> 2026-01-02 22:35:44.863 UTC [8517] ERROR: could not read blocks 0..0 in >> file "global/1262": Invalid argument >> ``` >> >>
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
Si, Evan <evsi@amazon.com> — 2026-01-12T21:16:48Z
Hi Surya, Sure, thanks for the interest. Mainly I think it’d be nice to for the engine to behave similarly as it does pre-5.1 and point out that the kernel version is too old. Or at least for the README to reflect the minimum kernel version (5.6) accurately. Evan From: surya poondla <suryapoondla4@gmail.com> Date: Wednesday, January 7, 2026 at 10:53 PM To: "Si, Evan" <evsi@amazon.com>, "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org> Subject: RE: [EXTERNAL] BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi Evan, In my previous email/findings, I didn't compile postgres with the liburing option. I did it this time and see the same symptoms as you saw. surya@surya:~/pgdata18$ uname -r 5.4.0-216-generic surya@surya:~/pgdata18$ uname -a Linux surya 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:55:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux surya@surya:~/pgdata18$ surya@surya:~/pgdata18$ ~/pg18/bin/pg_ctl -D ~/pgdata18 restart waiting for server to shut down.... done server stopped waiting for server to start....2026-01-08 06:20:36.301 UTC [17042] LOG: redirecting log output to logging collector process 2026-01-08 06:20:36.301 UTC [17042] HINT: Future log output will appear in directory "log". done server started surya@surya:~/pgdata18$ ls log/ postgresql-2026-01-08_061910.log postgresql-2026-01-08_062036.log postgresql-2026-01-08_061955.log surya@surya:~/pgdata18$ ~/pg18/bin/psql -d postgres psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument In the logs, I see 026-01-08 06:24:29.858 UTC [17126] DEBUG: InitPostgres 2026-01-08 06:24:29.858 UTC [17127] DEBUG: logical replication launcher started 2026-01-08 06:24:29.858 UTC [17127] DEBUG: InitPostgres 2026-01-08 06:24:29.858 UTC [17126] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-08 06:24:29.858 UTC [17126] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-08 06:24:29.858 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 1, ref_gen: 1, cycle 0 2026-01-08 06:24:29.859 UTC [17126] LOG: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-08 06:24:29.859 UTC [17126] DEBUG: drained 1/1, now expecting 0 2026-01-08 06:24:29.859 UTC [17126] DEBUG: io 6400 |op invalid|target invalid|state IDLE : wait_one io_gen: 2, ref_gen: 1, cycle 0 2026-01-08 06:24:29.859 UTC [17126] DEBUG: wait_one with 0 sleeps 2026-01-08 06:24:29.859 UTC [17126] ERROR: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-08 06:24:30.859 UTC [17126] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cbhe logs I will do some more testing on this and update my findings. -Surya Poondla On Wed, Jan 7, 2026 at 8:15 PM surya poondla <suryapoondla4@gmail.com<mailto:suryapoondla4@gmail.com>> wrote: Hi Evsi, I created a new Ubuntu server and am able to reproduce the same behavior you reported for Postgres 18.1 on aarch64 architecture. surya@surya:~/pgdata18/log$ uname -r 5.4.0-216-generic surya@surya:~/pgdata18/log$ uname -a Linux surya 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:55:34 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux I did some testing and see the below In the postgresql.conf if I put io_method = 'worker' or io_method = 'sync', postgres is able to restart with no errors. But if io_method = 'io_uring' then I see the below on my console surya@surya:~/postgresql-18.1$ vim ~/pgdata18/postgresql.conf surya@surya:~/postgresql-18.1$ ~/pg18/bin/pg_ctl -D ~/pgdata18 restart waiting for server to shut down....2026-01-08 04:06:07.081 UTC [44973] DEBUG: logger shutting down 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(0): 0 before_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(0): 0 on_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: proc_exit(0): 0 callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: exit(0) 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-01-08 04:06:07.081 UTC [44973] DEBUG: proc_exit(-1): 0 callbacks to make done server stopped waiting for server to start....2026-01-08 04:06:07.172 GMT [44988] LOG: invalid value for parameter "io_method": "io_uring" 2026-01-08 04:06:07.172 GMT [44988] HINT: Available values: sync, worker. 2026-01-08 04:06:07.172 UTC [44988] FATAL: configuration file "/home/surya/pgdata18/postgresql.conf" contains errors 2026-01-08 04:06:07.172 UTC [44988] DEBUG: shmem_exit(1): 0 before_shmem_exit callbacks to make 2026-01-08 04:06:07.172 UTC [44988] DEBUG: shmem_exit(1): 0 on_shmem_exit callbacks to make 2026-01-08 04:06:07.172 UTC [44988] DEBUG: proc_exit(1): 0 callbacks to make 2026-01-08 04:06:07.172 UTC [44988] DEBUG: exit(1) stopped waiting pg_ctl: could not start server Examine the log output. From the above we can see that the stderr says that "invalid value for parameter "io_method": "io_uring"" And in the logs I see similar thing like you reported 2026-01-08 04:05:31.438 UTC [44972] DEBUG: starting background worker process "logical replication launcher" 2026-01-08 04:05:31.438 UTC [44981] DEBUG: autovacuum launcher started 2026-01-08 04:05:31.438 UTC [44981] DEBUG: InitPostgres 2026-01-08 04:05:31.438 UTC [44982] DEBUG: logical replication launcher started 2026-01-08 04:05:31.438 UTC [44982] DEBUG: InitPostgres 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-08 04:05:31.438 UTC [44981] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-08 04:05:31.439 UTC [44974] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status OK, id 0, error_data: 0, result 1), raw_result: 8192 2026-01-08 04:05:31.439 UTC [44981] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status OK, id 0, error_data 0, result 1), raw_result: 8192 I am working on a patch and will fix the inconsistencies in documentation about io_uring. Best, Surya Poondla On Wed, Jan 7, 2026 at 7:54 PM PG Bug reporting form <noreply@postgresql.org<mailto:noreply@postgresql.org>> wrote: The following bug has been logged on the website: Bug reference: 19369 Logged by: Evan Si Email address: evsi@amazon.com<mailto:evsi@amazon.com> PostgreSQL version: 18.1 Operating system: Amazon Linux 2 Description: Hello, The documentation doesn't explicitly state a minimum kernel version for io_uring, but in src/backend/storage/aio/README.md, it claims that: > io_method=io_uring is available on Linux 5.1+. Attempting to set io_method to io_uring on versions without io_uring at all (below 5.1) causes the server to graciously report that "[the k]ernel does not support io_uring". But setting io_uring on kernel versions between 5.1 and 5.6 sees the server start but connections failing with EINVAL. ``` [ec2-user@ip-172-31-56-49 postgres]$ psql postgres psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument ``` It seems like the implementation in postgres leverages non-vectored I/O (i.e. io_uring_prep_read and io_uring_prep_write) in src/backend/storage/aio/method_io_uring.c. But, these opcodes aren't actually available until kernel 5.6 (https://github.com/axboe/liburing/issues/44), (https://man7.org/linux/man-pages/man2/io_uring_enter.2.html - under IORING_OP_READ, IORING_OP_WRITE). It seems kernel 5.4 LTS ended last month, but this seems worth clarifying, I think. -- Precise reproduction example: OS details ``` [ec2-user@ip-172-31-56-49 postgres]$ uname -a Linux ip-172-31-56-49.us-west-2.compute.internal 5.4.301-221.450.amzn2.x86_64 #1 SMP Tue Nov 18 16:40:04 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux ``` Build postgres with liburing 2.8, set postgresql.conf with io_method=io_uring, log_min_messages = debug3. Postgres starts without complaint, but trying to connect will fail: ``` [ec2-user@ip-172-31-56-49 postgres]$ psql postgres psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument ``` Inside logfile: ``` 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-02 22:35:43.862 UTC [8517] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 3, ref_gen: 3, cycle 0 2026-01-02 22:35:43.863 UTC [8517] LOG: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-02 22:35:43.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 2026-01-02 22:35:43.863 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state IDLE : wait_one io_gen: 4, ref_gen: 3, cycle 0 2026-01-02 22:35:43.863 UTC [8517] DEBUG: wait_one with 0 sleeps 2026-01-02 22:35:43.863 UTC [8517] ERROR: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-02 22:35:44.057 UTC [8512] DEBUG: assigned pm child slot 1 for backend 2026-01-02 22:35:44.058 UTC [8512] DEBUG: forked new client backend, pid=8521 socket=151 2026-01-02 22:35:44.058 UTC [8521] DEBUG: InitPostgres 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 1, ref_gen: 1, cycle 0 2026-01-02 22:35:44.059 UTC [8521] LOG: could not read blocks 0..0 in file "global/1260": Invalid argument 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-02 22:35:44.059 UTC [8521] DEBUG: drained 1/1, now expecting 0 2026-01-02 22:35:44.059 UTC [8521] DEBUG: io 0 |op invalid|target invalid|state IDLE : wait_one io_gen: 2, ref_gen: 1, cycle 0 2026-01-02 22:35:44.059 UTC [8521] DEBUG: wait_one with 0 sleeps 2026-01-02 22:35:44.059 UTC [8521] FATAL: could not read blocks 0..0 in file "global/1260": Invalid argument 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 5 before_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(1): 7 on_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(1): 2 callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: exit(1) 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 before_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: shmem_exit(-1): 0 on_shmem_exit callbacks to make 2026-01-02 22:35:44.059 UTC [8521] DEBUG: proc_exit(-1): 0 callbacks to make 2026-01-02 22:35:44.060 UTC [8512] DEBUG: releasing pm child slot 1 2026-01-02 22:35:44.060 UTC [8512] DEBUG: client backend (PID 8521) exited with exit code 1 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state HANDED_OUT : adding cb #1, id 2/aio_shared_buffer_readv_cb 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target smgr|state HANDED_OUT : adding cb #2, id 1/aio_md_readv_cb 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state DEFINED : calling cb #1 2/aio_shared_buffer_readv_cb->stage(0) 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state STAGED : staged (synchronous: 0, in_batch: 1) 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state SUBMITTED : wait_one io_gen: 4, ref_gen: 4, cycle 0 2026-01-02 22:35:44.863 UTC [8517] LOG: could not read blocks 0..0 in file "global/1262": Invalid argument 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_IO : after shared completion: distilled result: (status ERROR, id 1, error_data: 22, result 0), raw_result: -22 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op readv|target smgr|state COMPLETED_SHARED: after local completion: result: (status ERROR, id 1, error_data 22, result 0), raw_result: -22 2026-01-02 22:35:44.863 UTC [8517] DEBUG: drained 1/1, now expecting 0 2026-01-02 22:35:44.863 UTC [8517] DEBUG: io 6400 |op invalid|target invalid|state IDLE : wait_one io_gen: 5, ref_gen: 4, cycle 0 2026-01-02 22:35:44.863 UTC [8517] DEBUG: wait_one with 0 sleeps 2026-01-02 22:35:44.863 UTC [8517] ERROR: could not read blocks 0..0 in file "global/1262": Invalid argument ```
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
Andres Freund <andres@anarazel.de> — 2026-01-13T16:59:59Z
Hi, On 2026-01-12 21:16:48 +0000, Si, Evan wrote: > Hi Surya, > > Sure, thanks for the interest. Mainly I think it’d be nice to for the engine > to behave similarly as it does pre-5.1 and point out that the kernel version > is too old. Or at least for the README to reflect the minimum kernel version > (5.6) accurately. Yea, this clearly is just an oversight. I had forgotten that those operations weren't available in earlier kernel versions. While it would be possible to make things work on the older kernel, by always using the vectorized op on old kernels, I'm doubtful it's worth the increase in test matrix. Any chance you want to provide a patch? Greetings, Andres Freund
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
surya poondla <suryapoondla4@gmail.com> — 2026-01-13T19:06:20Z
Hi Evan, Andres, Here is the patch (attached). I believe this change would also be a good candidate for backpatching to previous supported versions. Regards, Surya Poondla
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
Andres Freund <andres@anarazel.de> — 2026-01-13T19:11:04Z
Hi, On 2026-01-13 11:06:20 -0800, surya poondla wrote: > Hi Evan, Andres, > > Here is the patch (attached). I think we ought to not just document the difference, but also emit a useful error message at startup... > I believe this change would also be a good candidate for backpatching to > previous supported versions. Yes, once we have something mergeable, it should definitely be added to 18 too. Greetings, Andres Freund
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
pierre.forstmann@gmail.com — 2026-01-14T18:15:17Z
Hello, Is asynchronous I/O as introduced in PG 18 also available for Windows platform and possibly for Solaris, AIX, FreeBSD platforms ? If not maybe we should also mention in Server Configuration section that asynchronous I/O configuration is only available on Linux platforms. Is there any other feature only available for specific platforms ? I have not found a documentation section mentioning such cases (except for installations https://www.postgresql.org/docs/18/installation-platform-notes.html). PF Le 13/01/2026 à 20:06, surya poondla a écrit : > > Hi Evan, Andres, > > Here is the patch (attached). > > I believe this change would also be a good candidate for backpatching > to previous supported versions. > > Regards, Surya Poondla >
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
Andres Freund <andres@anarazel.de> — 2026-01-14T18:18:36Z
Hi, On 2026-01-14 19:15:17 +0100, Pierre Forstmann wrote: > Is asynchronous I/O as introduced in PG 18 also available for Windows > platform and possibly for Solaris, AIX, FreeBSD platforms ? Yes, but only using io_method=worker. Greetings, Andres Freund
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
Laurenz Albe <laurenz.albe@cybertec.at> — 2026-01-14T20:26:46Z
On Wed, 2026-01-14 at 13:18 -0500, Andres Freund wrote: > On 2026-01-14 19:15:17 +0100, Pierre Forstmann wrote: > > Is asynchronous I/O as introduced in PG 18 also available for Windows > > platform and possibly for Solaris, AIX, FreeBSD platforms ? > > Yes, but only using io_method=worker. Well, AIX is currently unsupported... Yours, Laurenz Albe
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
surya poondla <suryapoondla4@gmail.com> — 2026-01-14T21:07:15Z
"I think we ought to not just document the difference, but also emit a useful error message at startup..." Sure Andres, I am working on a patch which emits a useful error message too.
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
pierre.forstmann@gmail.com — 2026-01-15T07:32:43Z
I have already proposed a patch in pgsql-hackers mailing lists: see https://www.postgresql.org/message-id/5c251f33-8463-4736-aef7-6873b37e48a5%40gmail.com Le 14/01/2026 à 22:07, surya poondla a écrit : > "I think we ought to not just document the difference, but also emit a > useful > error message at startup..." > Sure Andres, I am working on a patch which emits a useful error > message too.
-
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
surya poondla <suryapoondla4@gmail.com> — 2026-01-15T20:27:23Z
> > Sure Andres, I am working on a patch which emits a useful error message > too. > Hi All, I prepared a patch that implements an early detection of unsupported io_uring operations during PostgreSQL startup, before any I/O is attempted. The patch focuses on 1. Clear error message at startup instead of cryptic EINVAL during queries. 2. Immediate failure with actionable hints (upgrade kernel or use io_method=worker) 3. Prevents PostgreSQL from starting in a broken state Before the patch: 1. PostgreSQL started successfully 2. Connection attempts failed with EINVAL errors After patch: 1. PostgreSQL refuses to start 2. Clear error message that looks like: "FATAL: kernel does not support required io_uring operations" "DETAIL: The kernel supports io_uring but lacks one or more of the required opcodes (IORING_OP_READ, IORING_OP_WRITE, IORING_OP_READV, IORING_OP_WRITEV). This typically occurs on Linux kernels older than 5.6." "HINT: Either upgrade your kernel to version 5.6 or newer, or use io_method=worker" Modified files in the Patch: 1. configure.ac: Added io_uring_opcode_supported to AC_CHECK_FUNCS 2. meson.build: Added corresponding function check for Meson build 3. src/backend/storage/aio/method_io_uring.c: Added is_uring_read_write_unsupported() function, that is integrated into pgaio_uring_init() and reports clear error with details and hints. I tested the patch on Ubuntu server with Linux kernel 5.4.0-216-generic, and when io_uring is enabled I see that postgres doesn’t start (expected behavior). The existing error handling for kernels < 5.1 (ENOSYS) is preserved. Regards, Surya Poondla -
Re: BUG #19369: Not documented that io_uring on kernel versions between 5.1 and below 5.6 does not work
surya poondla <suryapoondla4@gmail.com> — 2026-01-23T18:22:38Z
Hi All, I created a commitfest entry for this patch ( https://commitfest.postgresql.org/patch/6428/) Kindly review it and let me know of any changes. Regards, Surya Poondla >