Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup
Jelte Fennema-Nio <postgres@jeltef.nl>
From: Jelte Fennema-Nio <postgres@jeltef.nl>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: Andres Freund <andres@anarazel.de>
Date: 2025-02-11T18:52:34Z
Lists: pgsql-hackers
Attachments
- v1-0001-Bump-soft-open-file-limit-RLIMIT_NOFILE-to-hard-l.patch (text/x-patch) patch v1-0001
The default open file limit of 1024 is extremely low, given modern resources and kernel architectures. The reason that this hasn't changed change is because doing so would break legacy programs that use the select(2) system call in hard to debug ways. So instead programs that want to opt-in to a higher open file limit are expected to bump their soft limit to their hard limit on startup. Details on this are very well explained in a blogpost by the systemd author[1]. There's also a similar change done by the Go language[2]. So this starts bumping postmaster and pgbench its soft open file limit to the hard open file limit. Doing so is especially useful for the AIO work that Andres is doing, because io_uring consumes a lot of file descriptors. But even without looking at AIO there is a large number of reports from people that require changing their soft file limit before starting postgres, sometimes falling back to lowering max_files_per_process when they fail to do so[3-8]. It's also not all that strange to fail at setting the soft open file limit because there are multiple places where one can configure such limits and usually only one of them is effective (which one depends on how Postgres is started). [1]: https://0pointer.net/blog/file-descriptor-limits.html [2]: https://github.com/golang/go/issues/46279 [3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres [4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie [5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com [6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com [7]: https://github.com/abiosoft/colima/discussions/836 [8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
Commits
-
Redefine max_files_per_process to control additionally opened files
- adb5f85fa5a0 18.0 landed
-
pgbench: Increase RLIMIT_NOFILE if necessary
- d38bab5edd60 18.0 landed
-
Move extra code out of the Pre/PostRestoreCommand() section.
- 8fb13dd6ab5b 17.0 cited