Postmaster self-deadlock due to PLT linkage resolution
Tom Lane <tgl@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
To: pgsql-hackers@lists.postgresql.org
Date: 2022-08-29T19:43:55Z
Lists: pgsql-hackers
Attachments
- fix-PLT-links-before-unblocking-signals.patch (text/x-diff) patch
Buildfarm member mamba (NetBSD-current on prairiedog's former hardware) has failed repeatedly since I set it up. I have now run the cause of that to ground [1], and here's what's happening: if the postmaster receives a signal just before it first waits at the select() in ServerLoop, it can self-deadlock. During the postmaster's first use of select(), the dynamic loader needs to resolve the PLT branch table entry that the core executable uses to reach select() in libc.so, and it locks the loader's internal data structures while doing that. If we enter a signal handler while the lock is held, and the handler needs to do anything that also requires the lock, the postmaster is frozen. The probability of this happening seems remarkably small, since there's only one narrow window per postmaster lifetime, and there's just not that many potential signal causes active at that time either. Nonetheless I have traces showing it happening (1) because we receive SIGCHLD for startup process termination and (2) because we receive SIGUSR1 from the startup process telling us to start walreceivers. I guess that mamba's slow single-CPU hardware interacts with the NetBSD scheduler in just the right way to make it more probable than you'd think. On typical modern machines, the postmaster would almost certainly manage to wait before the startup process is able to signal it. Still, "almost certainly" is not "certainly". The attached patch seems to fix the problem, by forcing resolution of the PLT link before we unblock signals. It depends on the assumption that another select() call appearing within postmaster.c will share the same PLT link, which seems pretty safe. I'd originally intended to make this code "#ifdef __NetBSD__", but on looking into the FreeBSD sources I find much the same locking logic in their dynamic loader, and now I'm wondering if such behavior isn't pretty standard. The added calls should have negligible cost, so it doesn't seem unreasonable to do them everywhere. (Of course, a much better answer is to get out of the business of doing nontrivial stuff in signal handlers. But even if we get that done soon, we'd surely not back-patch it.) Thoughts? regards, tom lane [1] https://gnats.netbsd.org/56979
Commits
-
Use WaitEventSet API for postmaster's event loop.
- 7389aad63666 16.0 cited
-
On NetBSD, force dynamic symbol resolution at postmaster start.
- 5bed28e699ec 10.23 landed
- 6fd58ca7703a 11.18 landed
- 68bfe36c5173 12.13 landed
- a94b019d44e5 13.9 landed
- 464db46760d2 14.6 landed
- 529d8f3736a1 15.0 landed
- 8acd8f8690ed 16.0 landed