Problem with pg_notify / listen

Mikael Gustavsson <mikael.gustavsson@smhi.se>

From: Gustavsson Mikael <mikael.gustavsson@smhi.se>

To: pgsql-generallists.postgresql.org <pgsql-general@lists.postgresql.org>

Cc: Svensson Peter <peter.svensson@smhi.se>, Almen Anders <anders.almen@smhi.se>

Date: 2020-11-27T12:56:53Z

Lists: pgsql-bugs

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix a recently-introduced race condition in LISTEN/NOTIFY handling.
- f6324bbbe63c 10.16 landed
- f5de090cc175 13.2 landed
- cbc7a7a10c2d 12.6 landed
- 9c83b54a9ccd 14.0 landed
- 8a4069766079 9.6.21 landed
- 60d6c71430f9 9.5.25 landed
- 40f2fbe71ad6 11.11 landed
Prevent concurrent SimpleLruTruncate() for any given SLRU.
- d4031d78460c 11.10 cited

Hi.

After applying the latest patch we have encountered a problem with the pg_notify queue.

The queue is filling up and starts issuing warnings like
WARNING:  NOTIFY queue is 87% full
DETAIL:  The server process with PID 2969993 is among those with the oldest transactions.
NOTIFY queue cannot be emptied until that process ends its current transaction.

There is no long-running 'active' or 'idle in transaction' transactions on the server.

We restarted the application with the said pid but the result was that the new process pid appeared in the logs instead.
We tried to stop everything except postgresql it self but the queue was not purged. After restarting postgresql the queue is empty and seems to be holding up for an hour or so before it starts growing again.

We have been using pg_notify / listen for about three years and haven’t run in to this problem before so it seems to me that something happened in the latest patch.

PG version: 11.10 with slony. The notify triggers is on the master node.

KR.
Mikael Gustavsson
SMHI
Swedish Meteorological and Hydrological Institute