Horton

Re: Optimize LISTEN/NOTIFY

Joel Jacobson <joel@compiler.org>

View thread
From: "Joel Jacobson" <joel@compiler.org>
To: "Chao Li" <li.evan.chao@gmail.com>
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>, "Thomas Munro" <thomas.munro@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, "Heikki Linnakangas" <hlinnaka@iki.fi>, "Rishu Bagga" <rishu.postgres@gmail.com>
Date: 2025-09-28T10:24:06Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Optimize LISTEN/NOTIFY via shared channel map and direct advancement.

  2. Fix incorrect logic for caching ResultRelInfos for triggers

On Fri, Sep 26, 2025, at 11:44, Chao Li wrote:
>> On Sep 26, 2025, at 17:32, Joel Jacobson <joel@compiler.org> wrote:
>> 
>> On Fri, Sep 26, 2025, at 04:26, Chao Li wrote:
>> 
>>> I think what you explained is partially correct.
>>> 
>>> Based on my understanding, any backend process may call 
>>> SignalBackends(), which means that it’s possible that multiple backend 
>>> processes may call SignalBackends() concurrently.
>>> 
>>> Looking at your code, between checking 
>>> QUEUE_BACKEND_WAKEUP_PENDING_FLAG(i) and set the flag to true, there is 
>>> a block of code (the “if-else”) to run, so that it’s possible that 
>>> multiple backend processes have passed the 
>>> QUEUE_BACKEND_WAKEUP_PENDING_FLAG(i) check, then multiple signals will 
>>> be sent to a process, which will lead to duplicate timeout enabled in 
>>> the receiver process.
>> 
>> I don't see how that can happen; we're checking wakeup_pending_flag
>> while holding an exclusive lock, so I don't see how multiple backend
>> processes could be within the region where we check/set
>> wakeup_pending_flag, at the same time?
>> 
>> /Joel
>
> I might miss the factor of holding an exclusive lock. I will revisit 
> that part again.

I've re-read this entire thread, and I actually think my original
approaches are more promising, that is, the
0001-optimize_listen_notify-v4.patch patch, doing multicast targeted
signaling.

Therefore, merely consider the latest patch as PoC with some possible
interesting ideas.

Before this patch, I had never used PostgreSQL's timeout mechanism
before, so I didn't consider it when thinking about how to solve the
remaining problems with 0001-optimize_listen_notify-v4.patch, which
currently can't guarantee that all listening backends will eventually
catch up, since it just kicks one of the most lagging ones, for each
notification. This could be a problem in practise if there is a long
period of time with no notifications coming in. Then some listening
backends could end up not being signaled and would stay behind,
preventing the queue tail from advancing.

I'm thinking maybe somehow we can use the timeout mechanism here, but
I'm not sure how yet. Any ideas?

/Joel