Re: Use SIGTERM instead of SIGUSR1 for slotsync worker to exit during promotion?

On Tue, Mar 24, 2026 at 2:45 PM Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Tue, Mar 24, 2026 at 3:00 PM Fujii Masao <masao.fujii@gmail.com> wrote:
> >
> > On Tue, Mar 24, 2026 at 1:01 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> > > Hi Fujii-san,
> > >
> > > I tried reproducing the wait scenario as you mentioned, but could not
> > > reproduce it.
> > > Steps I followed:
> > > 1) Place a debugger in the slotsync worker and hold it at
> > > fetch_remote_slots() ... -> libpqsrv_get_result()
> > > 2) Kill the primary.
> > > 3) Triggered promotion of the standby and release debugger from slotsync worker.
> > >
> > > The slot sync worker stops when the promotion is triggered and then
> > > restarts, but fails to connect to the primary. The promotion happens
> > > immediately.
> > > ```
> > > LOG:  received promote request
> > > LOG:  redo done at 0/0301AD40 system usage: CPU: user: 0.00 s, system:
> > > 0.02 s, elapsed: 4574.89 s
> > > LOG:  last completed transaction was at log time 2026-03-23
> > > 17:13:15.782313+05:30
> > > LOG:  replication slot synchronization worker will stop because
> > > promotion is triggered
> > > LOG:  slot sync worker started
> > > ERROR:  synchronization worker "slotsync worker" could not connect to
> > > the primary server: connection to server at "127.0.0.1", port 9933
> > > failed: Connection refused
> > > Is the server running on that host and accepting TCP/IP connections?
> > > ```
> > >
> > > I’ll debug this further to understand it better.
> > > In the meantime, please let me know if I’m missing any step, or if you
> > > followed a specific setup/script to reproduce this scenario.
> >
> > Thanks for testing!
> >
> > If you killed the primary with a signal like SIGTERM, an RST packet might have
> > been sent to the slotsync worker at that moment. That allowed the worker to
> > detect the connection loss and exited the wait state, so promotion could
> > complete as expected.
> >
> > To reproduce the issue, you'll need a scenario where the worker cannot detect
> > the connection loss. For example, you could block network traffic (e.g., with
> > iptables) between the primary and the slotsync worker. The key is to create
> > a situation where the worker remains stuck waiting for input for a long time.
>
> Here's one way to reproduce the issue using iptables:
>

Thank you, Fujii-san, for sharing the steps. I am now able to
reproduce the behavior where promotion gets stuck because the slot
sync worker remains in a wait loop.

As an experiment, I tried setting tcp_user_timeout to 7000 / 15000
(using slightly higher values for debugging). With this setting, the
TCP stack terminates the connection if data sent to the primary
remains unacknowledged beyond the configured timeout (e.g., due to a
network drop). In such cases the slot sync worker exits instead of
waiting indefinitely. With an appropriately tuned timeout, this could
help avoid the promotion issue by ensuring the worker does not remain
stuck when the connection to the primary is lost.

Thanks,
Nisha