walsender-cannot-exit.txt

text/plain

Filename: walsender-cannot-exit.txt
Type: text/plain
Part: 0
Message: Re: Why is src/test/modules/committs/t/002_standby.pl flaky?
With pg_usleep(10000L) added in pqReadData() you can see several tests
failed when running make check-world/meson test.
For example:
meson test recovery/019_replslot_limit 

results in:
[16:25:18.510](6.771s) ok 16 - check if checkpoint command is not blocked
### Stopping node "primary2" using mode fast
# Running: pg_ctl -D .../build/testrun/recovery/019_replslot_limit/data/t_019_replslot_limit_primary2_data/pgdata -m fast stop
waiting for server to shut down............................................................... failed
pg_ctl: server does not shut down
# pg_ctl stop failed: 256
# Postmaster PID for node "primary2" is 209253
[16:26:18.567](60.057s) Bail out!  pg_ctl stop failed

As I understand, it happens because WalSndLoop() performs this in a loop:
...
			if (got_SIGUSR2)
				WalSndDone(send_data);
...
			/* Sleep until something happens or we time out */
			WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_MAIN);
...

where WalSndDone() does this:
	replicatedPtr = XLogRecPtrIsInvalid(MyWalSnd->flush) ?
		MyWalSnd->write : MyWalSnd->flush;
	if (WalSndCaughtUp && sentPtr == replicatedPtr &&
		!pq_is_send_pending())
	{
...
		proc_exit(0);
	}
	if (!waiting_for_ping_response)
		WalSndKeepalive(true, InvalidXLogRecPtr);

(WalSndWait's call chain is:
WalSndWait -> WaitEventSetWait -> WaitEventSetWaitBlock -> epoll_wait)

So, WalSndDone waits here for replicatedPtr to become equal to sentPtr.
In other words, walsender sends keepalive to walreceiver and waits for a
new flush position from it.
And the standby loop is as follows:
WalReceiverMain:
					for (;;)
					{
						if (len > 0)
						{
...
							XLogWalRcvProcessMsg(buf[0], &buf[1], len - 1,
												 startpointTLI);
...
						}
						else if (len == 0)
							break;
					}
...
					XLogWalRcvFlush(false, startpointTLI);

where XLogWalRcvProcessMsg() calls XLogWalRcvSendReply() if a keepalive
message received.

So if walreceiver constantly receives some messages (keepalive messages in
this case), it just can't get out of the loop and perform XLogWalRcvFlush()
/move Flush position what primary is waiting for.

But if something delays walsender for more than 10 ms between epoll_wait()
and WalSndDone() on a next iteration, standby manages to get len = 0 and to
perform Flush.