Horton
Menu

Re: Implement waiting for wal lsn replay: reloaded

Xuneng Zhou <xunengzhou@gmail.com>

View thread
From: Xuneng Zhou <xunengzhou@gmail.com>
To: Alexander Korotkov <aekorotkov@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>, Heikki Linnakangas <hlinnaka@iki.fi>, Peter Eisentraut <peter@eisentraut.org>, Thomas Munro <thomas.munro@gmail.com>, Álvaro Herrera <alvherre@kurilemu.de>, Chao Li <li.evan.chao@gmail.com>, pgsql-hackers <pgsql-hackers@lists.postgresql.org>, Michael Paquier <michael@paquier.xyz>, jian he <jian.universality@gmail.com>, Tomas Vondra <tomas@vondra.me>, Yura Sokolov <y.sokolov@postgrespro.ru>
Date: 2026-04-08T03:23:54Z
Lists: pgsql-hackers

Attachments

On Wed, Apr 8, 2026 at 7:23 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
>
> Hi, Xuneng!
>
> > Here is some analysis of the issue reported by Tom:
> >
> > 1) The problem
> >
> > WAIT FOR LSN with standby_write or standby_flush mode can block
> > indefinitely on an idle primary even when the target LSN is already
> > satisfied by WAL on disk.
> >
> > The walreceiver initializes its process-local LogstreamResult.Write
> > and LogstreamResult.Flush from GetXLogReplayRecPtr() at connect time,
> > reflecting all WAL already present on the standby (from a base backup,
> > archive restore, or prior streaming). The shared-memory positions used
> > by WAIT FOR LSN, however, are not seeded from this value:
> >
> > WalRcv->writtenUpto is zero-initialized by ShmemInitStruct and remains
> > 0 until XLogWalRcvWrite() processes incoming streaming data.
> > WalRcv->flushedUpto is initialized to the segment-aligned streaming
> > start point by RequestXLogStreaming(), which may be significantly
> > behind the replay position. It advances only when XLogWalRcvFlush()
> > processes new data — which itself requires LogstreamResult.Flush <
> > LogstreamResult.Write, a condition that never holds at startup since
> > both fields are initialized to the same value.
> >
> > When the primary is idle and sends no new WAL, both positions stay at
> > their initial stale values indefinitely.
> >
> > 2) The fix
> > Seed writtenUpto and flushedUpto from LogstreamResult immediately
> > after the walreceiver initializes those process-local fields, then
> > call WaitLSNWakeup() to wake any already-blocked waiters.
> >
> > This broadens the semantics of these fields. writtenUpto and
> > flushedUpto  used to track only WAL written or flushed by the current
> > walreceiver session — WAL received from the primary since the most
> > recent connect. After this change, they are initialized to the replay
> > position, so they also cover WAL that was already on disk before
> > streaming began. This affects pg_stat_wal_receiver.written_lsn and
> > flushed_lsn, which will now report the replay position immediately at
> > walreceiver startup rather than 0 and the segment boundary
> > respectively. I am still considering whether this semantic change is
> > acceptable though it does shorten the runtime of the tap tests
> > reported by Tom in my test. Another approach is to modify the logic of
> > GetCurrentLSNForWaitType to cope with this special case and leave the
> > publisher side alone without changing the semantics. But this seems to
> > be more subtle.
>
> Patch 0001 looks OK for me.
> Regarding patch 0002.  Changes made for GetCurrentLSNForWaitType()
> looks reliable for me.  PerformWalRecovery() sets replayed positions
> before starting recovery, and in turn before standby can accept
> connections.  So, changes to WalReceiverMain() don't look necessary to
> me.

Yeah, GetCurrentLSNForWaitType seems to be the right place to place
the fix. Please see the attached patch 2.

I also noticed another relevent problem:

During pure archive recovery (no walreceiver), a backend that issues
'WAIT FOR LSN ... MODE 'standby_write' with a target ahead of the
current replay position will sleep forever; the startup process
replays past the target but only wakes 'STANDBY_REPLAY' waiters.

This also affects mixed scenarios: the walreceiver may lag behind
replay (e.g., archive restore has delivered WAL faster than
streaming), so a 'standby_write' waiter could be waiting on WAL that
replay has already consumed.

I will write a patch to address this soon.

--
Best,
Xuneng

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Clean up 019_replslot_limit.pl comments

  2. Stabilize 019_replslot_limit.pl: wait on slot restart_lsn

  3. Fix memory ordering in WAIT FOR LSN wakeup mechanism

  4. Improve WAIT FOR LSN test coverage

  5. Remove redundant WAIT FOR LSN caller-side pre-checks

  6. Use barrier semantics when reading/writing writtenUpto

  7. Use replay position as floor for WAIT FOR LSN standby_(write|flush)

  8. Wake standby_write/standby_flush waiters from the WAL replay loop

  9. Minimal fix for WAIT FOR ... MODE 'standby_flush'

  10. Avoid syscache lookup while building a WAIT FOR tuple descriptor

  11. Document that WAIT FOR may be interrupted by recovery conflicts

  12. Use WAIT FOR LSN in PostgreSQL::Test::Cluster::wait_for_catchup()

  13. Wake LSN waiters before recovery target stop

  14. Remove redundant pg_unreachable() after elog(ERROR) from ExecWaitStmt()

  15. Revert "Use WAIT FOR LSN in PostgreSQL::Test::Cluster::wait_for_catchup()"

  16. Fix variable usage in wakeupWaiters()

  17. Add tab completion for the WAIT FOR LSN MODE option

  18. Add the MODE option to the WAIT FOR LSN command

  19. Extend xlogwait infrastructure with write and flush wait types

  20. Unify error messages

  21. Optimize shared memory usage for WaitLSNProcInfo

  22. Fix WaitLSNWakeup() fast-path check for InvalidXLogRecPtr

  23. Fix incorrect function name in comments

  24. Add infrastructure for efficient LSN waiting

  25. Add pairingheap_initialize() for shared memory usage

  26. Implement WAIT FOR command