Re: Shutdown indefinitely stuck due to unflushed FPI_FOR_HINT record
Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
From: Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
To: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2026-02-24T09:46:51Z
Lists: pgsql-hackers
Attachments
- v2-0001-Fix-stuck-shutdown-due-to-unflushed-records.patch (application/octet-stream) patch v2-0001
Some additional informations:
XLogSendLogical is stuck in the following infinite loop:
- It attempt to read the next record with XLogReadAhead + XLogDecodeNextRecord
- The page with the record header is read
- It has the record header, it goes back to XLogDecodeNextRecord
- tot_len > len, the record needs to be reassembled
- The next page containing the rest of the record is read with
ReadPageInternal. It fails since this page was never written.
- It jumps to the err label, XLogReaderInvalReadState(state) is called
and reset the reader state
- It goes back to the start of WalSndLoop's loop
There are some attempts done by the walsender to flush the WAL using
XLogBackgroundFlush:
/*
* If we're shutting down, trigger pending WAL to be written out,
* otherwise we'd possibly end up waiting for WAL that never gets
* written, because walwriter has shut down already.
*/
if (got_STOPPING)
XLogBackgroundFlush();
However, XLogBackgroundFlush only writes completed blocks or the
latest async xact known. With the issue triggered, I have the
following state:
XLogCtl->LogwrtRqst: (Write = 39128056, Flush = 39124992)
LogwrtResult: (Write = 39124992, Flush = 39124992)
XLogCtl->asyncXactLSN: 39119776
There are 3064 bytes (39128056 - 39124992) that contain the next page
with the rest of the cont record that still needs to be written.
However, XLogBackgroundFlush backs off to the previous page boundary:
/* back off to last completed page boundary */
WriteRqst.Write -= WriteRqst.Write % XLOG_BLCKSZ;
Meaning WriteRqst.Write will be 39124992, which is already written and
flushed and asyncXactLSN is behind both write and flush.
So, it looks like the root issue is more that the async LSN isn't
updated when a transaction without xid is rollbacked.
When going through CommitTransaction, such a transaction would still
go through XLogSetAsyncXactLSN.
I've updated the patch with this new approach: XLogSetAsyncXactLSN is
now called in RecordTransactionAbort even when a xid wasn't assigned.
With this, the logical walsender is able to force the flush of the
last partial page using XLogBackgroundFlush.
Commits
-
Fix WAL flush LSN used by logical walsender during shutdown
- f15471464650 14.23 landed
- fa9f2e317562 15.18 landed
- da21ecf5798c 16.14 landed
- 8ee536c89517 17.10 landed
- 9804981386a0 18.4 landed
- d927b4bd97c8 19 (unreleased) landed
-
Use GetXLogInsertEndRecPtr in gistGetFakeLSN
- b1f14c967202 19 (unreleased) cited
-
Fix publisher shutdown hang caused by logical walsender busy loop.
- 3bf6f22ce151 14.23 landed
- 42734f296615 15.18 landed
- 82935467a033 16.14 landed
- bbbc0888b356 17.10 landed
- 3eb2fecdbbfc 18.4 landed
- 6eedb2a5fd88 19 (unreleased) landed