Horton

Re: Race between KeepFileRestoredFromArchive() and restartpoint

Kyotaro Horiguchi <horikyota.ntt@gmail.com>

View thread View on postgr.es
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: don@seiler.us
Cc: david@pgmasters.net, noah@leadboat.com, pgsql-hackers@postgresql.org
Date: 2022-08-03T02:24:17Z
Lists: pgsql-hackers 
At Tue, 2 Aug 2022 16:03:42 -0500, Don Seiler <don@seiler.us> wrote in 
> On Tue, Aug 2, 2022 at 10:01 AM David Steele <david@pgmasters.net> wrote:
> 
> >
> > > That makes sense.  Each iteration of the restartpoint recycle loop has a
> > 1/N
> > > chance of failing.  Recovery adds >N files between restartpoints.
> > Hence, the
> > > WAL directory grows without bound.  Is that roughly the theory in mind?
> >
> > Yes, though you have formulated it better than I had in my mind.

I'm not sure I understand it correctly, but isn't the cause of the
issue in the other thread due to skipping many checkpoint records
within the checkpoint_timeout?  I remember that I proposed a GUC
variable to disable that checkpoint skipping.  As another measure for
that issue, we could force replaying checkpoint if max_wal_size is
already filled up or known to be filled in the next checkpoint cycle.
If this is correct, this patch is irrelevant to the issue.

> > Let's see if Don can confirm that he is seeing the "could not link file"
> > messages.
> 
> 
> During my latest incident, there was only one occurrence:
> 
> could not link file “pg_wal/xlogtemp.18799" to
> > “pg_wal/000000010000D45300000010”: File exists

(I noticed that the patch in the other thread is broken:()

Hmm.  It seems like a race condition betwen StartupXLOG() and
RemoveXlogFIle(). We need wider extent of ContolFileLock. Concretely
taking ControlFileLock before deciding the target xlog file name in
RemoveXlogFile() seems to prevent this happening. (If this is correct
this is a live issue on the master branch.)

> WAL restore/recovery seemed to continue on just fine then. And it would
> continue on until the pg_wal volume ran out of space unless I was manually
> rm'ing already-recovered WAL files from the side.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Skip WAL recycling and preallocation during archive recovery.

  2. Don't ERROR on PreallocXlogFiles() race condition.

  3. Remove XLogFileInit() ability to unlink a pre-existing file.

  4. In XLogFileInit(), fix *use_existent postcondition to suit callers.

  5. Remove XLogFileInit() ability to skip ControlFileLock.

  6. Add HINT for restartpoint race with KeepFileRestoredFromArchive().

  7. Complete TODO item: