Horton

Re: [BUG] Panic due to incorrect missingContrecPtr after promotion

Kyotaro Horiguchi <horikyota.ntt@gmail.com>

View thread View on postgr.es
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: simseih@amazon.com
Cc: michael@paquier.xyz, alvherre@alvh.no-ip.org, pgsql-hackers@postgresql.org
Date: 2022-08-10T08:37:18Z
Lists: pgsql-hackers 
Hello.

> Yes, that is correct.

Mmm. I believed that the log came from a single server run, since the
PID (I believe the [359], [357] are PID) did not change through the
log lines.

> 2022-08-05 18:50:13 UTC::@:[359]:LOG:  creating missing WAL directory "pg_wal/archive_status"

This means that someone removes the content of pg_wal directory.
Removing some WAL files in pg_wal may lead to this symptom.

> 2022-08-05 18:50:13 UTC::@:[359]:LOG:  entering standby mode
> recovering 00000002.history
> 2022-08-05 18:50:13 UTC::@:[359]:LOG:  ### [S] @6/B8000198: abort=(0/0)0/0, miss=(0/0)0/0, SbyMode=0, SbyModeReq=1
> 2022-08-05 18:50:13 UTC::@:[359]:LOG:  database system was not properly shut down; automatic recovery in progress
> 2022-08-05 18:50:13 UTC::@:[359]:LOG:  redo starts at 6/B80000E8
> 
> And a few hours later, is when we see a panic

So, it seems that the *standby* received the inconsistent WAL stream
(aborted-contrecord not followed by a overwriting-missing-contrecord)
from the primary.  Thus the inconsistency happened on the primary, not
on the standby.

So... I'm still failing to draw the big picutre of what is happening
here.

Could you show us the server configuration (dbservers involved and
their roles (primary/standby..)), and the exact steps when you restart
the server after carsh?

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Prevent WAL corruption after a standby promotion.

  2. Fix "missing continuation record" after standby promotion