Re: [BUG] Panic due to incorrect missingContrecPtr after promotion
Sami Imseih <simseih@amazon.com>
From: "Imseih (AWS), Sami" <simseih@amazon.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>, "alvherre@alvh.no-ip.org" <alvherre@alvh.no-ip.org>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2022-05-26T19:57:41Z
Lists: pgsql-hackers
We see another occurrence of this bug with the last patch applied in 13.7.
After a promotion we observe the following in the logs:
2022-05-25 00:35:38 UTC::@:[371]:PANIC: xlog flush request 10/B1FA3D88 is not satisfied --- flushed only to 7/A8000060
2022-05-25 00:35:38 UTC:172.31.26.238(38610):administrator@postgres:[23433]:ERROR: current transaction is aborted, commands ignored until end of transaction block
However, The logs do not show "LOG: successfully skipped missing contrecord",
therefore we know that VerifyOverwriteContrecord
is not being called to invalidate the missingContrecPtr.
VerifyOverwriteContrecord(xl_overwrite_contrecord *xlrec, XLogReaderState *state)
{
if (xlrec->overwritten_lsn != state->overwrittenRecPtr)
elog(FATAL, "mismatching overwritten LSN %X/%X -> %X/%X",
(uint32) (xlrec->overwritten_lsn >> 32),
(uint32) xlrec->overwritten_lsn,
(uint32) (state->overwrittenRecPtr >> 32),
(uint32) state->overwrittenRecPtr);
/* We have safely skipped the aborted record */
abortedRecPtr = InvalidXLogRecPtr;
missingContrecPtr = InvalidXLogRecPtr;
ereport(LOG,
(errmsg("successfully skipped missing contrecord at %X/%X, overwritten at %s",
(uint32) (xlrec->overwritten_lsn >> 32),
(uint32) xlrec->overwritten_lsn,
timestamptz_to_str(xlrec->overwrite_time))));
We think it's because VerifyOverwriteContrecord was not
called which is why we see this behavior.
Are there are other places where missingContrecPtr
should be invalidated, such as after a successful promotion?
--
Sami Imseih
Amazon Web Services
Commits
-
Prevent WAL corruption after a standby promotion.
- 0b3ff539dd34 10.23 landed
- 002fba80ebde 11.18 landed
- 3c0ef0832748 12.13 landed
- 3f2701cda590 13.9 landed
- 0e54a5e27494 14.6 landed
- eb29fa38895c 15.0 landed
- 6672d7913929 16.0 landed
-
Fix "missing continuation record" after standby promotion
- caaeb88ff7ae 14.3 landed
- c714ebd0e8c1 12.11 landed
- 9d92582abf91 15.0 landed
- 9ade3c09a5d1 10.21 landed
- 98eb3e06ce3a 13.7 landed
- 199cd7b59a16 11.16 landed