Horton

Re: BUG #17928: Standby fails to decode WAL on termination of primary

Tom Lane <tgl@sss.pgh.pa.us>

View thread
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>, Alexander Lakhin <exclusion@gmail.com>, Sergei Kornilov <sk@zsrv.org>, Noah Misch <noah@leadboat.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@oss.nttdata.com>, pgbf@twiska.com
Date: 2023-09-25T04:59:09Z
Lists: pgsql-bugs

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Correct assertion and comments about XLogRecordMaxSize.

  2. Fix edge-case for xl_tot_len broken by bae868ca.

  3. Don't use Perl pack('Q') in 039_end_of_wal.pl.

  4. Don't trust unvalidated xl_tot_len.

  5. Make recovery report error message when invalid page header is found.

  6. Add more protections in WAL record APIs against overflows

Thomas Munro <thomas.munro@gmail.com> writes:
> On Mon, Sep 25, 2023 at 4:07 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That one is mine, so let me know if there's something particular
>> you'd like me to check.

> Thanks!  But after spending the best part of a day installing and
> compiling stuff at snail's pace, I have just this moment managed to
> reproduce this reliably on an emulator running Debian armhf (= armv7).
> So hopefully I can track the issue down from here without bothering
> build farm owners.

FWIW, here's what I see on mamba:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  pg_comp_crc32c_sb8 (crc=3197562372, crc@entry=4294967295, 
    data=data@entry=0xfda6e030, len=<optimized out>) at pg_crc32c_sb8.c:56
56                      uint32          a = *p4++ ^ crc;
(gdb) bt
#0  pg_comp_crc32c_sb8 (crc=3197562372, crc@entry=4294967295, 
    data=data@entry=0xfda6e030, len=<optimized out>) at pg_crc32c_sb8.c:56
#1  0x0195ae5c in ValidXLogRecord (state=0xfde5cc88, record=0xfda6e018, 
    recptr=<optimized out>) at xlogreader.c:1195
#2  0x0195cf18 in XLogDecodeNextRecord (state=state@entry=0xfde5cc88, 
    nonblocking=<optimized out>) at xlogreader.c:842
#3  0x0195d0c8 in XLogReadAhead (state=0xfde5cc88, 
    nonblocking=nonblocking@entry=false) at xlogreader.c:969
#4  0x01959760 in XLogPrefetcherNextBlock (pgsr_private=4259700408, 
    lsn=0xfda6b8a0) at xlogprefetcher.c:496
#5  0x0195a72c in lrq_prefetch (lrq=<optimized out>) at xlogprefetcher.c:256
#6  lrq_complete_lsn (lsn=50520048, lrq=0xfda6b858) at xlogprefetcher.c:294
#7  XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0xfde5deb8, 
    errmsg=errmsg@entry=0xffffdd88) at xlogprefetcher.c:1041
#8  0x01960054 in ReadRecord (xlogprefetcher=0xfde5deb8, emode=emode@entry=15, 
    fetching_ckpt=fetching_ckpt@entry=false, replayTLI=replayTLI@entry=3)
    at xlogrecovery.c:3067
#9  0x01963188 in PerformWalRecovery () at xlogrecovery.c:1756
#10 0x01952594 in StartupXLOG () at xlog.c:5470
#11 0x01c26e3c in StartupProcessMain () at startup.c:267
#12 0x01c1b504 in AuxiliaryProcessMain (auxtype=auxtype@entry=StartupProcess)
    at auxprocess.c:141
#13 0x01c2276c in StartChildProcess (type=StartupProcess) at postmaster.c:5347
#14 0x01c260ec in PostmasterMain (argc=argc@entry=4, 
    argv=argv@entry=0xffffe1e4) at postmaster.c:1457
#15 0x01ee39d8 in main (argc=4, argv=0xffffe1e4) at main.c:198

(gdb) f 1
#1  0x0195ae5c in ValidXLogRecord (state=0xfde5cc88, record=0xfda6e018, 
    recptr=<optimized out>) at xlogreader.c:1195
1195            COMP_CRC32C(crc, ((char *) record) + SizeOfXLogRecord, record->xl_tot_len - SizeOfXLogRecord);
(gdb) p *record
$3 = {xl_tot_len = 0, xl_xid = 0, xl_prev = 0, xl_info = 0 '\000', 
  xl_rmid = 0 '\000', xl_crc = 0}

Sure looks like ValidXLogRecord is assuming that record->xl_tot_len
can be trusted without reservation.

			regards, tom lane