Horton

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

Matthias van de Meent <boekewurm+postgres@gmail.com>

View thread View on postgr.es
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>, Andres Freund <andres@anarazel.de>, Ian Lawrence Barwick <barwick@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, David Zhang <david.zhang@highgo.ca>, Robert Haas <robertmhaas@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2023-04-08T14:24:35Z
Lists: pgsql-hackers 
On Fri, 7 Apr 2023 at 08:05, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Apr 07, 2023 at 08:59:22AM +0900, Michael Paquier wrote:
> > Okay, cool!
>
> Done this one with 8fcb32d.

Thanks a lot! I'll post the separation of record construction and
write-out to xlog in a future thread for 17.

One remaining question: Considering that the changes and checks of
that commit are mostly internal to xloginsert.c (or xlog.c in older
releases), and that no special public-facing changes were made, would
it be safe to backport this to older releases?

PostgreSQL 15 specifically would benefit from this as it supports
external rmgrs which may generate WAL records and would benefit from
these additional checks, but all supported releases of PostgreSQL have
pg_logical_emit_message and are thus easily subject to the issue of
writing oversized WAL records and subsequent recovery- and replication
stream failures.

Kind regards,

Matthias van de Meent

Commits

  1. Add more protections in WAL record APIs against overflows

  2. Add overflow protection for block-related data in WAL records

  3. Change internal RelFileNode references to RelFileNumber or RelFileLocator.

  4. Revamp the WAL record format.