Horton

Re: better page-level checksums

Stephen Frost <sfrost@snowman.net>

View thread
From: Stephen Frost <sfrost@snowman.net>
To: Andrey Borodin <x4m@double.cloud>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Robert Haas <robertmhaas@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-06-10T16:13:34Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Rethink method for assigning OIDs to the template0 and postgres DBs.

  2. pg_upgrade: Preserve database OIDs.

  3. pg_upgrade: Preserve relfilenodes and tablespace OIDs.

  4. Fix for new Boolean node

  5. Improve error handling of HMAC computations

  6. Add macro RelationIsPermanent() to report relation permanence

  7. Enhance nbtree index tuple deletion.

Greetings,

* Andrey Borodin (x4m@double.cloud) wrote:
> On Fri, Jun 10, 2022 at 5:00 AM Matthias van de Meent <
> boekewurm+postgres@gmail.com> wrote:
> > Can't we add some extra fork that stores this extra per-page
> > information, and contains this extra metadata
> 
> +1 for this approach. I had observed some painful corruption cases where
> block storage simply returned stale version of a rage of blocks. This is
> only possible because checksum is stored on the page itself.
> A special fork for checksums would allow us to better detect failures in
> SSD firmawares, MMU SEUs etc, OS page cache, backup software and storage.
> It may seems that these kind of stuff never happen. But probability of such
> failure is drastically bigger than probability of hardware failure being
> undetected due to CRC16 collision.

This is another possible approach, sure, but it has its own downsides:
clearly more IO ends up being involved and then you also have to deal
with the fact that the fork's page would certainly end up covering a lot
of the pages in the main relation, not to mention the question of what
to do when we want to get checksums *on forks*, which we surely will
want to have...

> Also I'm skeptical about correcting detected errors with the information
> from checksum. This approach requires very very large checksum. It's much
> easier to obtain fresh block copy from HA standby.

Yeah, error correcting checksums are yet another use-case and one that
would require a lot more space.

Thanks,

Stephen