Re: better page-level checksums

Aleksander Alekseev <aleksander@timescale.com>

From: Aleksander Alekseev <aleksander@timescale.com>

To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Cc: Robert Haas <robertmhaas@gmail.com>, Stephen Frost <sfrost@snowman.net>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Peter Geoghegan <pg@bowt.ie>, Andrey Borodin <x4mmm@yandex-team.ru>

Date: 2022-06-13T13:23:18Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Rethink method for assigning OIDs to the template0 and postgres DBs.
- 2cb1272445d2 15.0 landed
pg_upgrade: Preserve database OIDs.
- aa01051418f1 15.0 landed
pg_upgrade: Preserve relfilenodes and tablespace OIDs.
- 9a974cbcba00 15.0 landed
Fix for new Boolean node
- cf925936ecc0 15.0 cited
Improve error handling of HMAC computations
- 5513dc6a304d 15.0 cited
Add macro RelationIsPermanent() to report relation permanence
- 95d77149c535 14.0 landed
Enhance nbtree index tuple deletion.
- d168b666823b 14.0 cited

Hi hackers,

> > Can't we add some extra fork that stores this extra per-page
> > information, and contains this extra metadata
> >
> +1 for this approach. I had observed some painful corruption cases where block storage simply returned stale version of a rage of blocks. This is only possible because checksum is stored on the page itself.

That's very interesting, Andrey. Thanks for sharing.

> One of my questions is what algorithm(s) we'd want to support.

Should it necessarily be a fixed list? Why not support plugable algorithms?

An extension implementing a checksum algorithm is going to need:

- several hooks: check_page_after_reading, calc_checksum_before_writing
- register_checksum()/deregister_checksum()
- an API to save the checksums to a seperate fork

By knowing the block number and the hash size the extension knows
exactly where to look for the checksum in the fork.

-- 
Best regards,
Aleksander Alekseev