Horton

Re: better page-level checksums

Peter Geoghegan <pg@bowt.ie>

View thread
From: Peter Geoghegan <pg@bowt.ie>
To: Robert Haas <robertmhaas@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-06-09T21:33:55Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Rethink method for assigning OIDs to the template0 and postgres DBs.

  2. pg_upgrade: Preserve database OIDs.

  3. pg_upgrade: Preserve relfilenodes and tablespace OIDs.

  4. Fix for new Boolean node

  5. Improve error handling of HMAC computations

  6. Add macro RelationIsPermanent() to report relation permanence

  7. Enhance nbtree index tuple deletion.

On Thu, Jun 9, 2022 at 2:13 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I'm interested in assessing the feasibility of a "better page-level
> checksums" feature. I have a few questions, and a few observations.
> One of my questions is what algorithm(s) we'd want to support. I did a
> quick Google search and found that brtfs supports CRC-32C, XXHASH,
> SHA256, and BLAKE2B. I don't know that we want to support that many
> options (but maybe we do) and I don't think CRC-32C makes any sense
> here, for two reasons. First, we've already got a 16-bit checksum, and
> a 32-bit checksum doesn't seem like it's gaining enough to be worth
> the implementation complexity.

Why not? The only problems that it won't solve are all related to
crypto. Which is perfectly fine, but it seems like there is a
terminology issue here. ISTM that you're really talking about adding a
cryptographic hash function, not a checksum. These are rather
different things.

> Even if we only offer one new kind of checksum, making space for a
> wider checksum makes the page format variable in a way that it
> currently isn't.

I believe that the page special area was designed to be
variable-sized, and even anticipates dynamic resizing of the special
area. At least in index AMs, where it's not that hard to make extra
space in the special area by shifting the tuples back, and then fixing
line pointers to point to the new offsets. So you have a dynamic
variable-sized array that's a little like a second line pointer array
(though probably not added to all that often).

My preference is for an approach that builds on that, or at least
doesn't significantly complicate it. So a cryptographic hash or nonce
can go in the special area proper (structs like BTPageOpaqueData don't
need any changes), but at a page offset before the special area proper
-- not after.

What disadvantages does that approach have, if any, from your point of view?

-- 
Peter Geoghegan