Horton

Re: Enable data checksums by default

Jeff Davis <pgsql@j-davis.com>

View thread
From: Jeff Davis <pgsql@j-davis.com>
To: Tomas Vondra <tomas@vondra.me>, Greg Burd <greg@burd.me>, Daniel Gustafsson <daniel@yesql.se>, Andres Freund <andres@anarazel.de>
Cc: Laurenz Albe <laurenz.albe@cybertec.at>, Peter Eisentraut <peter@eisentraut.org>, Heikki Linnakangas <hlinnaka@iki.fi>, Nathan Bossart <nathandbossart@gmail.com>, Greg Sabino Mullane <htamfids@gmail.com>, Bruce Momjian <bruce@momjian.us>, Michael Paquier <michael@paquier.xyz>, Michael Banck <mbanck@gmx.net>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2025-07-31T22:10:30Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Avoid BufferGetLSNAtomic() calls during nbtree scans.

  2. doc PG 18 relnotes: Add incompatibility note about checksums now default

  3. Fix pg_upgrade's cross-version tests when old < 18

  4. initdb: Change default to using data checksums.

  5. Allow TAP tests to force checksums off when calling init()

  6. initdb: Add new option "--no-data-checksums"

  7. Tweak docs to reduce possible impact of data checksums

On Thu, 2025-07-31 at 17:21 +0200, Tomas Vondra wrote:
> On 7/31/25 15:39, Greg Burd wrote:
> > I recall a conversation at the last PGConf.dev (2025) with a
> > representative
> > from Intel and Jeff Davis (CC’ed) that had to do with checksums and
> > a vast
> > performance difference between Intel and AMD the latter winning by
> > a mile.
> 
> I don't know the Intel vs. AMD situation exactly, but e.g. [1] does
> not
> suggest AMD wins by a mile. In fact, it suggests Intel does much
> better
> in this particular benchmark (with AVX-512 improvements). Of course,
> this is a fairly recent *kernel* improvement, maybe it wouldn't work
> for
> our data checksums that well.
> 
> However, I don't think the cost of the checksum calculation itself is
> the main concern. It's probably negligible compared to all the other
> costs, triggered by checksums - having to WAL-log hint bits, doing
> more
> expensive checks (that's what the btree regression was about), etc.

The issue Greg and I discussed, explained to me earlier by Andres, was
a memory bandwidth issue.

IIRC (Andres please correct me): The new IO infrastructure enables us
to bypass a memory copy (from userspace to kernel space) when writing
out a page. Unfortunately, checksums require reading the data to
calculate the checksum, which effectively defeats that optimization.

Those memory copies mostly happen in the bgwriter, where the page isn't
generally in the cache, which means that memory bandwidth can become
the bottleneck. Intel seems to have poor per-core memory bandwidth
compared with AMD:

https://sites.utexas.edu/jdm4372/2023/04/25/the-evolution-of-single-core-bandwidth-in-multicore-processors/

so it's more likely to become the bottleneck on Intel.

That lead to an interesting discussion about calculating the checksum
on a page in the backend eagerly when it dirties a page, while it's
still in cache. As you point out, that's quite cheap.

Regards,
	Jeff Davis