Re: Proposal for Updating CRC32C with AVX-512 Algorithm.

Nathan Bossart <nathandbossart@gmail.com>

From: Nathan Bossart <nathandbossart@gmail.com>

To: Andres Freund <andres@anarazel.de>

Cc: John Naylor <johncnaylorls@gmail.com>, Bruce Momjian <bruce@momjian.us>, Alvaro Herrera <alvherre@alvh.no-ip.org>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, "Shankaran, Akash" <akash.shankaran@intel.com>, "Devulapalli, Raghuveer" <raghuveer.devulapalli@intel.com>

Date: 2024-12-12T22:14:33Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Compute CRC32C using AVX-512 instructions where available
- 3c6e8c123896 18.0 landed

On Thu, Dec 12, 2024 at 10:45:29AM -0500, Andres Freund wrote:
> Frankly, we should just move away from using CRCs. They're good for cases
> where short runs of bit flips are much more likely than other kinds of errors
> and where the amount of data covered by them has a low upper bound. That's not
> at all the case for WAL records. It'd not matter too much if CRCs were cheap
> to compute - but they aren't.  We should instead move to some more generic
> hashing algorithm, decent ones are much faster.

Upthread [0], I wondered aloud about trying to reuse the page checksum code
for this.  IIRC there was a lot of focus on performance when that was
added, and IME it catches problems decently well.

[0] https://postgr.es/m/ZrUcX2kq-0doNBea%40nathan

-- 
nathan