Horton

Re: Proposal for Updating CRC32C with AVX-512 Algorithm.

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: John Naylor <johncnaylorls@gmail.com>
Cc: Bruce Momjian <bruce@momjian.us>, Alvaro Herrera <alvherre@alvh.no-ip.org>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, Nathan Bossart <nathandbossart@gmail.com>, "Shankaran, Akash" <akash.shankaran@intel.com>, "Devulapalli, Raghuveer" <raghuveer.devulapalli@intel.com>
Date: 2024-12-12T15:45:29Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Compute CRC32C using AVX-512 instructions where available

Hi,

On 2024-12-12 18:32:20 +0700, John Naylor wrote:
> I went and looked at the Chromium source, and found the following
> snippet that uses the same technique, but only requires 128-bit CLMUL
> and has a minimum input size of 64 bytes, rather than 256. This seems
> like it might be better suited for shorter inputs. Also seems much
> easier than trying to get the AVX-512 hippo to dance. It uses the IEEE
> polynomial, so would need new constants calculated for ours, but that
> had to be done for the shared patch, too.

Frankly, we should just move away from using CRCs. They're good for cases
where short runs of bit flips are much more likely than other kinds of errors
and where the amount of data covered by them has a low upper bound. That's not
at all the case for WAL records. It'd not matter too much if CRCs were cheap
to compute - but they aren't.  We should instead move to some more generic
hashing algorithm, decent ones are much faster.

Greetings,

Andres