RE: Proposal for Updating CRC32C with AVX-512 Algorithm.

Amonson, Paul D <paul.d.amonson@intel.com>

From: "Amonson, Paul D" <paul.d.amonson@intel.com>

To: Andres Freund <andres@anarazel.de>

Cc: "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, Nathan Bossart <nathandbossart@gmail.com>, "Shankaran, Akash" <akash.shankaran@intel.com>

Date: 2024-06-17T22:42:54Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Compute CRC32C using AVX-512 instructions where available
- 3c6e8c123896 18.0 landed

> This is extremely workload dependent, it's not hard to find workloads with
> lots of very small record and very few big ones...  What you observed might
> have "just" been the warmup behaviour where more full page writes have to
> be written.

Can you tell me how to avoid capturing this "warm-up" so that the numbers are more accurate?
 
> There a very frequent call computing COMP_CRC32C over just 20 bytes, while
> holding a crucial lock.  If we were to do introduce something like this
> AVX-512 algorithm, it'd probably be worth to dispatch differently in case of
> compile-time known small lengths.

So are you suggesting that we be able to directly call into the 64/32 bit based algorithm directly from these known small byte cases in the code? I think that we can do that with a separate API being exposed.

> How does the latency of the AVX-512 algorithm compare to just using the
> CRC32C instruction?

I think I need more information on this one as I am not sure I understand the use case? The same function pointer indirect methods are used with or without the AVX-512 algorithm?

Paul