Re: AIO v2.0

Hi,

On 2025-01-09 20:10:24 +0200, Ants Aasma wrote:
> On Thu, 9 Jan 2025 at 18:25, Andres Freund <andres@anarazel.de> wrote:
> > > I'm curious about this because the checksum code should be fast enough
> > > to easily handle that throughput.
> >
> > It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
> > workstation. But we don't have a good ready-made way of testing that without
> > also doing IO, so it's kinda hard to say.
>
> Interesting, I wonder if it's related to Intel increasing vpmulld
> latency to 10 already back in Haswell. The Zen 3 I'm testing on has
> latency 3 and has twice the throughput.

> Attached is a naive and crude benchmark that I used for testing here.
> Compiled with:
>
> gcc -O2 -funroll-loops -ftree-vectorize -march=native \
>   -I$(pg_config --includedir-server) \
>   bench-checksums.c -o bench-checksums-native
>
> Just fills up an array of pages and checksums them, first argument is
> number of checksums, second is array size. I used 1M checksums and 100
> pages for in cache behavior and 100000 pages for in memory
> performance.
>
> 869.85927ms @ 9.418 GB/s - generic from memory
> 772.12252ms @ 10.610 GB/s - generic in cache
> 442.61869ms @ 18.508 GB/s - native from memory
> 137.07573ms @ 59.763 GB/s - native in cache

printf '%16s\t%16s\t%s\n' march mem result; for mem in 100 100000 1000000; do for march in x86-64 x86-64-v2 x86-64-v3 x86-64-v4 native; do printf "%16s\t%16s\t" $march $mem; gcc -g -g3 -O2 -funroll-loops -ftree-vectorize -march=$march -I ~/src/postgresql/src/include/ -I src/include/ /tmp/bench-checksums.c -o bench-checksums-native && numactl --physcpubind 1 --membind 0 ./bench-checksums-native 1000000 $mem;done; done

Workstation w/ 2x Xeon Gold 6442Y:

           march	             mem	result
          x86-64	             100	731.87779ms @ 11.193 GB/s
       x86-64-v2	             100	327.18580ms @ 25.038 GB/s
       x86-64-v3	             100	264.03547ms @ 31.026 GB/s
       x86-64-v4	             100	282.08065ms @ 29.041 GB/s
          native	             100	246.13766ms @ 33.282 GB/s
          x86-64	          100000	842.66827ms @ 9.722 GB/s
       x86-64-v2	          100000	604.52959ms @ 13.551 GB/s
       x86-64-v3	          100000	477.16239ms @ 17.168 GB/s
       x86-64-v4	          100000	476.07039ms @ 17.208 GB/s
          native	          100000	456.08080ms @ 17.962 GB/s
          x86-64	         1000000	845.51132ms @ 9.689 GB/s
       x86-64-v2	         1000000	612.07973ms @ 13.384 GB/s
       x86-64-v3	         1000000	485.23738ms @ 16.882 GB/s
       x86-64-v4	         1000000	483.86411ms @ 16.930 GB/s
          native	         1000000	462.88461ms @ 17.698 GB/s



Zen 4 laptop (AMD Ryzen 7 PRO 7840U):
           march	             mem	result
          x86-64	             100	417.19762ms @ 19.636 GB/s
       x86-64-v2	             100	130.67596ms @ 62.689 GB/s
       x86-64-v3	             100	97.07758ms @ 84.386 GB/s
       x86-64-v4	             100	95.67704ms @ 85.621 GB/s
          native	             100	95.15734ms @ 86.089 GB/s
          x86-64	          100000	431.38370ms @ 18.990 GB/s
       x86-64-v2	          100000	215.74856ms @ 37.970 GB/s
       x86-64-v3	          100000	199.74492ms @ 41.012 GB/s
       x86-64-v4	          100000	186.98300ms @ 43.811 GB/s
          native	          100000	187.68125ms @ 43.648 GB/s
          x86-64	         1000000	433.87893ms @ 18.881 GB/s
       x86-64-v2	         1000000	217.46561ms @ 37.670 GB/s
       x86-64-v3	         1000000	200.40667ms @ 40.877 GB/s
       x86-64-v4	         1000000	187.51978ms @ 43.686 GB/s
          native	         1000000	190.29273ms @ 43.049 GB/s


Workstation w/ 2x Xeon Gold 5215:
           march	             mem	result
          x86-64	             100	780.38881ms @ 10.497 GB/s
       x86-64-v2	             100	389.62005ms @ 21.026 GB/s
       x86-64-v3	             100	323.97294ms @ 25.286 GB/s
       x86-64-v4	             100	274.19493ms @ 29.877 GB/s
          native	             100	283.48674ms @ 28.897 GB/s
          x86-64	          100000	1112.63898ms @ 7.363 GB/s
       x86-64-v2	          100000	831.45641ms @ 9.853 GB/s
       x86-64-v3	          100000	696.20789ms @ 11.767 GB/s
       x86-64-v4	          100000	685.61636ms @ 11.948 GB/s
          native	          100000	689.78023ms @ 11.876 GB/s
          x86-64	         1000000	1128.65580ms @ 7.258 GB/s
       x86-64-v2	         1000000	843.92594ms @ 9.707 GB/s
       x86-64-v3	         1000000	718.78848ms @ 11.397 GB/s
       x86-64-v4	         1000000	687.68258ms @ 11.912 GB/s
          native	         1000000	705.34731ms @ 11.614 GB/s


That's quite the drastic difference between amd and intel. Of course it's also
comparing a multi-core server uarch (lower per-core bandwidth, much higher
aggregate bandwidth) with a client uarch.


The difference between the baseline CPU target and a more modern profile is
also rather impressive.  Looks like some cpu-capability based dispatch would
likely be worth it, even if it didn't matter in my case due to -march=native.


I just realized that

a) The meson build doesn't use the relevant flags for bufpage.c - it didn't
   matter in my numbers though because I was building with -O3 and
   march=native.

   This clearly ought to be fixed.

b) Neither build uses the optimized flags for pg_checksum and pg_upgrade, both
   of which include checksum_imp.h directly.

   This probably should be fixed too - perhaps by building the relevant code
   once as part of fe_utils or such?


It probably matters less than it used to - these days -O2 turns on
-ftree-loop-vectorize -ftree-slp-vectorize. But loop unrolling isn't
enabled.

I do see a perf difference at -O2 between using/not using
-funroll-loops. Interestingly not at -O3, despite -funroll-loops not actually
being enabled by -O3. I think the relevant option that *is* turned on by O3 is
-fpeel-loops.

Here's a comparison of different flags run the 6442Y

printf '%16s\t%32s\t%16s\t%s\n' march flags mem result; for mem in 100 100000; do for march in x86-64 x86-64-v2 x86-64-v3 x86-64-v4 native; do for flags in "-O2" "-O2 -funroll-loops" "-O3" "-O3 -funroll-loops"; do printf "%16s\t%32s\t%16s\t" "$march" "$flags" "$mem"; gcc $flags -march=$march -I ~/src/postgresql/src/include/ -I src/include/ /tmp/bench-checksums.c -o bench-checksums-native && numactl --physcpubind 3 --membind 0 ./bench-checksums-native 3000000 $mem;done; done;done
           march	                           flags	             mem	result
          x86-64	                             -O2	             100	2280.86253ms @ 10.775 GB/s
          x86-64	              -O2 -funroll-loops	             100	2195.66942ms @ 11.193 GB/s
          x86-64	                             -O3	             100	2422.57588ms @ 10.145 GB/s
          x86-64	              -O3 -funroll-loops	             100	2243.75826ms @ 10.953 GB/s
       x86-64-v2	                             -O2	             100	1243.68063ms @ 19.761 GB/s
       x86-64-v2	              -O2 -funroll-loops	             100	979.67783ms @ 25.086 GB/s
       x86-64-v2	                             -O3	             100	988.80296ms @ 24.854 GB/s
       x86-64-v2	              -O3 -funroll-loops	             100	991.31632ms @ 24.791 GB/s
       x86-64-v3	                             -O2	             100	1146.90165ms @ 21.428 GB/s
       x86-64-v3	              -O2 -funroll-loops	             100	785.81395ms @ 31.275 GB/s
       x86-64-v3	                             -O3	             100	800.53627ms @ 30.699 GB/s
       x86-64-v3	              -O3 -funroll-loops	             100	790.21230ms @ 31.101 GB/s
       x86-64-v4	                             -O2	             100	883.82916ms @ 27.806 GB/s
       x86-64-v4	              -O2 -funroll-loops	             100	831.55372ms @ 29.554 GB/s
       x86-64-v4	                             -O3	             100	843.23141ms @ 29.145 GB/s
       x86-64-v4	              -O3 -funroll-loops	             100	821.19969ms @ 29.927 GB/s
          native	                             -O2	             100	1197.41357ms @ 20.524 GB/s
          native	              -O2 -funroll-loops	             100	718.05253ms @ 34.226 GB/s
          native	                             -O3	             100	747.94090ms @ 32.858 GB/s
          native	              -O3 -funroll-loops	             100	751.52379ms @ 32.702 GB/s
          x86-64	                             -O2	          100000	2911.47087ms @ 8.441 GB/s
          x86-64	              -O2 -funroll-loops	          100000	2525.45504ms @ 9.731 GB/s
          x86-64	                             -O3	          100000	2497.42016ms @ 9.841 GB/s
          x86-64	              -O3 -funroll-loops	          100000	2346.33551ms @ 10.474 GB/s
       x86-64-v2	                             -O2	          100000	2124.10102ms @ 11.570 GB/s
       x86-64-v2	              -O2 -funroll-loops	          100000	1819.09659ms @ 13.510 GB/s
       x86-64-v2	                             -O3	          100000	1613.45823ms @ 15.232 GB/s
       x86-64-v2	              -O3 -funroll-loops	          100000	1607.09245ms @ 15.292 GB/s
       x86-64-v3	                             -O2	          100000	1972.89390ms @ 12.457 GB/s
       x86-64-v3	              -O2 -funroll-loops	          100000	1432.58229ms @ 17.155 GB/s
       x86-64-v3	                             -O3	          100000	1533.18003ms @ 16.029 GB/s
       x86-64-v3	              -O3 -funroll-loops	          100000	1539.39779ms @ 15.965 GB/s
       x86-64-v4	                             -O2	          100000	1591.96881ms @ 15.437 GB/s
       x86-64-v4	              -O2 -funroll-loops	          100000	1434.91828ms @ 17.127 GB/s
       x86-64-v4	                             -O3	          100000	1454.30133ms @ 16.899 GB/s
       x86-64-v4	              -O3 -funroll-loops	          100000	1429.13733ms @ 17.196 GB/s
          native	                             -O2	          100000	1980.53734ms @ 12.409 GB/s
          native	              -O2 -funroll-loops	          100000	1373.95337ms @ 17.887 GB/s
          native	                             -O3	          100000	1517.90164ms @ 16.191 GB/s
          native	              -O3 -funroll-loops	          100000	1508.37021ms @ 16.293 GB/s



> > > Is it just that the calculation is slow, or is it the fact that checksumming
> > > needs to bring the page into the CPU cache. Did you notice any hints which
> > > might be the case?
> >
> > I don't think the issue is that checksumming pulls the data into CPU caches
> >
> > 1) This is visible with SELECT that actually uses the data
> >
> > 2) I added prefetching to avoid any meaningful amount of cache misses and it
> >    doesn't change the overall timing much
> >
> > 3) It's visible with buffered IO, which has pulled the data into CPU caches
> >    already
>
> I didn't yet check the code, when doing aio completions checksumming
> be running on the same core as is going to be using the page?

With io_uring normally yes, the exception being that another backend that
needs the same page could end up running the completion.

With worker mode normally no.

Greetings,

Andres Freund