touchpages_vs_numa_inquiry_results.txt
text/plain
Filename: touchpages_vs_numa_inquiry_results.txt
Type: text/plain
Part: 0
IMPACT OF pg_numa_touch_mem_if_required() from pg_numa.h: so we are touching like ~16k * 8kB buffers =~ 128MB (close to pgbench_accounts size):
postgres=# \dt+ pgbench_accounts
List of tables
Schema | Name | Type | Owner | Persistence | Access method | Size | Description
--------+------------------+-------+----------+-------------+---------------+--------+-------------
public | pgbench_accounts | table | postgres | permanent | heap | 128 MB |
pgbench -i -s 10 ~~ 128MB size, and "select relfilenode, count(*) from pg_buffercache where relfilenode is not null group by relfilenode order by 2 desc limit 3;" shows:
relfilenode | count
-------------+-------
28345 | 16402 // <-- pgbench_accounts
1249 | 46
2619 | 44
EXPECTED RESULT: at least 16k+ buffers split into multiple NUMA zones
patch(-touchpages) hugepages=off INVALID RESULTS (-2)
postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;'
[..]
done in 2.26 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.40 s, vacuum 0.18 s, primary keys 0.54 s).
huge_pages_status
-------------------
off
(1 row)
numa_zone_id | count
--------------+--------
| 507568
0 | 98
2 | 98
-2 | 16524
patch(-touchpages) hugepages=on INVALID RESULTS (-2)
postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;'
[..]
done in 2.54 s (drop tables 0.20 s, create tables 0.02 s, client-side generate 1.57 s, vacuum 0.21 s, primary keys 0.55 s).
huge_pages_status
-------------------
on
(1 row)
numa_zone_id | count
--------------+--------
| 507490
0 | 256
2 | 256
-2 | 15774
1 | 512
patch(touchpages) hugepages=off CORRECT RESULT
postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;'
[..]
done in 2.42 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.55 s, vacuum 0.19 s, primary keys 0.54 s).
huge_pages_status
-------------------
off
(1 row)
numa_zone_id | count
--------------+--------
| 507589
0 | 8349
2 | 8350
^^ it's interesting (different split without huge with huge pages)
patch(touchpages) hugepages=on CORRECT RESULT
postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;' -c '\l+'
[..]
done in 2.21 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.41 s, vacuum 0.19 s, primary keys 0.47 s).
huge_pages_status
-------------------
on
(1 row)
numa_zone_id | count
--------------+--------
| 507602
3 | 4096
0 | 4352
2 | 4096
1 | 4142
^^ it's interesting (different split without huge with huge pages)
patch(-touchpages)+MAP_POPULATE hugepages=off INVALID RESULTS (-2)
postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;' -c '\l+'
[..]
done in 2.25 s (drop tables 0.13 s, create tables 0.01 s, client-side generate 1.45 s, vacuum 0.19 s, primary keys 0.48 s).
huge_pages_status
-------------------
off
(1 row)
numa_zone_id | count
--------------+--------
| 507594
3 | 98
-2 | 16498
1 | 98
(4 rows)
patch(-touchpages)+MAP_POPULATE hugepages=on INVALID RESULTS (-2)
postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;' -c '\l+'
[..]
done in 4.89 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.50 s, vacuum 1.32 s, primary keys 1.93 s).
huge_pages_status
-------------------
on
(1 row)
numa_zone_id | count
--------------+--------
| 507596
0 | 256
2 | 256
-2 | 15616
1 | 564
(5 rows)
cross-check that MAP_POPULATE was used above:
postgres@jw-test3:~$ strace -ffe mmap numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart 2>&1 | grep MAP_SHARED
mmap(NULL, 27028, PROT_READ, MAP_SHARED, 3, 0) = 0x7f954aaa1000
[pid 15686] mmap(NULL, 27028, PROT_READ, MAP_SHARED, 3, 0) = 0x7fecd3d46000
[pid 15686] mmap(NULL, 4454350848, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS|MAP_POPULATE|MAP_HUGETLB, -1, 0) = 0x7febc7200000
postgres@jw-test3:~$ uname -r
6.10.11+bpo-cloud-amd64