touchpages_vs_numa_inquiry_results.txt

text/plain

Filename: touchpages_vs_numa_inquiry_results.txt
Type: text/plain
Part: 0
Message: Re: Draft for basic NUMA observability
IMPACT OF pg_numa_touch_mem_if_required() from pg_numa.h: so we are touching like ~16k * 8kB buffers =~ 128MB (close to pgbench_accounts size):
	postgres=# \dt+ pgbench_accounts
											  List of tables
	 Schema |       Name       | Type  |  Owner   | Persistence | Access method |  Size  | Description
	--------+------------------+-------+----------+-------------+---------------+--------+-------------
	 public | pgbench_accounts | table | postgres | permanent   | heap          | 128 MB |

	pgbench -i -s 10 ~~ 128MB size, and "select relfilenode, count(*) from pg_buffercache where relfilenode is not null group by relfilenode  order by 2 desc limit 3;" shows:
	 relfilenode | count
	-------------+-------
		   28345 | 16402 // <-- pgbench_accounts
			1249 |    46
			2619 |    44

EXPECTED RESULT: at least 16k+ buffers split into multiple NUMA zones


patch(-touchpages)               hugepages=off		INVALID RESULTS (-2)

	postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench  -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;'
[..]
	done in 2.26 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.40 s, vacuum 0.18 s, primary keys 0.54 s).
	 huge_pages_status
	-------------------
	 off
	(1 row)

	 numa_zone_id | count
	--------------+--------
				  | 507568
				0 |     98
				2 |     98
			   -2 |  16524


patch(-touchpages)               hugepages=on		INVALID RESULTS (-2)

	postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench  -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;'
[..]
	done in 2.54 s (drop tables 0.20 s, create tables 0.02 s, client-side generate 1.57 s, vacuum 0.21 s, primary keys 0.55 s).
	 huge_pages_status
	-------------------
	 on
	(1 row)

	 numa_zone_id | count
	--------------+--------
				  | 507490
				0 |    256
				2 |    256
			   -2 |  15774
				1 |    512

patch(touchpages)                hugepages=off		CORRECT RESULT 
 
	postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench  -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;'
[..]
	done in 2.42 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.55 s, vacuum 0.19 s, primary keys 0.54 s).
	 huge_pages_status
	-------------------
	 off
	(1 row)

	 numa_zone_id | count
	--------------+--------
				  | 507589
				0 |   8349
				2 |   8350
    ^^ it's interesting (different split without huge with huge pages)

patch(touchpages)                hugepages=on		CORRECT RESULT

	postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench  -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;' -c '\l+'
[..]
	done in 2.21 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.41 s, vacuum 0.19 s, primary keys 0.47 s).
	 huge_pages_status
	-------------------
	 on
	(1 row)

	 numa_zone_id | count
	--------------+--------
				  | 507602
				3 |   4096
				0 |   4352
				2 |   4096
				1 |   4142
	^^ it's interesting (different split without huge with huge pages)
 

patch(-touchpages)+MAP_POPULATE   hugepages=off		INVALID RESULTS (-2)

	postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench  -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;' -c '\l+'
[..]
	done in 2.25 s (drop tables 0.13 s, create tables 0.01 s, client-side generate 1.45 s, vacuum 0.19 s, primary keys 0.48 s).
	 huge_pages_status
	-------------------
	 off
	(1 row)

	 numa_zone_id | count
	--------------+--------
				  | 507594
				3 |     98
			   -2 |  16498
				1 |     98
	(4 rows)


patch(-touchpages)+MAP_POPULATE   hugepages=on		INVALID RESULTS (-2)

	postgres@jw-test3:~$ numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart ; /usr/pgsql18numa/bin/pgbench  -i -s 10; /usr/pgsql18numa/bin/psql -c 'show huge_pages_status;' -c 'select numa_zone_id, count(*) from pg_buffercache_numa group by numa_zone_id;' -c '\l+'
[..]
	done in 4.89 s (drop tables 0.12 s, create tables 0.01 s, client-side generate 1.50 s, vacuum 1.32 s, primary keys 1.93 s).
	 huge_pages_status
	-------------------
	 on
	(1 row)

	 numa_zone_id | count
	--------------+--------
				  | 507596
				0 |    256
				2 |    256
			   -2 |  15616
				1 |    564
	(5 rows)


	cross-check that MAP_POPULATE was used above:
		postgres@jw-test3:~$ strace -ffe mmap numactl --interleave=all /usr/pgsql18numa/bin/pg_ctl -c -D /tmp/numa -l logfile restart 2>&1 | grep MAP_SHARED
		mmap(NULL, 27028, PROT_READ, MAP_SHARED, 3, 0) = 0x7f954aaa1000
		[pid 15686] mmap(NULL, 27028, PROT_READ, MAP_SHARED, 3, 0) = 0x7fecd3d46000
		[pid 15686] mmap(NULL, 4454350848, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS|MAP_POPULATE|MAP_HUGETLB, -1, 0) = 0x7febc7200000

		postgres@jw-test3:~$ uname -r
		6.10.11+bpo-cloud-amd64