Re: MultiXact\SLRU buffers configuration
Thomas Munro <thomas.munro@gmail.com>
From: Thomas Munro <thomas.munro@gmail.com>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Gilles Darold <gilles@darold.net>,
Tomas Vondra <tomas.vondra@enterprisedb.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>,
Alexander Korotkov <aekorotkov@gmail.com>, Anastasia Lubennikova <a.lubennikova@postgrespro.ru>,
Daniel Gustafsson <daniel@yesql.se>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-03-26T03:46:06Z
Lists: pgsql-hackers
Attachments
- v12-0001-Make-all-SLRU-buffer-sizes-configurable.patch (text/x-patch) patch v12-0001
- v12-0002-Add-buffer-mapping-table-for-SLRUs.patch (text/x-patch) patch v12-0002
Hi Andrey, all, I propose some changes, and I'm attaching a new version: I renamed the GUCs as clog_buffers etc (no "_slru_"). I fixed some copy/paste mistakes where the different GUCs were mixed up. I made some changes to the .conf.sample. I rewrote the documentation so that it states the correct unit and defaults, and refers to the subdirectories that are cached by these buffers instead of trying to give a new definition of each of the SLRUs. Do you like those changes? Some things I thought about but didn't change: I'm not entirely sure if we should use the internal and historical names well known to hackers (CLOG), or the visible directory names (I mean, we could use pg_xact_buffers instead of clog_buffers). I am not sure why these GUCs need to be PGDLLIMPORT, but I see that NBuffers is like that. I wanted to do some very simple smoke testing of CLOG sizes on my local development machine: pgbench -i -s1000 postgres pgbench -t4000000 -c8 -j8 -Mprepared postgres I disabled autovacuum after running that just to be sure it wouldn't interfere with my experiment: alter table pgbench_accounts set (autovacuum_enabled = off); Then I shut the cluster down and made a copy, so I could do some repeated experiments from the same initial conditions each time. At this point I had 30 files 0000-001E under pg_xact, holding 256kB = ~1 million transactions each. It'd take ~960 buffers to cache it all. So how long does VACUUM FREEZE pgbench_accounts take? I tested with just the 0001 patch, and also with the 0002 patch (improved version, attached): clog_buffers=128: 0001=2:28.499, 0002=2:17:891 clog_buffers=1024: 0001=1:38.485, 0002=1:29.701 I'm sure the speedup of the 0002 patch can be amplified by increasing the number of transactions referenced in the table OR number of clog_buffers, considering that the linear search produces O(transactions * clog_buffers) work. That was 32M transactions and 8MB of CLOG, but I bet if you double both of those numbers once or twice things start to get hot. I don't see why you shouldn't be able to opt to cache literally all of CLOG if you want (something like 50MB assuming default autovacuum_freeze_max_age, scale to taste, up to 512MB for the theoretical maximum useful value). I'm not saying the 0002 patch is bug-free yet though, it's a bit finickity.
Commits
-
Rework new SLRU test with injection points
- 94a3373ac5c3 18.0 landed
-
injection_point: Add injection_points.stats
- 2e35c67f9568 18.0 landed
-
injection_points: Add initialization of shmem state when loading module
- b2b023aa3706 18.0 landed
-
Add injection-point test for new multixact CV usage
- 768a9fd5535f 18.0 landed
-
Use conditional variable to wait for next MultiXact offset
- a0e0fb1ba56f 17.0 landed
-
Improve performance of subsystems on top of SLRU
- 53c2a97a9266 17.0 landed
-
Adjust VACUUM hastup LP_REDIRECT comments.
- 325bc54eed4e 16.0 cited
-
Test replay of regression tests, attempt II.
- f47ed79cc8a0 15.0 cited