Re: MultiXact\SLRU buffers configuration

Thomas Munro <thomas.munro@gmail.com>

From: Thomas Munro <thomas.munro@gmail.com>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Gilles Darold <gilles@darold.net>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <aekorotkov@gmail.com>, Anastasia Lubennikova <a.lubennikova@postgrespro.ru>, Daniel Gustafsson <daniel@yesql.se>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-03-26T03:46:06Z
Lists: pgsql-hackers

Attachments

Hi Andrey, all,

I propose some changes, and I'm attaching a new version:

I renamed the GUCs as clog_buffers etc (no "_slru_").  I fixed some
copy/paste mistakes where the different GUCs were mixed up.  I made
some changes to the .conf.sample.  I rewrote the documentation so that
it states the correct unit and defaults, and refers to the
subdirectories that are cached by these buffers instead of trying to
give a new definition of each of the SLRUs.

Do you like those changes?

Some things I thought about but didn't change:

I'm not entirely sure if we should use the internal and historical
names well known to hackers (CLOG), or the visible directory names (I
mean, we could use pg_xact_buffers instead of clog_buffers).  I am not
sure why these GUCs need to be PGDLLIMPORT, but I see that NBuffers is
like that.

I wanted to do some very simple smoke testing of CLOG sizes on my
local development machine:

  pgbench -i -s1000 postgres
  pgbench -t4000000 -c8 -j8 -Mprepared postgres

I disabled autovacuum after running that just to be sure it wouldn't
interfere with my experiment:

  alter table pgbench_accounts set (autovacuum_enabled = off);

Then I shut the cluster down and made a copy, so I could do some
repeated experiments from the same initial conditions each time.  At
this point I had 30 files 0000-001E under pg_xact, holding 256kB = ~1
million transactions each.  It'd take ~960 buffers to cache it all.
So how long does VACUUM FREEZE pgbench_accounts take?

I tested with just the 0001 patch, and also with the 0002 patch
(improved version, attached):

clog_buffers=128:  0001=2:28.499, 0002=2:17:891
clog_buffers=1024: 0001=1:38.485, 0002=1:29.701

I'm sure the speedup of the 0002 patch can be amplified by increasing
the number of transactions referenced in the table OR number of
clog_buffers, considering that the linear search produces
O(transactions * clog_buffers) work.  That was 32M transactions and
8MB of CLOG, but I bet if you double both of those numbers once or
twice things start to get hot.  I don't see why you shouldn't be able
to opt to cache literally all of CLOG if you want (something like 50MB
assuming default autovacuum_freeze_max_age, scale to taste, up to
512MB for the theoretical maximum useful value).

I'm not saying the 0002 patch is bug-free yet though, it's a bit finickity.

Commits

  1. Rework new SLRU test with injection points

  2. injection_point: Add injection_points.stats

  3. injection_points: Add initialization of shmem state when loading module

  4. Add injection-point test for new multixact CV usage

  5. Use conditional variable to wait for next MultiXact offset

  6. Improve performance of subsystems on top of SLRU

  7. Adjust VACUUM hastup LP_REDIRECT comments.

  8. Test replay of regression tests, attempt II.