Horton

Re: POC: make mxidoff 64 bits

Maxim Orlov <orlovmg@gmail.com>

View thread
From: Maxim Orlov <orlovmg@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: wenhui qiu <qiuwenhuifx@gmail.com>, Alexander Korotkov <aekorotkov@gmail.com>, Postgres hackers <pgsql-hackers@lists.postgresql.org>
Date: 2024-11-19T17:53:49Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix partial read handling in pg_upgrade's multixact conversion

  2. Increase timeout in multixid_conversion upgrade test

  3. Improve sanity checks on multixid members length

  4. Clarify comment on multixid offset wraparound check

  5. Never store 0 as the nextMXact

  6. Add runtime checks for bogus multixact offsets

  7. Widen MultiXactOffset to 64 bits

  8. Move pg_multixact SLRU page format definitions to a separate header

  9. Convert confusing macros in multixact.c to static inline functions

  10. Index SLRUs by 64-bit integers rather than by 32-bit integers

  11. Cope with possible failure of the oldest MultiXact to exist.

Attachments

Oops! Sorry for the noise. I've must have been overworking yesterday and
messed up the working branches. v7 was a correct set and v8 don't. Here is
the correction with extended Perl test.

The test itself is in src/bin/pg_upgrade/t/005_offset.pl It is rather heavy
and took about 45 minutes on my i5 with 2.7 Gb data generated. Basically,
each test here is creating a cluster and fill it with multixacts. Thus,
dozens of segments are created using two methods. One is with prepared
transactions, and it creates, roughly, the same amount of segments for
members and for offsets. The other one is based on Heikki's multixids.py
and creates more members than offsets. I've used both of these methods to
generate as much diverse data as possible.

Here is how I test this patch set:

   1. You need two pg clusters: the "old" one, i.e. without patch set, and
   the "new" with patch set v9 applied.
   2. Apply v9-0005-TEST-initdb-option-to-initialize-cluster-with-non.patch.txt
   to the "old" and "new" clusters. Note, this is only patch required for
   "old" cluster. This will allow you to create a cluster with non-standard
   initial multixact and multixact offset. Unfortunately, this patch was not
   did not arouse public interest since it is assumed that there is similar
   functionality to the pg_resetwal utility. But similar is not mean equal.
   See, pg_resetwal must be used after cluster init, thus, we step into some
   problems with vacuum and some SLRU segments must be filled with zeroes.
   Also, template0 datminmxid must be manually updated. So, in me view,
   using this patch is justified and very handy here.
   3. Also, apply all the "TEST" (0006 and 0007) patches to the "new"
   cluster.
   4. Build "old" and "new" pg clusters.
   5. Run the test with: PROVE_TESTS=t/005_offset.pl PG_TEST_NOCLEAN=1
   oldinstall=/home/orlov/proj/OFFSET3/pgsql-old make check -s -C
   src/bin/pg_upgrade/
   6. In my case, it took around 45 minutes and generate roughly 2.7 Gb of
   data.

"TEST" patches, of course, are for the test purposes and not to be
committed.

In src/bin/pg_upgrade/t/005_offset.pl I try to consider next cases:

   - Basic sanity checks.
   Here I test various initial multi and offset values (including
   wraparound) and see how appropriate segments are generated.
   - pg_upgarde tests.
   Here is oldinstall ENV is for. Run pg_upgrade for old cluster with multi
   and offset values just like in previous step. i.e. with various
   combinations.
   - Self pg_upgarde.


-- 
Best regards,
Maxim Orlov.