Horton

Re: XID formatting and SLRU refactorings

Thomas Munro <thomas.munro@gmail.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Alexander Korotkov <aekorotkov@gmail.com>, Maxim Orlov <orlovmg@gmail.com>, Aleksander Alekseev <aleksander@timescale.com>, Postgres hackers <pgsql-hackers@lists.postgresql.org>, Japin Li <japinli@hotmail.com>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Pavel Borisov <pashkin.elfe@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Date: 2023-12-17T21:14:03Z
Lists: pgsql-hackers 
On Tue, Nov 28, 2023 at 11:14 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> I think it's pretty sloppy that the "short" filenames can be 4, 5 or 6
> chars long. For pg_multixact/members, which introduced the 5-char case,
> I think we should always pad the filenames 5 characters, and for
> commit_ts which introduced the 6 char case, always pad to 6 characters.
>
> Instead of a "long_segment_names" boolean, how about an integer field,
> to specify the length.
>
> That means that we'll need pg_upgrade to copy pg_multixact/members files
> under the new names. That should be pretty straightforward.

Do you think it could be useful if the file names were not sequential
numbers ...0000, ...0001, ...0002 but instead used the 64 bit 'index'
number for the contained data?  In the cases where the index is an
fxid, such as pg_xact, pg_serial etc that seems easy to grok, and for
the multixacts or notify it's a bit more arbitrary but that's not
worse (and it is perhaps even meaningful, number of multixacts etc).
For example, pg_serial holds a uint64_t for every xid, so that's 32768
= 0x8000 xids per 256kB file, and you might see the following files on
disk:

0000000000000000
0000000000008000
0000000000010000

... so that it's very clear what fxid ranges are being held.  It might
also make the file unlinking logic more straightforward in the
non-modulo cases (not sure).  Of course you can work it out with
simple arithmetic but I wonder if human administrators who don't have
a C-level understanding of PostgreSQL would find this scheme more
cromulent when trying to understand, quickly, whether the system is
retaining expected data.

(Assuming we actually get the indexes to be 64 bit in the first place.
I started thinking/hacking around how to do that for the specific case of
pg_serial because it's [by its own admission] a complete mess right now,
and I was investigating its disk usage, see nearby thread, but then
I found my way here and realised I'm probably duplicating work that's
already been/being done so I'm trying to catch up here... forgive me if
the above was already covered, so many messages...)

Commits

  1. Fix more holes with SLRU code in need of int64 for segment numbers

  2. Use more consistently int64 for page numbers in SLRU-related code

  3. Improve comments in slru.{c,h} about segment name format

  4. Reorder actions in ProcArrayApplyRecoveryInfo()

  5. Fix typo in 5a1dfde8334b

  6. Fix warning due non-standard inline declaration in 4ed8f0913bfdb5f355

  7. Add SLRU tests for 64-bit page case

  8. Make use FullTransactionId in 2PC filenames

  9. Use larger segment file names for pg_notify

  10. Index SLRUs by 64-bit integers rather than by 32-bit integers

  11. Keep track of transaction commit timestamps

  12. Handle 5-char filenames in SlruScanDirectory