Horton

Re: POC: make mxidoff 64 bits

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Maxim Orlov <orlovmg@gmail.com>
Cc: Alexander Korotkov <aekorotkov@gmail.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>, wenhui qiu <qiuwenhuifx@gmail.com>, Postgres hackers <pgsql-hackers@lists.postgresql.org>, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Date: 2025-12-03T12:04:11Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix partial read handling in pg_upgrade's multixact conversion

  2. Increase timeout in multixid_conversion upgrade test

  3. Improve sanity checks on multixid members length

  4. Clarify comment on multixid offset wraparound check

  5. Never store 0 as the nextMXact

  6. Add runtime checks for bogus multixact offsets

  7. Widen MultiXactOffset to 64 bits

  8. Move pg_multixact SLRU page format definitions to a separate header

  9. Convert confusing macros in multixact.c to static inline functions

  10. Index SLRUs by 64-bit integers rather than by 32-bit integers

  11. Cope with possible failure of the oldest MultiXact to exist.

On 03/12/2025 11:54, Maxim Orlov wrote:
> The biggest problem with compression, in my opinion, is that losing
> even one byte causes the loss of the entire compressed block in the
> worst case scenario. After all, we still don't have checksums for the
> SLRU's, which is a shame by itself.
> 
> Again, I'm not against the idea of compression, but the risks need to
> be considered.

There are plenty of such critical bytes in the system where a single bit 
flip renders the whole block unreadable. Actually, if we had checksums 
on SLRU pages, a single bit flip anywhere in the page would make the 
checksum fail and render the block unreadable.

If things go really bad and you need to open a hex editor and try to fix 
the data manually, it shouldn't be too hard to deduce the correct base 
offset from surrounding data.

> As a software developer, I definitely want to implement compression and
> save a few gigabytes. However, given my previous experience using
> Postgres in real-world applications, reliability at the cost of several
> gigabytes would not have caused me any trouble. Just saying.

+1. If we decide to do some kind of compression here, I want it to be 
very simple. Otherwise it's just not worth the code complexity and risk.

Let's do the math of how much disk space we'd save. Let's assume the 
worst case that every multixid consists of only one transaction ID. 
Currently, every such multixid takes up 4 bytes in the offsets SLRU, and 
5 bytes in the members SLRU (one flag byte and 4 bytes for the XID). So 
that's 9 bytes. With 64-bit offsets, it becomes 13 bytes. With the 
compression, we're back to 9 bytes again (ignoring the one base offset 
per page). So in an extreme case that you have 1 billion multixids, with 
only one XID per multixid, the difference is between 9 GB and 13 GB. 
That seems acceptable.

And having just one XID per multixid is a rare corner case. Much more 
commonly, you have at at least two XIDs. With two XIDs per multixid, the 
difference is between 14 bytes and 18 bytes.

And having a billion multixids is pretty extreme. Your database is 
likely very large too if you reach that point, and a few gigabytes won't 
matter.

One could argue that the memory needed for the SLRU cache matters more 
than the disk space. That's perhaps true, but I think this is totally 
acceptable from that point of view, too.

- Heikki