Thread

Support for 8-byte TOAST values, round two

Michael Paquier <michael@paquier.xyz> — 2026-05-08T06:07:13Z

Hi all,

This is a follow-up of the previous thread about $subject, with a
reworked patch set for discussion in v20, as I care about the subject:
https://www.postgresql.org/message-id/aFOnKHG7Wn-Srnpv@paquier.xyz

The main feedback of the previous thread is that the previous
implementation with its callbacks for each vartag was not liked much,
and their were concerns with pointer redirections and performance.
This patch set uses what I am calling the "brutal" approach, relying
on a vartag_external of a varlena or the atttype of the TOAST relation
to decide which external toast pointer we should use.  This uses no
function pointers, and patches the code to deal with Oid or Oid8 TOAST
values where it matters.  So, this time, performance cannot really be
an issue.

The patch set is structured so as all the ground work happens first
(most of it comes from the previous patch set, reorganized a bit), and
the introduction of the varatt pieces are last, based on the following
rules:
- A table can use a 8-byte TOAST value with a new reloption, named
toast_value_type that can be set to "oid" (default) or "oid8",
creating a TOAST table with a value of the assigned type.  This
includes support for dumps as well as binary upgrades, so as the
atttype of the chunk_id of the TOAST table is preserved.  A table with
a TOAST type assigned cannot be changed to a different type through a
VACUUM FULL or a rewrite, as a matter of implementation simplicity.
- Renames and cleanup of various areas related to varatt_external,
renaming things to use OID.
- The 8-byte TOAST values rely on a Oid8, whose value is retrieved
from the control file extended by 4 bytes.  The code supports
wraparound of values so as we don't assign anything between 0 and
FirstNormalObjectId for the lower bytes, same way as before.

The last patch introduces a new vartag_external and the new
varatt_external_oid8, with an Oid8 as value.  Well, not exactly, the
patch uses two uint32 fields so as the structure is packed without
padding, as of:
typedef struct varatt_external_oid8
{
   int32       va_rawsize;
   uint32      va_extinfo;
   uint32      va_valueid_lo;
   uint32      va_valueid_hi;
   Oid         va_toastrelid;
} varatt_external_oid8;

Note that if applying all the patches except the last one, the code
would use an varatt_external_oid with an oid8 TOAST table.  This
works, the split is to make reviews easier.  An oid8 TOAST table
always uses a varatt_external_oid8.

I have done a lot of back-and-forth in the patch to try to find a good
balance between the manipulation of the varlenas in the detoast and
compression paths, as well as reorderbuffer.c and amcheck.  And I have
finished with the attached, which is kind of nice.  The last patch has
a low footprint:
9 files changed, 537 insertions(+), 208 deletions(-)

Happy to discuss about all that at pgconf.dev.  This is of course
intended for v20, added to the next CF.
Thanks,
--
Michael