Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Wed, Jul 09, 2025 at 08:20:31AM +0900, Michael Paquier wrote:
> Sure, you could do that as well, but I suspect that we'll need the
> steps of at least up to 0003 to be able to handle more easily multiple
> external TOAST pointer types, or the code will be messier than it
> currently is.  :D

Please find attached a v3, that I have spent some time polishing to
fix the value ID problem of this thread.  v2 had some conflicts, and
the CI previously failed with warning job (CI is green here now).

There are two things that have changed in this patch series:
- Addition of separate patch to rename varatt_external to
varatt_external_oid and VARTAG_ONDISK to VARTAG_ONDISK_OID, in 0003.
- In the refactoring to introduce the translation layer for external
ondisk TOAST pointers, aka 0004 in this set labelled v3, I was not
happy about the way we grab the vartag_external that gets assigned to
the TOAST varlena, and it depends on two factors in this patch set:
-- The type of TOAST table.
-- The value assigned.  If, for example, we have a value less than 4
billion, we can make the TOAST pointer shorter.  The interface of 0004
has been rethought to take that into consideration.  That's the
function called toast_external_assign_vartag(), reused on heaptoast.c
to get TOAST_POINTER_SIZE as the compressibility threshold when
inserting a tuple and if it should be compressed.

As things stand, I am getting pretty happy with the patch set up to
0005 and how things are getting in shape for the interface, and I am
planning to begin applying this stuff up to 0005 in the next couple of
weeks.

This stuff introduces all the callbacks and the concept of
toast_external to get the refactoring into the tree first, which is
something that other patches that want to extend the compression
methods or the TID layer are going to need anyway.  The rest of the
patch set is more a group shot, where they actually matter only when
the new vartag_external is added for the TOAST tables able to hold
8-byte values.  As of now, we would need to add more one more
vartag_external:
- For values lower than 4 billion, perhaps we could just reuse the OID
vartag_external here?  The code is modular now with
toast_external_assign_vartag(), and it is only a matter of using 
VARTAG_ONDISK_OID if we have a value less than 4 billion.  So there is
only one place in the code to change, and that's a one-line change
with the v3 patch set attached.  And that's less duplication.
- The second one when we have more than 4 billion values.
--
Michael