Thread

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hannu Krosing <hannuk@google.com> — 2025-07-21T00:15:27Z

I have been evolving details for Direct TOAST design in
https://wiki.postgresql.org/wiki/DirectTOAST

The top level goals are

* 8-byte TOAST pointer - just (header:1, tag:1 and TID:6)
* all other info moved from toast pointer to actual toast record(s),
so heap rows are smaller and faster.
* all extra fields are bytea with internal encoding (maybe will create
full new types for these, or maybe just introspection functions are
enough)
  the reasons for this are
  - PostgresSQL arrays add 20 byte overhead
  - bytea gives other freedoms in encoding for minimal space usage

No solution yet for va_toastrelid , but hope is
- to use some kind of mapping and find one or two free bits somewhere
(tid has one free),
- or add a 12-byte toast pointer just for this.
- or to make sure that CLUSTER and VACUUM FULL can be done without
needing va_toastrelid. I assume it is there for clustering the TOAST
which will be not possible separately from the main heap with direct
toast tid pointers anyway.

Please take a look and poke holes in it !

On Sun, Jul 20, 2025 at 10:28 AM Nikhil Kumar Veldanda
<veldanda.nikhilkumar17@gmail.com> wrote:
>
> Hi,
>
> > v26-0014-Design-to-extend-the-varattrib_4b-and-toast-poin.patch:
> > Design proposal covering varattrib_4b, TOAST pointer layouts, and
> > related macro updates.
> > v26-0015-Implement-Zstd-compression-no-dictionary-support.patch: Plain
> > ZSTD (non dict) support and few basic tests.
>
> Sending v27 patch with a small update over v26 patch.
>
> v27-0014-Design-to-extend-the-varattrib_4b-and-toast-poin.patch:
> Design proposal covering varattrib_4b, TOAST pointer layouts, and
> related macro updates.
> v27-0015-Implement-Zstd-compression-no-dictionary-support.patch: Plain
> ZSTD (non dict) support and few basic tests.
>
> > --
> > Nikhil Veldanda
>
> --
> Nikhil Veldanda