Re: [WIP]Vertical Clustered Index (columnar store extension) - take2

On 6/4/25 19:59, Jim Nasby wrote:
> 
> 
> On Fri, May 23, 2025 at 4:29 PM Tomas Vondra <tomas@vondra.me
> <mailto:tomas@vondra.me>> wrote:
> 
>     Also, Alvaro seemed to think TAM is the way to go, and in order to keep
>     the OLTP performance he suggested to use both heap and VCI at the same
>     time, in different "forks". I'm not sure how would that work, or if we
>     can already do that - AFAIK we can't, because ForkNumber does not allow
>     adding custom forks. We'd have to relax that, or invent some sort of
>     federated TAM (that just multiplexes it to two TAMs). Maybe.
> 
>     But it's not like the IAM approach doesn't need to do this. The first
>     patch had to add stuff to a lot of random places to make this work. And
>     some of the places touch stuff that we don't expect indexes to worry
>     about, like ALTER TABLE, etc.
> 
> 
> I suspect another option would be to handle this with table inheritance:
> have one child that is heap-based, a second that's VCI, and a background
> job to move data from heap to VCI (and vice-versa for updates and maybe
> deletes).
> 
> Note that you could actually implement all that in user-space.
> Personally I'd much rather have a way to do pure VCI / column-store
> sooner and manage it myself than have to wait another release (or more)
> to get a complete solution...  

I don't see how could this ever work with the optimizer, which assumes
scanning an inheritance hierarchy means scanning all parts. But this
would require making planner "smarter" to know it should scan only one
of the child relations. And I believe it's not possible to do that while
constructing scans for the heap/VCI parts, those places are not aware of
what other parts are being scanned etc.

Sure, you could do this in "user-space" by constructing queries that
reference either the heap or VCI part. But then why put that into
inheritance tree at all? It certainly does not help with moving data
between the parts.

I may be missing something, of course. But it's not clear to me how is
this supposed to work ...

What I can imagine is "VCI" as a "proxy" TAM on top of heap, keeping the
columnar format in a separate fork. And using either that from custom
scans, or the heap as a fallback for cases not supported by VCI.


regars

-- 
Tomas Vondra