Re: AW: update on TOAST status'

Zeugswetter Andreas SB wrote:
>
> > > I don't like that --- seems it would put a definite crimp in the
> > > whole point of TOAST, which is not to have arbitrary limits on field
> > > sizes.
> >
> >     If we can solve it, let's do so. If we cannot, let's restrict
> >     it for 7.1.
>
> How are you doing the index toasting currently ? Is it on the same
> line as table toasting ? That is: toast some index column values if the key
> exceeds 2k ?

    The current CVS is broken in that area. You'll notice as soon
    as you have many huge "text" values in an index, update them,
    vacuum and continue to update.

    The  actual  behaviour  of the toaster is to toast each tuple
    until it has a delicious looking, brown and  crispy  surface.
    The  indicator  for  beeing delicious is that it shrank below
    MaxTupleSize/4 - that's a little less than 2K in a default 8K
    blocksize setup.

    It  then  sticks  the  new  tuple into the HeapTuple's t_data
    pointer.

    Index  inserts  are  allways  done  after  heap_insert()   or
    heap_update().   At that time, the index tuples will be built
    from the values found in the now  replaced  heap  tuple.  And
    since  the  heap  tuple found now is allways smaller than 2K,
    any combination of attributes out of it  must  be  too  (it's
    impossible  to  specify  one  and the same attribute multiple
    times in one index).

    So the indices simply inherit the toasting result. If a value
    got  compressed,  the index will store the compressed format.
    If it got moved off, the index  will  hold  the  toast  entry
    reference for it.

    One  of the biggest advantages is this: In the old system, an
    indexed column of 2K caused 2K be stored in the heap plus  2K
    stored in the index. Plus all the 2K instances in upper index
    block range specs.  Now, the heap and  the  index  will  only
    hold references or compressed items.

    Absolutely  no  problem for compressed items. All information
    to recreate the original value is in the Datum itself.

    For external stored ones, the reference tells the OIDs of the
    secondary  relation and it's index (where to find the data of
    this entry), a unique identifier of the  item  (another  OID)
    and  some  other  info.   So  the  reference contains all the
    information required to fetch the data just by looking at the
    reference.  And  since  the  detoaster  scans  the  secondary
    relation with a visibility of SnapShotAny, it'll  succeed  to
    find  them  even  if they've been deleted long ago by another
    committed transaction. So index  traversal  will  succeed  on
    that in any case.

    What  I  didn't  knew  at the time of implementation is, that
    btree indices can keep such a reference in upper level blocks
    range specifications even after a vacuum successfully deleted
    the index tuple holding  the  reference  itself.  That's  the
    current pity.

    Thus,  if  vacuum  finally  removed  deleted  tuples from the
    secondary relations (after  the  heap  and  index  have  been
    vacuumed),   the   detoaster   cannot   find  those  entries,
    referenced by upper index blocks, any more.

    Maybe we could propagate key range changes into upper  blocks
    at index_delete() time. Will look at the btree code now.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #