Re: update on TOAST status'

Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> >     After debugging something I thought was a bug in the toaster,
> >     I've found something really causing headaches.
> >     TOAST AS IS IS NOT CAPABLE OF HOLDING INDEXED VALUES!
> >     It appears that brtee indices (at least) can keep  references
> >     to  old  toast  values  that  survive  a  VACUUM! Seems these
> >     references live in nodes actually not  referring  to  a  heap
> >     tuple   any   more,   but   used  during  tree  traversal  in
> >     comparisions.  As if an index tuple delete from a  btree  not
> >     necessarily  causes  the  index  value  to disappear from the
> >     btree completely.  It'll never be returned by an index  scan,
> >     but the value is still there somewhere.
>
> Oooh, nasty.  Probably the keys you are looking at are in upper-
> level btree pages and indicate the ranges of keys found in lower
> pages, rather than being pointers to real tuples.

    So  our  btree  implementation  is  closer  to  an  ISAM file
    organization than to a real tree? Anyway, either one  or  the
    other  is  the  reason  that an attempt to insert a new value
    results in an lztext_cmp() call that cannot be  resolved  due
    to a missing toast value.

    I  added  some  checks  to  the  detoaster  just  to throw an
    elog(ERROR) instead of a coredump  in  such  a  case  earlier
    today.

> One answer is to rebuild indexes from scratch during VACUUM,
> before we vacuum the TOAST relation.  We've been talking about
> doing that for a long time.  Maybe it's time to bite the bullet
> and do it.  (Of course that means fixing the relation-versioning
> problem, which it seems we don't have a consensus on yet...)

    Doesn't  matter if we do it before or after, because the main
    heap shouldn't contain any more toast references  to  deleted
    (later to be vacuumed) toast entries at that time.

    Anyway,  it's  a nice idea that should solve the problem. For
    indices, which can allways be rebuilt from the heap  data,  I
    don't  see  such  a  big need for the versioning. Only that a
    partially rebuilt index (rebuild crashed in the middle) needs
    another  vacuum  before  the  the DB is accessible again. How
    often does that happen?

    So why not having vacuum truncating the index  file  to  zero
    and rebuilding it from scratch in place? Can anyone access an
    index while vacuum has a lock on it's heap?

>
> >     Seems  the  designs of btree and toast are colliding. As soon
> >     as "text" is toastable, this'll hurt - be warned.
>
> Text *is* marked toastable in current CVS...

    Whow - haven't noticed.

    Will run my tests against text ... parallel. Does it have any
    impact  on the regression test execution time? Does any toast
    table (that should now be there in the regression  DB)  loose
    it's zero size during the tests?

>
> >     AFAICS, we need to detoast values for index  inserts  allways
> >     and  have  another toaster inside the index access methods at
> >     some day.  In the meantime we should decide  a  safe  maximum
> >     index tuple size and emit an explanative error message on the
> >     attempt to insert oversized index entries instead of possibly
> >     corrupting the index.
>
> I don't like that --- seems it would put a definite crimp in the
> whole point of TOAST, which is not to have arbitrary limits on field
> sizes.

    If we can solve it, let's do so. If we cannot, let's restrict
    it for 7.1.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #