Re: string_to_array eats too much memory?

> > > I'm playing with GIN to make a full text search system. GIN comes 
> > > with built-in TEXT[] support and I use string_to_array() 
> to make a 
> > > TEXT[]. Problem is, if there's large number of array elemets,
> > > string_to_array() consumes too much memory. For example, to make 
> > > ~70k array elements, string_to_array seems to eat several 
> Gig bytes 
> > > of memory. ~70k array elements means there are same 
> number of words 
> > > in a document which is not too big in a large text IMO.
> > 
> > Do you mean 70k unique lexemes? Ugh.
> 
> I'm testing how GIN scales.
> 
> > Why do not you use tsearch framework?
> 
> ? I thought GIN is superior than tsearch2. 
> 
> From your GIN proposal posted to pgsql-hackers:
> 
> "The primary goal of the Gin index is a scalable full text 
> search in PostgreSQL"

tsearch2 *uses* GIN in 8.2. Just CREATE INDEX foo ON bar USING
gin(mytsvector).

And tsearch2 in 8.2 with GIN can be a *lot* faster than with GIST. I've
been running experiments on the website search with tsearch2/GIN and
i've been seeing fantastic performance compared top revious versions.



//Magnus