Re: Sequential Scans

On Thursday 06 Mar 2003 9:15 pm, Ericson Smith wrote:
> Hi,
>
> I have a table with about 3.2 Million records.
>
> There is one process that I run that needs to process batches of records
> 1000 at a time out of a set of approximately 220,000 records.
>
> So my query looks like this:
> SELECT a.*, b.url FROM listings a, urls b WHERE a.urlindex=b.index AND
> a.haslid=1 ORDER BY a.index LIMIT 1000 OFFSET 0;
>
> Doing the above query with an offset of up to 5000 (the 5th batch) shows
> (with EXPLAIN) that index scans are being used.
>
> Exceeding an OFFSET of 5000 produces sequential scans. The whole process
> goes horribly slow at that point.

To get to an offset of 5000 it has to find the first 4999 and then ignore 
them. There comes a point at which the planner decides the cost of grabbing 
many index items exceeds the cost of just reading the table. It might be in 
your case the planner's estimating the wrong changeover point - read up on 
the runtime environment settings, I think CPU_INDEX_TUPLE_COST may be your 
friend (see archives for discussion).

In your case though, you might want to look at using a cursor and then 
fetching blocks of 1000 rows at a time.

-- 
  Richard Huxton