Re: BUG #14932: SELECT DISTINCT val FROM table gets stuck in an infinite loop

Todd A. Cook <tcook@blackducksoftware.com>

From: "Todd A. Cook" <tcook@blackducksoftware.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Thomas Munro <thomas.munro@enterprisedb.com>, PostgreSQL Bugs <pgsql-bugs@postgresql.org>
Date: 2018-01-30T18:57:44Z
Lists: pgsql-bugs
On 01/29/18 14:26, Andres Freund wrote:
> On 2018-01-29 13:56:37 -0500, Todd A. Cook wrote:
>> With those patches applied, I'm able to process a 175 million row data set
>> without any problems.
> 
> Cool!
> 
>> I'll try the 900 million row set next.

We're also able to process those 900 million rows without problems. :)
It took about 28 minutes (average of 3 runs).

Out of curiosity, I then modified hashint8() as previously described.
With that change, run time dropped to 11 minutes (also average of 3
runs).

FWIW, the data values in these sets are sort-of random (where I can't
explain the "sort-of" in a public forum), but strongly biased towards
negative infinity.  Starting again from scratch, we could probably remove
the bias, but we have 28-30 billion of these things collected over the
last 14 years (starting in PostgreSQL 7.4) and it's kinda tough to change
directions at this point...

-- todd


Commits

  1. Improve bit perturbation in TupleHashTableHash.

  2. Prevent growth of simplehash tables when they're "too empty".

  3. Add stack-overflow guards in set-operation planning.