Re: BUG #14932: SELECT DISTINCT val FROM table gets stuck in an infinite loop
Todd A. Cook <tcook@blackducksoftware.com>
From: "Todd A. Cook" <tcook@blackducksoftware.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Thomas Munro <thomas.munro@enterprisedb.com>, PostgreSQL Bugs <pgsql-bugs@postgresql.org>
Date: 2018-01-30T18:57:44Z
Lists: pgsql-bugs
On 01/29/18 14:26, Andres Freund wrote: > On 2018-01-29 13:56:37 -0500, Todd A. Cook wrote: >> With those patches applied, I'm able to process a 175 million row data set >> without any problems. > > Cool! > >> I'll try the 900 million row set next. We're also able to process those 900 million rows without problems. :) It took about 28 minutes (average of 3 runs). Out of curiosity, I then modified hashint8() as previously described. With that change, run time dropped to 11 minutes (also average of 3 runs). FWIW, the data values in these sets are sort-of random (where I can't explain the "sort-of" in a public forum), but strongly biased towards negative infinity. Starting again from scratch, we could probably remove the bias, but we have 28-30 billion of these things collected over the last 14 years (starting in PostgreSQL 7.4) and it's kinda tough to change directions at this point... -- todd
Commits
-
Improve bit perturbation in TupleHashTableHash.
- d18d4bca81f8 10.2 landed
- c068f87723ca 11.0 landed
-
Prevent growth of simplehash tables when they're "too empty".
- d1aac2998789 10.2 landed
- ab9f2c429d8f 11.0 landed
-
Add stack-overflow guards in set-operation planning.
- 1b2a3860d3ea 10.2 cited