Re: BUG #14932: SELECT DISTINCT val FROM table gets stuck in an infinite loop

Todd A. Cook <tcook@blackducksoftware.com>

From: "Todd A. Cook" <tcook@blackducksoftware.com>
To: Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Thomas Munro <thomas.munro@enterprisedb.com>, PostgreSQL Bugs <pgsql-bugs@postgresql.org>
Date: 2018-01-29T18:20:25Z
Lists: pgsql-bugs
On 01/28/18 20:02, Andres Freund wrote:
> On 2018-01-26 18:48:35 -0500, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> On 2017-12-10 23:09:42 +0100, Tomas Vondra wrote:
>>>> FWIW I do agree the data sets shared in this thread are pretty extreme
>>>> and it doesn't make much sense to slow the regular cases. I'll be
>>>> perfectly happy if we stop the OOM, making those cases fast is a bonus.
>>
>>> Yea, agreed on that. I'm kinda inclined to go for stop-growing in 10,
>>> and so something better in 11. And then later possibly backpatch if
>>> we've grown some confidence?
>>
>> +1.  I'd like to see some response to this included in 10.2, and time
>> grows short for that.
> 
> Here are two patches that I think we want for 10.2, and the start of one
> that I think we want for master.  0002 is needed because otherwise the
> lack of extra growth leads to noticeably worse performance when filling
> an underestimated a coordinator hash table from the workers - turns out
> our hash combine (and most hash combines) let a lot of clustering
> survive. By adding a final hashing round the bit perturbation is near
> perfect.  The commit messages need to be polished a bit, but other than
> that I think these are reasonable fixes. Plan to push by Monday evening
> at the latest.

I applied 0001 and 0002 to REL_10_STABLE at 1b2a3860d3ea81825e9bbad2c7dbf66db87445c1
and got the following build error:

	execGrouping.c:26:29: fatal error: utils/hashutils.h: No such file or directory
	 #include "utils/hashutils.h"

My build sequence was

	git pull
	git status  (showing clean tree)
	make distclean
	./configure ...
	make

-- todd


Commits

  1. Improve bit perturbation in TupleHashTableHash.

  2. Prevent growth of simplehash tables when they're "too empty".

  3. Add stack-overflow guards in set-operation planning.