Horton

Re: Memory-Bounded Hash Aggregation

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Jeff Davis <pgsql@j-davis.com>
Cc: Andres Freund <andres@anarazel.de>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Melanie Plageman <melanieplageman@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Taylor Vesely <tvesely@pivotal.io>, Adam Lee <ali@pivotal.io>, Melanie Plageman <mplageman@pivotal.io>, pgsql-hackers@postgresql.org
Date: 2020-03-19T06:42:22Z
Lists: pgsql-hackers 
On Sun, Mar 15, 2020 at 04:05:37PM -0700, Jeff Davis wrote:
> > +	if (from_tape)
> > +		partition_mem += HASHAGG_READ_BUFFER_SIZE;
> > +	partition_mem = npartitions * HASHAGG_WRITE_BUFFER_SIZE;
> > 
> > => That looks wrong ; should say += ?
> 
> Good catch! Fixed.

> +++ b/src/backend/executor/nodeAgg.c
> @@ -2518,9 +3499,36 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
>  	 */
>  	if (use_hashing)
>  	{
> +		Plan   *outerplan = outerPlan(node);
> +		uint64	totalGroups = 0;
> +		for (i = 0; i < aggstate->num_hashes; i++)
> +			totalGroups = aggstate->perhash[i].aggnode->numGroups;
> +
> +		hash_agg_set_limits(aggstate->hashentrysize, totalGroups, 0,

I realize that I missed the train but .. that looks like another += issue?

Also, Andres was educating me about the range of behavior of "long" type, and I
see now while rebasing that you did the same thing.
https://www.postgresql.org/message-id/20200306175859.d56ohskarwldyrrw%40alap3.anarazel.de

-- 
Justin

Commits

  1. Fix costing for disk-based hash aggregation.

  2. Fixes for Disk-based Hash Aggregation.

  3. Disk-based Hash Aggregation.

  4. Extend ExecBuildAggTrans() to support a NULL pointer check.

  5. Save calculated transitionSpace in Agg node.

  6. Minor refactor of nodeAgg.c.

  7. Logical Tape Set: lazily allocate read buffer.

  8. Introduce TupleHashTableHash() and LookupTupleHashEntryHash().

  9. Logical Tape Set: use min heap for freelist.

  10. Fix comments in execGrouping.c

  11. Avoid integer overflow while sifting-up a heap in tuplesort.c.

  12. Make the overflow guards in ExecChooseHashTableSize be more protective.

  13. Improve performance of our private version of qsort. Per recent testing,