Horton

Re: Memory-Bounded Hash Aggregation

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Jeff Davis <pgsql@j-davis.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Taylor Vesely <tvesely@pivotal.io>, Adam Lee <ali@pivotal.io>, Melanie Plageman <mplageman@pivotal.io>, pgsql-hackers@postgresql.org
Date: 2020-01-08T10:38:18Z
Lists: pgsql-hackers 
On 28/12/2019 01:35, Jeff Davis wrote:
> I've attached a new patch that adds some basic costing for disk during
> hashagg.

This patch (hashagg-20191227.patch) doesn't compile:

nodeAgg.c:3379:7: error: ‘hashagg_mem_overflow’ undeclared (first use in 
this function)
    if (hashagg_mem_overflow)
        ^~~~~~~~~~~~~~~~~~~~

Looks like the new GUCs got lost somewhere between 
hashagg-20191220.patch and hashagg-20191227.patch.

> /*
>  * find_aggregated_cols
>  *	  Construct a bitmapset of the column numbers of aggregated Vars
>  *	  appearing in our targetlist and qual (HAVING clause)
>  */
> static Bitmapset *
> find_aggregated_cols(AggState *aggstate)
> {
> 	Agg		   *node = (Agg *) aggstate->ss.ps.plan;
> 	Bitmapset  *colnos = NULL;
> 	ListCell   *temp;
> 
> 	/*
> 	 * We only want the columns used by aggregations in the targetlist or qual
> 	 */
> 	if (node->plan.targetlist != NULL)
> 	{
> 		foreach(temp, (List *) node->plan.targetlist)
> 		{
> 			if (IsA(lfirst(temp), TargetEntry))
> 			{
> 				Node *node = (Node *)((TargetEntry *)lfirst(temp))->expr;
> 				if (IsA(node, Aggref) || IsA(node, GroupingFunc))
> 					find_aggregated_cols_walker(node, &colnos);
> 			}
> 		}
> 	}

This makes the assumption that all Aggrefs or GroupingFuncs are at the 
top of the TargetEntry. That's not true, e.g.:

select 0+sum(a) from foo group by b;

I think find_aggregated_cols() and find_unaggregated_cols() should be 
merged into one function that scans the targetlist once, and returns two 
Bitmapsets. They're always used together, anyway.

- Heikki

Commits

  1. Fix costing for disk-based hash aggregation.

  2. Fixes for Disk-based Hash Aggregation.

  3. Disk-based Hash Aggregation.

  4. Extend ExecBuildAggTrans() to support a NULL pointer check.

  5. Save calculated transitionSpace in Agg node.

  6. Minor refactor of nodeAgg.c.

  7. Logical Tape Set: lazily allocate read buffer.

  8. Introduce TupleHashTableHash() and LookupTupleHashEntryHash().

  9. Logical Tape Set: use min heap for freelist.

  10. Fix comments in execGrouping.c

  11. Avoid integer overflow while sifting-up a heap in tuplesort.c.

  12. Make the overflow guards in ExecChooseHashTableSize be more protective.

  13. Improve performance of our private version of qsort. Per recent testing,