Horton

Re: Default setting for enable_hashagg_disk

Jeff Davis <pgsql@j-davis.com>

View thread View on postgr.es
From: Jeff Davis <pgsql@j-davis.com>
To: Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Justin Pryzby <pryzby@telsasoft.com>, Bruce Momjian <bruce@momjian.us>, David Rowley <dgrowleyml@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, pgsql-hackers@postgresql.org
Date: 2020-06-25T16:42:33Z
Lists: pgsql-hackers, pgsql-docs 
On Wed, 2020-06-24 at 12:31 -0700, Andres Freund wrote:
> nodeAgg.c already treats those separately:
> 
> void
> hash_agg_set_limits(double hashentrysize, uint64 input_groups, int
> used_bits,
> 					Size *mem_limit, uint64
> *ngroups_limit,
> 					int *num_partitions)
> {
> 	int			npartitions;
> 	Size		partition_mem;
> 
> 	/* if not expected to spill, use all of work_mem */
> 	if (input_groups * hashentrysize < work_mem * 1024L)
> 	{
> 		if (num_partitions != NULL)
> 			*num_partitions = 0;
> 		*mem_limit = work_mem * 1024L;
> 		*ngroups_limit = *mem_limit / hashentrysize;
> 		return;
> 	}

The reason this code exists is to decide how much of work_mem to set
aside for spilling (each spill partition needs an IO buffer).

The alternative would be to fix the number of partitions before
processing a batch, which didn't seem ideal. Or, we could just ignore
the memory required for IO buffers, like HashJoin.

Granted, this is an example where an underestimate can give an
advantage, but I don't think we want to extend the concept into other
areas.

Regards,
	Jeff Davis

Commits

  1. Add hash_mem_multiplier GUC.

  2. HashAgg: use better cardinality estimate for recursive spilling.

  3. Remove hashagg_avoid_disk_plan GUC.

  4. Doc fixup for hashagg_avoid_disk_plan GUC.

  5. Rework HashAgg GUCs.

  6. Disk-based Hash Aggregation.

  7. Implement partition-wise grouping/aggregation.

  8. Defer creation of partially-grouped relation until it's needed.