Horton

Re: Memory Accounting

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Jeff Davis <pgsql@j-davis.com>, pgsql-hackers@postgresql.org
Date: 2019-07-22T08:30:37Z
Lists: pgsql-hackers 
On 18/07/2019 21:24, Jeff Davis wrote:
> Previous discussion:
> https://postgr.es/m/1407012053.15301.53.camel%40jeff-desktop
> 
> This patch introduces a way to ask a memory context how much memory it
> currently has allocated. Each time a new block (not an individual
> chunk, but a new malloc'd block) is allocated, it increments a struct
> member; and each time a block is free'd, it decrements the struct
> member. So it tracks blocks allocated by malloc, not what is actually
> used for chunks allocated by palloc.
> 
> The purpose is for Memory Bounded Hash Aggregation, but it may be
> useful in more cases as we try to better adapt to and control memory
> usage at execution time.

Seems handy.

> * I changed it to only update mem_allocated for the current context,
> not recursively for all parent contexts. It's now up to the function
> that reports memory usage to recurse or not (as needed).

Is that OK for memory bounded hash aggregation? Might there be a lot of 
sub-contexts during hash aggregation?

- Heikki

Commits

  1. Change MemoryContextMemAllocated to return Size

  2. Use Size instead of int64 to track allocated memory

  3. Add transparent block-level memory accounting

  4. Change the way pre-reading in external sort's merge phase works.

  5. Improve memory management for external sorts.

  6. In array_agg(), don't create a new context for every group.