Horton

Re: Memory Accounting

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>, Jeff Davis <pgsql@j-davis.com>, Pg Hackers <pgsql-hackers@postgresql.org>
Date: 2019-09-24T05:21:40Z
Lists: pgsql-hackers 
On Wed, Jul 24, 2019 at 11:52:28PM +0200, Tomas Vondra wrote:
> I think Heikki was asking about places with a lot of sub-contexts, which a
> completely different issue. It used to be the case that some aggregates
> created a separate context for each group - like array_agg. That would
> make Jeff's approach to accounting rather inefficient, because checking
> how much memory is used would be very expensive (having to loop over a
> large number of contexts).

The patch has been marked as ready for committer for a week or so, but
it seems to me that this comment has not been addressed, no?  Are we
sure that we want this method if it proves to be inefficient when
there are many sub-contexts and shouldn't we at least test such
scenarios with a worst-case, customly-made, function?
--
Michael

Commits

  1. Change MemoryContextMemAllocated to return Size

  2. Use Size instead of int64 to track allocated memory

  3. Add transparent block-level memory accounting

  4. Change the way pre-reading in external sort's merge phase works.

  5. Improve memory management for external sorts.

  6. In array_agg(), don't create a new context for every group.