Re: Memory Accounting

Melanie Plageman <melanieplageman@gmail.com>

From: Melanie Plageman <melanieplageman@gmail.com>
To: Jeff Davis <pgsql@j-davis.com>
Cc: Soumyadeep Chakraborty <sochakraborty@pivotal.io>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Robert Haas <robertmhaas@gmail.com>, Pg Hackers <pgsql-hackers@postgresql.org>, soumyadeep2007@gmail.com
Date: 2019-09-24T18:46:49Z
Lists: pgsql-hackers
On Thu, Sep 19, 2019 at 11:00 AM Jeff Davis <pgsql@j-davis.com> wrote:

> On Wed, 2019-09-18 at 13:50 -0700, Soumyadeep Chakraborty wrote:
> > Hi Jeff,
>
> Hi Soumyadeep and Melanie,
>
> Thank you for the review!
>
> > max_stack_depth       max level       lazy (ms)       eager (ms)
> (eage
> > r/lazy)
> > 2MB   82      302.715 427.554 1.4123978
> > 3MB   3474    567.829 896.143 1.578191674
> > 7.67MB        8694    2657.972        4903.063        1.844663149
>
> Thank you for collecting data on this. Were you able to find any
> regression when compared to no memory accounting at all?
>
>
We didn't spend much time comparing performance with and without
memory accounting, as it seems like this was discussed extensively in
the previous thread.


> It looks like you agree with the approach and the results. Did you find
> any other issues with the patch?
>

We didn't observe any other problems with the patch and agree with the
approach. It is a good start.


>
> I am also including Robert in this thread. He had some concerns the
> last time around due to a small regression on POWER.
>

I think it would be helpful if we could repeat the performance tests
Robert did on that machine with the current patch (unless this version
of the patch is exactly the same as the ones he tested previously).

Thanks,
Soumyadeep & Melanie

Commits

  1. Change MemoryContextMemAllocated to return Size

  2. Use Size instead of int64 to track allocated memory

  3. Add transparent block-level memory accounting

  4. Change the way pre-reading in external sort's merge phase works.

  5. Improve memory management for external sorts.

  6. In array_agg(), don't create a new context for every group.