Memory Accounting
Jeff Davis <pgsql@j-davis.com>
From: Jeff Davis <pgsql@j-davis.com>
To: pgsql-hackers@postgresql.org
Date: 2019-07-18T18:24:25Z
Lists: pgsql-hackers
Attachments
- 0001-Memory-accounting-at-block-level.patch (text/x-patch) patch 0001
Previous discussion:
https://postgr.es/m/1407012053.15301.53.camel%40jeff-desktop
This patch introduces a way to ask a memory context how much memory it
currently has allocated. Each time a new block (not an individual
chunk, but a new malloc'd block) is allocated, it increments a struct
member; and each time a block is free'd, it decrements the struct
member. So it tracks blocks allocated by malloc, not what is actually
used for chunks allocated by palloc.
The purpose is for Memory Bounded Hash Aggregation, but it may be
useful in more cases as we try to better adapt to and control memory
usage at execution time.
I ran the same tests as Robert did before[1] on my laptop[2]. The only
difference is that I also set max_parallel_workers[_per_gather]=0 to be
sure. I did 5 runs, alternating between memory-accounting and master,
and I got the following results for "elapsed" (as reported by
trace_sort):
regression=# select version, min(s), max(s), percentile_disc(0.5)
within group (order by s) as median, avg(s)::numeric(10,2) from tmp
group by version;
version | min | max | median | avg
-------------------+-------+-------+--------+-------
master | 13.92 | 14.40 | 14.06 | 14.12
memory accounting | 13.43 | 14.46 | 14.11 | 14.09
(2 rows)
So, I don't see any significant performance impact for this patch in
this test. That may be because:
* It was never really significant except on PPC64.
* I changed it to only update mem_allocated for the current context,
not recursively for all parent contexts. It's now up to the function
that reports memory usage to recurse or not (as needed).
* A lot of changes to sort have happened since that time, so perhaps
it's not a good test of memory contexts any more.
pgbench didn't show any slowdown either.
I also did another test with hash aggregation that uses significant
memory (t10m is a table with 10M distinct values and work_mem is 1GB):
postgres=# select (select (i, count(*)) from t10m group by i having
count(*) > n) from (values(1),(2),(3),(4),(5)) as s(n);
I didn't see any noticable difference there, either.
Regards,
Jeff Davis
[1]
https://postgr.es/m/CA%2BTgmobnu7XEn1gRdXnFo37P79bF%3DqLt46%3D37ajP3Cro9dBRaA%40mail.gmail.com
[2] Linux jdavis 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Commits
-
Change MemoryContextMemAllocated to return Size
- 36425ece5d6c 13.0 landed
-
Use Size instead of int64 to track allocated memory
- f2369bc610a1 13.0 landed
-
Add transparent block-level memory accounting
- 5dd7fc151946 13.0 landed
-
Change the way pre-reading in external sort's merge phase works.
- e94568ecc10f 10.0 cited
-
Improve memory management for external sorts.
- 0011c0091e88 9.6.0 cited
-
In array_agg(), don't create a new context for every group.
- b419865a814a 9.5.0 cited