Horton

Re: Reducing the chunk header sizes on all memory context types

Yura Sokolov <y.sokolov@postgrespro.ru>

View thread View on postgr.es
From: Yura Sokolov <y.sokolov@postgrespro.ru>
To: David Rowley <dgrowleyml@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Amit Kapila <amit.kapila16@gmail.com>, Andres Freund <andres@anarazel.de>, Robert Haas <robertmhaas@gmail.com>, PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>
Date: 2022-09-07T07:15:24Z
Lists: pgsql-hackers 
В Ср, 07/09/2022 в 16:13 +1200, David Rowley пишет:
> On Tue, 6 Sept 2022 at 01:41, David Rowley <dgrowleyml@gmail.com> wrote:
> > 
> > On Fri, 2 Sept 2022 at 20:11, David Rowley <dgrowleyml@gmail.com> wrote:
> > > 
> > > On Thu, 1 Sept 2022 at 12:46, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > > 
> > > > David Rowley <dgrowleyml@gmail.com> writes:
> > > > > Maybe we should just consider always making room for a sentinel for
> > > > > chunks that are on dedicated blocks. At most that's an extra 8 bytes
> > > > > in some allocation that's either over 1024 or 8192 (depending on
> > > > > maxBlockSize).
> > > > 
> > > > Agreed, if we're not doing that already then we should.
> > > 
> > > Here's a patch to that effect.
> > 
> > If there are no objections, then I plan to push that patch soon.
> 
> I've now pushed the patch which adds the sentinel space in more cases.
> 
> The final analysis I did on the stats gathered during make
> installcheck show that we'll now allocate about 19MBs more over the
> entire installcheck run out of about 26GBs total allocations.
> 
> That analysis looks something like:
> 
> Before:
> 
> SELECT CASE
>          WHEN pow2_size > 0
>               AND pow2_size = size THEN 'No'
>          WHEN pow2_size = 0
>               AND size = maxalign_size THEN 'No'
>          ELSE 'Yes'
>        END                    AS has_sentinel,
>        Count(*)               AS n_allocations,
>        Sum(CASE
>              WHEN pow2_size > 0 THEN pow2_size
>              ELSE maxalign_size
>            END) / 1024 / 1024 mega_bytes_alloc
> FROM   memstats
> GROUP  BY 1;
> has_sentinel | n_allocations | mega_bytes_alloc
> --------------+---------------+------------------
>  No           |      26445855 |            21556
>  Yes          |      37602052 |             5044
> 
> After:
> 
> SELECT CASE
>          WHEN pow2_size > 0
>               AND pow2_size = size THEN 'No'
>          WHEN pow2_size = 0
>               AND size = maxalign_size THEN 'Yes' -- this part changed
>          ELSE 'Yes'
>        END                    AS has_sentinel,
>        Count(*)               AS n_allocations,
>        Sum(CASE
>              WHEN pow2_size > 0 THEN pow2_size
>              WHEN size = maxalign_size THEN maxalign_size + 8
>              ELSE maxalign_size
>            END) / 1024 / 1024 mega_bytes_alloc
> FROM   memstats
> GROUP  BY 1;
> has_sentinel | n_allocations | mega_bytes_alloc
> --------------+---------------+------------------
>  No           |      23980527 |             2177
>  Yes          |      40067380 |            24442
> 
> That amounts to previously having about 58.7% of allocations having a
> sentinel up to 62.6% currently, during the installcheck run.
> 
> It seems a pretty large portion of allocation request sizes are
> power-of-2 sized and use AllocSet.

19MB over 26GB is almost nothing. If it is only for enable-casserts
builds, then it is perfectly acceptable.

regards
Yura

Commits

  1. Harden memory context allocators against bogus chunk pointers.

  2. Improve our ability to detect bogus pointers passed to pfree et al.

  3. Remove MemoryContextContains().

  4. Remove uses of MemoryContextContains in nodeAgg.c and nodeWindowAgg.c.

  5. Temporarily make MemoryContextContains return false

  6. Make MemoryContextContains work correctly again

  7. Make more effort to put a sentinel at the end of allocated memory

  8. Fix some possibly latent bugs in slab.c

  9. Various cleanups of the new memory context header code

  10. Revert "Add missing padding from MemoryChunk struct"

  11. Use MAXALIGN() in calculations using sizeof(SlabBlock)

  12. Add missing padding from MemoryChunk struct

  13. Improve performance of and reduce overheads of memory management