Horton

Re: Reducing the chunk header sizes on all memory context types

David Rowley <dgrowleyml@gmail.com>

View thread View on postgr.es
From: David Rowley <dgrowleyml@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Amit Kapila <amit.kapila16@gmail.com>, Andres Freund <andres@anarazel.de>, Robert Haas <robertmhaas@gmail.com>, Yura Sokolov <y.sokolov@postgrespro.ru>, PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>
Date: 2022-09-07T04:13:37Z
Lists: pgsql-hackers 
On Tue, 6 Sept 2022 at 01:41, David Rowley <dgrowleyml@gmail.com> wrote:
>
> On Fri, 2 Sept 2022 at 20:11, David Rowley <dgrowleyml@gmail.com> wrote:
> >
> > On Thu, 1 Sept 2022 at 12:46, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >
> > > David Rowley <dgrowleyml@gmail.com> writes:
> > > > Maybe we should just consider always making room for a sentinel for
> > > > chunks that are on dedicated blocks. At most that's an extra 8 bytes
> > > > in some allocation that's either over 1024 or 8192 (depending on
> > > > maxBlockSize).
> > >
> > > Agreed, if we're not doing that already then we should.
> >
> > Here's a patch to that effect.
>
> If there are no objections, then I plan to push that patch soon.

I've now pushed the patch which adds the sentinel space in more cases.

The final analysis I did on the stats gathered during make
installcheck show that we'll now allocate about 19MBs more over the
entire installcheck run out of about 26GBs total allocations.

That analysis looks something like:

Before:

SELECT CASE
         WHEN pow2_size > 0
              AND pow2_size = size THEN 'No'
         WHEN pow2_size = 0
              AND size = maxalign_size THEN 'No'
         ELSE 'Yes'
       END                    AS has_sentinel,
       Count(*)               AS n_allocations,
       Sum(CASE
             WHEN pow2_size > 0 THEN pow2_size
             ELSE maxalign_size
           END) / 1024 / 1024 mega_bytes_alloc
FROM   memstats
GROUP  BY 1;
has_sentinel | n_allocations | mega_bytes_alloc
--------------+---------------+------------------
 No           |      26445855 |            21556
 Yes          |      37602052 |             5044

After:

SELECT CASE
         WHEN pow2_size > 0
              AND pow2_size = size THEN 'No'
         WHEN pow2_size = 0
              AND size = maxalign_size THEN 'Yes' -- this part changed
         ELSE 'Yes'
       END                    AS has_sentinel,
       Count(*)               AS n_allocations,
       Sum(CASE
             WHEN pow2_size > 0 THEN pow2_size
             WHEN size = maxalign_size THEN maxalign_size + 8
             ELSE maxalign_size
           END) / 1024 / 1024 mega_bytes_alloc
FROM   memstats
GROUP  BY 1;
has_sentinel | n_allocations | mega_bytes_alloc
--------------+---------------+------------------
 No           |      23980527 |             2177
 Yes          |      40067380 |            24442

That amounts to previously having about 58.7% of allocations having a
sentinel up to 62.6% currently, during the installcheck run.

It seems a pretty large portion of allocation request sizes are
power-of-2 sized and use AllocSet.

David

Commits

  1. Harden memory context allocators against bogus chunk pointers.

  2. Improve our ability to detect bogus pointers passed to pfree et al.

  3. Remove MemoryContextContains().

  4. Remove uses of MemoryContextContains in nodeAgg.c and nodeWindowAgg.c.

  5. Temporarily make MemoryContextContains return false

  6. Make MemoryContextContains work correctly again

  7. Make more effort to put a sentinel at the end of allocated memory

  8. Fix some possibly latent bugs in slab.c

  9. Various cleanups of the new memory context header code

  10. Revert "Add missing padding from MemoryChunk struct"

  11. Use MAXALIGN() in calculations using sizeof(SlabBlock)

  12. Add missing padding from MemoryChunk struct

  13. Improve performance of and reduce overheads of memory management