Horton

Re: Reducing the chunk header sizes on all memory context types

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>, David Rowley <dgrowleyml@gmail.com>, Yura Sokolov <y.sokolov@postgrespro.ru>, PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>
Date: 2022-08-09T21:02:58Z
Lists: pgsql-hackers 
Hi,

On 2022-08-09 15:21:57 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I think it's fine, given that we can change this at any time, but it's
> > probably worth to explicitly agree that this will for now restrict us to 8
> > context methods?
> 
> Do we really need it to be that tight?  I know we only have 3 methods today,
> but 8 doesn't seem that far away.  If there were six bits reserved for
> this I'd be happier.

We only have so many bits available, so that'd have to come from some other
resource.  The current division is:

+ * 1.	3-bits to indicate the MemoryContextMethodID
+ * 2.	1-bit to indicate if the chunk is externally managed (see below)
+ * 3.	30-bits for the amount of memory which was reserved for the chunk
+ * 4.	30-bits for the number of bytes that must be subtracted from the chunk
+ *		to obtain the address of the block that the chunk is stored on.

I suspect we could reduce 3) here a bit, which I think would end up with slab
context's max chunkSize shrinking further. Which should still be fine.

But also, we could defer that to later, this is a limit that we can easily
change.


> >> # We also add a restriction that block sizes for all 3 of the memory
> >> # allocators cannot be 1GB or larger.  We would be unable to store the
> >> # number of bytes that the block is offset from the chunk stored beyond this
> >> #1GB boundary on any block that was larger than 1GB.
> 
> Losing MemoryContextAllocHuge would be very bad, so I assume this comment
> is not telling the full truth.

It's just talking about chunked allocation (except for slab, which doesn't
have anything else, but as David pointed out, it makes no sense to use slab
for that large allocations). I.e. it's the max you can pass to
AllocSetContextCreate()'s and GenerationContextCreate()'s maxBlockSize, and to
SlabContextCreate()'s chunkSize.   I don't think we have any code that
currently sets a bigger limit than 8MB.

Greetings,

Andres Freund

Commits

  1. Harden memory context allocators against bogus chunk pointers.

  2. Improve our ability to detect bogus pointers passed to pfree et al.

  3. Remove MemoryContextContains().

  4. Remove uses of MemoryContextContains in nodeAgg.c and nodeWindowAgg.c.

  5. Temporarily make MemoryContextContains return false

  6. Make MemoryContextContains work correctly again

  7. Make more effort to put a sentinel at the end of allocated memory

  8. Fix some possibly latent bugs in slab.c

  9. Various cleanups of the new memory context header code

  10. Revert "Add missing padding from MemoryChunk struct"

  11. Use MAXALIGN() in calculations using sizeof(SlabBlock)

  12. Add missing padding from MemoryChunk struct

  13. Improve performance of and reduce overheads of memory management