Horton

Re: Using per-transaction memory contexts for storing decoded tuples

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Fujii Masao <masao.fujii@oss.nttdata.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, David Rowley <dgrowleyml@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2024-10-15T17:45:01Z
Lists: pgsql-hackers

Attachments

On Sun, Oct 13, 2024 at 11:00 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Oct 11, 2024 at 3:40 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Please find the attached patches.
> >

Thank you for reviewing the patch!

>
> @@ -343,9 +343,9 @@ ReorderBufferAllocate(void)
>   */
>   buffer->tup_context = GenerationContextCreate(new_ctx,
>     "Tuples",
> -   SLAB_LARGE_BLOCK_SIZE,
> -   SLAB_LARGE_BLOCK_SIZE,
> -   SLAB_LARGE_BLOCK_SIZE);
> +   SLAB_DEFAULT_BLOCK_SIZE,
> +   SLAB_DEFAULT_BLOCK_SIZE,
> +   SLAB_DEFAULT_BLOCK_SIZE);
>
> Shouldn't we change the comment atop this change [1] which states that
> we should benchmark the existing values?

Agreed.

>
> One more thing we kept the max size as SLAB_DEFAULT_BLOCK_SIZE instead
> of something like we do with ALLOCSET_DEFAULT_SIZES, so we can
> probably write a comment as to why we choose to use the max_size same
> as init_size.

Agreed. I've updated the comment. Please review it.

> BTW, can we once try to use the max size as
> SLAB_LARGE_BLOCK_SIZE? Can it lead to the same problem with concurrent
> transactions where freeing larger blocks could be a problem, if so, we
> can at least write a comment for future reference.

I've tested with SLAB_LARGE_BLOCK_SIZE as the max size but it seems
that a huge memory fragmentation issue still happens. In the same
scenario I used before, the maximum amount of allocated memory during
logical decoding was 1.26GB with logical_decodiing_work_mem being
'256MB'.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Reduce memory block size for decoded tuple storage to 8kB.

  2. Generational memory allocator