Horton

Re: Improve eviction algorithm in ReorderBuffer

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2023-12-15T12:18:55Z
Lists: pgsql-hackers 
On Fri, Dec 15, 2023 at 2:59 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Dec 15, 2023 at 12:37 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Dec 13, 2023 at 6:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Tue, Dec 12, 2023 at 1:33 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Tue, Dec 12, 2023 at 9:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > I've heard the report internally that replication lag became huge when
> > > > > decoding transactions each consisting of 500k sub transactions. Note
> > > > > that ReorderBufferLargetstTXN() is used only in non-streaming mode.
> > > > >
> >
> > Can't you suggest them to use streaming mode to avoid this problem or
> > do you see some problem with that?
>
> Yeah, that's one option. But I can suggest
>

Sorry, it was still in the middle of editing.

Yeah, that's one option. But since there is a trade-off I cannot
suggest using streaming mode for every user. Also, the logical
replication client (e.g. third party tool receiving logical change
set) might not support the streaming mode yet.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Revert indexed and enlargable binary heap implementation.

  2. Replace binaryheap + index with pairingheap in reorderbuffer.c

  3. Improve eviction algorithm in ReorderBuffer using max-heap for many subtransactions.

  4. Add functions to binaryheap for efficient key removal and update.

  5. Make binaryheap enlargeable.