Horton

Re: logical decoding and replication of sequences, take 2

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Masahiko Sawada <sawada.mshk@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2023-12-07T04:03:00Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Migrate logical slots to the new node during an upgrade.

  2. Make test_decoding ddl.out shorter

  3. Fix snapshot handling in logicalmsg_decode

  4. doc: Adjust a few more references to "postmaster"

  5. Revert "Logical decoding of sequences"

On Wed, Dec 6, 2023 at 7:20 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
>
> On 12/6/23 11:19, Amit Kapila wrote:
> > On Sun, Dec 3, 2023 at 11:56 PM Tomas Vondra
> > <tomas.vondra@enterprisedb.com> wrote:
> >>
> >> On 12/3/23 18:52, Tomas Vondra wrote:
> >>> ...
> >>>
> >>
> >> Another idea is that maybe we could somehow inform ReorderBuffer whether
> >> the output plugin even is interested in sequences. That'd help with
> >> cases where we don't even want/need to replicate sequences, e.g. because
> >> the publication does not specify (publish=sequence).
> >>
> >> What happens now in that case is we call ReorderBufferQueueSequence(),
> >> it does the whole dance with starting/aborting the transaction, calls
> >> rb->sequence() which just does "meh" and doesn't do anything. Maybe we
> >> could just short-circuit this by asking the output plugin somehow.
> >>
> >> In an extreme case the plugin may not even specify the sequence
> >> callbacks, and we're still doing all of this.
> >>
> >
> > We could explore this but I guess it won't solve the problem we are
> > facing in cases where all sequences are published and plugin has
> > specified the sequence callbacks. I think it would add some overhead
> > of this check in positive cases where we decide to anyway do send the
> > changes.
>
> Well, the idea is the check would be very simple (essentially just a
> boolean flag somewhere), so not really measurable.
>
> And if the plugin requests decoding sequences, I guess it's natural it
> may have a bit of overhead. It needs to do more things, after all. It
> needs to be acceptable, ofc.
>

I agree with you that if it can be done cheaply or without a
measurable overhead then it would be a good idea and can serve other
purposes as well. For example, see discussion [1]. I had more of what
the patch in email [1] is doing where it needs to start/stop xact and
do so relcache access etc. which seems can add some overhead if done
for each change, though I haven't measured so can't be sure.

[1] - https://www.postgresql.org/message-id/CAGfChW5Qo2SrjJ7rU9YYtZbRaWv6v-Z8MJn%3DdQNx4uCSqDEOHA%40mail.gmail.com

-- 
With Regards,
Amit Kapila.