Horton

Re: Skipping logical replication transactions on subscriber side

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Greg Nancarrow <gregn4422@gmail.com>, vignesh C <vignesh21@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, Alexey Lesovsky <lesovsky@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2022-01-12T06:02:19Z
Lists: pgsql-hackers

Attachments

On Wed, Jan 12, 2022 at 12:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 12, 2022 at 5:49 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jan 11, 2022 at 7:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Tue, Jan 11, 2022 at 1:51 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On second thought, the same is true for other cases, for example,
> > > > preparing the transaction and clearing skip_xid while handling a
> > > > prepare message. That is, currently we don't clear skip_xid while
> > > > handling a prepare message but do that while handling commit/rollback
> > > > prepared message, in order to avoid the worst case. If we do both
> > > > while handling a prepare message and the server crashes between them,
> > > > it ends up that skip_xid is cleared and the transaction will be
> > > > resent, which is identical to the worst-case above.
> > > >
> > >
> > > How are you thinking to update the skip xid before prepare? If we do
> > > it in the same transaction then the changes in the catalog will be
> > > part of the prepared xact but won't be committed. Now, say if we do it
> > > after prepare, then the situation won't be the same because after
> > > restart the same xact won't appear again.
> >
> > I was thinking to commit the catalog change first in a separate
> > transaction while not updating origin LSN and then prepare an empty
> > transaction while updating origin LSN.
> >
>
> But, won't it complicate the handling if in the future we try to
> enhance this API such that it skips partial changes like skipping only
> for particular relation(s) or particular operations as discussed
> previously in this thread?

Right. I was thinking that if we accept the situation that the user
has to set skip_xid again in case of the server crashes, we might be
able to accept also the situation that the user has to clear skip_xid
in a case of the server crashes. But it seems the former is less
problematic.

I've attached an updated patch that incorporated all comments I got so far.

Regards,

--
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue