Horton

Re: Skipping logical replication transactions on subscriber side

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Greg Nancarrow <gregn4422@gmail.com>, vignesh C <vignesh21@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, Alexey Lesovsky <lesovsky@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-12-16T05:06:34Z
Lists: pgsql-hackers 
On Thu, Dec 16, 2021 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Dec 15, 2021 at 8:19 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, Dec 15, 2021 at 1:10 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Dec 15, 2021 at 8:19 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Tue, Dec 14, 2021 at 2:35 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
> > > > >
> > > > > On Tue, Dec 14, 2021 at 3:23 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > >
> > > > > > While the worker is skipping one of the skip transactions specified by
> > > > > > the user and immediately if the user specifies another skip
> > > > > > transaction while the skipping of the transaction is in progress this
> > > > > > new value will be reset by the worker while clearing the skip xid. I
> > > > > > felt once the worker has identified the skip xid and is about to skip
> > > > > > the xid, the worker can acquire a lock to prevent concurrency issues:
> > > > >
> > > > > That's a good point.
> > > > > If only the last_error_xid could be skipped, then this wouldn't be an
> > > > > issue, right?
> > > > > If a different xid to skip is specified while the worker is currently
> > > > > skipping a transaction, should that even be allowed?
> > > > >
> > > >
> > > > We don't expect such usage but yes, it could happen and seems not
> > > > good. I thought we can acquire Share lock on pg_subscription during
> > > > the skip but not sure it's a good idea. It would be better if we can
> > > > find a way to allow users to specify only XID that has failed.
> > > >
> > >
> > > Yeah, but as we don't have a definite way to allow specifying only
> > > failed XID, I think it is better to use share lock on that particular
> > > subscription. We are already using it for add/update rel state (see,
> > > AddSubscriptionRelState, UpdateSubscriptionRelState), so this will be
> > > another place to use a similar technique.
> >
> > Yes, but it seems to mean that we disallow users to change skip_xid
> > while the apply worker is skipping changes so we will end up having
> > the same problem we discussed so far;
> >
>
> I thought we just want to lock before clearing the skip_xid something
> like take the lock, check if the skip_xid in the catalog is the same
> as we have skipped, if it is the same then clear it, otherwise, leave
> it as it is. How will that disallow users to change skip_xid when we
> are skipping changes?

Oh I thought we wanted to keep holding the lock while skipping changes
(changing skip_xid requires acquiring the lock).

So if skip_xid is already changed, the apply worker would do
replorigin_advance() with WAL logging, instead of committing the
catalog change?

Regards,

-- 
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue