Horton

Re: Skipping logical replication transactions on subscriber side

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Mark Dilger <mark.dilger@enterprisedb.com>, Greg Nancarrow <gregn4422@gmail.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, Alexey Lesovsky <lesovsky@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-09-27T05:49:50Z
Lists: pgsql-hackers 
On Mon, Sep 27, 2021 at 12:46 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 24, 2021 at 7:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Fri, Sep 3, 2021 at 4:33 AM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> >
> > > I am attaching a version of such a function, plus some tests of your patch (since it does not appear to have any).  Would you mind reviewing these and giving comments or including them in your next patch version?
> > >
> >
> > I've looked at the patch and here are some comments:
> >
> > +
> > +-- no errors should be reported
> > +SELECT * FROM pg_stat_subscription_errors;
> > +
> >
> > +
> > +-- Test that the subscription errors view exists, and has the right columns
> > +-- If we expected any rows to exist, we would need to filter out unstable
> > +-- columns.  But since there should be no errors, we just select them all.
> > +select * from pg_stat_subscription_errors;
> >
> > The patch adds checks of pg_stat_subscription_errors in order to test
> > if the subscription doesn't have any error. But since the subscription
> > errors are updated in an asynchronous manner, we cannot say the
> > subscription is working fine by checking the view only once.
> >
>
> One question I have here is, can we reliably write few tests just for
> the new view patch? Right now, it has no test, having a few tests will
> be better. Here, because the apply worker will keep on failing till we
> stop it or resolve the conflict, can we rely on that fact?  The idea
> is that even if one of the entry is missed by stats collector, a new
> one (probably the same one) will be issued and we can wait till we see
> one error in view. We can add additional PostgresNode.pm
> infrastructure once the main patch is committed.

Yes, the new tests added by 0003 patch (skip_xid patch) use that fact.
After the error is shown in the view, we fetch the XID from the view
to specify as skip_xid. The tests just for the
pg_stat_subscription_errors view will be a subset of these tests. So
probably we can add it in 0001 patch and 0003 patch can extend the
tests so that it tests skip_xid option.

Regards,

--
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue