Horton

Re: Skipping logical replication transactions on subscriber side

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, Alexey Lesovsky <lesovsky@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>, Greg Nancarrow <gregn4422@gmail.com>
Date: 2021-10-19T03:37:56Z
Lists: pgsql-hackers 
On Tue, Oct 19, 2021 at 8:23 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Oct 18, 2021 at 6:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Oct 11, 2021 at 1:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Sun, Oct 10, 2021 at 11:04 PM Peter Eisentraut
> > > <peter.eisentraut@enterprisedb.com> wrote:
> > > >
> > > > On 04.10.21 02:31, Masahiko Sawada wrote:
> > > > > I guess disabling subscriptions on error/conflict and skipping the
> > > > > particular transactions are somewhat different types of functions.
> > > > > Disabling subscriptions on error/conflict seems likes a setting
> > > > > parameter of subscriptions. The users might want to specify this
> > > > > option at creation time. Whereas, skipping the particular transaction
> > > > > is a repair function that the user might want to use on the spot in
> > > > > case of a failure. I’m concerned a bit that combining these functions
> > > > > to one syntax could confuse the users.
> > > >
> > > > Also, would the skip option be dumped and restored using pg_dump?  Maybe
> > > > there is an argument for yes, but if not, then we probably need a
> > > > different path of handling it separate from the more permanent options.
> > >
> > > Good point. I don’t think the skip option should be dumped and
> > > restored using pg_dump since the utilization of transaction ids in
> > > another installation is different.
> > >
> >
> > This is a xid of publisher which subscriber wants to skip. So, even if
> > one restores the subscriber data in a different installation why would
> > it matter till it points to the same publisher?
> >
> > Either way, can't we handle this in pg_dump?
>
> Because of backups (dumps), I think we cannot expect that the user
> restore it somewhere soon. If the dump is restored several months
> later, the publisher could be a different installation (by rebuilding
> from scratch) or XID of the publisher could already be wrapped around.
> It might be useful to dump the skip_xid by pg_dump in some cases, but
> I think it should be optional if we want to do that.
>

Agreed, I think it depends on the use case, so we can keep it
optional, or maybe in the initial version let's not dump it, and only
if we later see the use case then we can add an optional parameter in
pg_dump. Do you think we need any special handling if we decide not to
dump it? I think if we decide to dump it either optionally or
otherwise, then we do need changes in pg_dump.

-- 
With Regards,
Amit Kapila.

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue