Horton

Re: Skipping logical replication transactions on subscriber side

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-05-27T10:04:37Z
Lists: pgsql-hackers 
On Thu, May 27, 2021 at 1:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, May 27, 2021 at 2:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Okay, that makes sense but still not sure how will you identify if we
> > need to reset XID in case of failure doing that in the previous
> > attempt.
>
> It's a just idea but we can record the failed transaction with XID as
> well as its commit LSN passed? The sequence I'm thinking is,
>
> 1. the worker records the XID and commit LSN of the failed transaction
> to a catalog.
>

When will you record this info? I am not sure if we can try to update
this when an error has occurred. We can think of using try..catch in
apply worker and then record it in catch on error but would that be
advisable? One random thought that occurred to me is to that apply
worker notifies such information to the launcher (or maybe another
process) which will log this information.

> 2. the user specifies how to resolve that conflict transaction
> (currently only 'skip' is supported) and writes to the catalog.
> 3. the worker does the resolution method according to the catalog. If
> the worker didn't start to apply those changes, it can skip the entire
> transaction. If did, it rollbacks the transaction and ignores the
> remaining.
>
> The worker needs neither to reset information of the last failed
> transaction nor to mark the conflicted transaction as resolved. The
> worker will ignore that information when checking the catalog if the
> commit LSN is passed.
>

So won't this require us to check the required info in the catalog
before applying each transaction? If so, that might be overhead, maybe
we can build some cache of the highest commitLSN that can be consulted
rather than the catalog table. I think we need to think about when to
remove rows for which conflict has been resolved as we can't let that
information grow infinitely.

> > Also, I am thinking that instead of a stat view, do we need
> > to consider having a system table (pg_replication_conflicts or
> > something like that) for this because what if stats information is
> > lost (say either due to crash or due to udp packet loss), can we rely
> > on stats view for this?
>
> Yeah, it seems better to use a catalog.
>

Okay.

-- 
With Regards,
Amit Kapila.

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue