Horton

Re: Skipping logical replication transactions on subscriber side

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-05-29T02:56:39Z
Lists: pgsql-hackers 
On Thu, May 27, 2021 at 7:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, May 27, 2021 at 1:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, May 27, 2021 at 2:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Okay, that makes sense but still not sure how will you identify if we
> > > need to reset XID in case of failure doing that in the previous
> > > attempt.
> >
> > It's a just idea but we can record the failed transaction with XID as
> > well as its commit LSN passed? The sequence I'm thinking is,
> >
> > 1. the worker records the XID and commit LSN of the failed transaction
> > to a catalog.
> >
>
> When will you record this info? I am not sure if we can try to update
> this when an error has occurred. We can think of using try..catch in
> apply worker and then record it in catch on error but would that be
> advisable? One random thought that occurred to me is to that apply
> worker notifies such information to the launcher (or maybe another
> process) which will log this information.

Yeah, I was concerned about that too and had the same idea. The
information still could not be written if the server crashes before
the launcher writes it. But I think it's an acceptable.

>
> > 2. the user specifies how to resolve that conflict transaction
> > (currently only 'skip' is supported) and writes to the catalog.
> > 3. the worker does the resolution method according to the catalog. If
> > the worker didn't start to apply those changes, it can skip the entire
> > transaction. If did, it rollbacks the transaction and ignores the
> > remaining.
> >
> > The worker needs neither to reset information of the last failed
> > transaction nor to mark the conflicted transaction as resolved. The
> > worker will ignore that information when checking the catalog if the
> > commit LSN is passed.
> >
>
> So won't this require us to check the required info in the catalog
> before applying each transaction? If so, that might be overhead, maybe
> we can build some cache of the highest commitLSN that can be consulted
> rather than the catalog table.

I think workers can cache that information when starts and invalidates
and reload the cache when the catalog gets updated.  Specifying to
skip XID will update the catalog, invalidating the cache.

> I think we need to think about when to
> remove rows for which conflict has been resolved as we can't let that
> information grow infinitely.

I guess we can update catalog tuples in place when another conflict
happens next time. The catalog tuple should be fixed size. The
already-resolved conflict will have the commit LSN older than its
replication origin's LSN.


Regards,

--
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue