Horton

Re: Skipping logical replication transactions on subscriber side

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Alexey Lesovsky <lesovsky@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-07-20T01:08:56Z
Lists: pgsql-hackers 
On Mon, Jul 19, 2021 at 5:47 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 19, 2021 at 12:10 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Sat, Jul 17, 2021 at 12:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > 1. How to clear apply worker errors. IIUC we've discussed that once
> > > the apply worker skipped the transaction we leave the error entry
> > > itself but clear its fields except for some fields such as failure
> > > counts. But given that the stats messages could be lost, how can we
> > > ensure to clear those error details? For table sync workers’ error, we
> > > can have autovacuum workers periodically check entires of
> > > pg_subscription_rel and clear the error entry if the table sync worker
> > > completes table sync (i.g., checking if srsubstate = ‘r’). But there
> > > is no such information for the apply workers and subscriptions. In
> > > addition to sending the message clearing the error details just after
> > > skipping the transaction, I thought that we can have apply workers
> > > periodically send the message clearing the error details but it seems
> > > not good.
> >
> > I think that the motivation behind the idea of leaving error entries
> > and clearing theirs some fields is that users can check if the error
> > is successfully resolved and the worker is working find. But we can
> > check it also in another way, for example, checking
> > pg_stat_subscription view. So is it worth considering leaving the
> > apply worker errors as they are?
> >
>
> I think so. Basically, we will send the clear message after skipping
> the exact but I think it is fine if that message is lost. At worst, it
> will be displayed as the last error details. If there is another error
> it will be overwritten or probably we should have a function *_reset()
> which allows the user to reset a particular subscription's error info.

That makes sense. I'll incorporate this idea in the next version patch.

Regards,

-- 
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue