Horton

Re: Skipping logical replication transactions on subscriber side

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Alexey Lesovsky <lesovsky@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-07-14T08:14:32Z
Lists: pgsql-hackers 
On Mon, Jul 12, 2021 at 8:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 12, 2021 at 11:13 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Jul 12, 2021 at 1:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, Jul 12, 2021 at 9:37 AM Alexey Lesovsky <lesovsky@gmail.com> wrote:
> > > >
> > > > On Mon, Jul 12, 2021 at 8:36 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >>
> > > >> >
> > > >> > Ok, looks nice. But I am curious how this will work in the case when there are two (or more) errors in the same subscription, but different relations?
> > > >> >
> > > >>
> > > >> We can't proceed unless the first error is resolved, so there
> > > >> shouldn't be multiple unresolved errors.
> > > >
> > > >
> > > > Ok. I thought multiple errors are possible when many tables are initialized using parallel workers (with max_sync_workers_per_subscription > 1).
> > > >
> > >
> > > Yeah, that is possible but that covers under the second condition
> > > mentioned by me and in such cases I think we should have separate rows
> > > for each tablesync. Is that right, Sawada-san or do you have something
> > > else in mind?
> >
> > Yeah, I agree to have separate rows for each table sync. The table
> > should not be processed by both the table sync worker and the apply
> > worker at a time so the pair of subscription OID and relation OID will
> > be unique. I think that we have a boolean column in the view,
> > indicating whether the error entry is reported by the table sync
> > worker or the apply worker, or maybe we also can have the action
> > column show "TABLE SYNC" if the error is reported by the table sync
> > worker.
> >
>
> Or similar to backend_type (text) in pg_stat_activity, we can have
> something like error_source (text) which will display apply worker or
> tablesync worker? I think if we have this column then even if there is
> a chance that both apply and sync worker operates on the same
> relation, we can identify it via this column.

Sounds good. I'll incorporate this in the next version patch that I'm
planning to submit this week.


Regards,

--
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue