Horton

Re: Skipping logical replication transactions on subscriber side

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Greg Nancarrow <gregn4422@gmail.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, Alexey Lesovsky <lesovsky@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-11-02T05:34:57Z
Lists: pgsql-hackers 
On Mon, Nov 1, 2021 at 7:18 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Oct 29, 2021 at 8:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Fair enough. So statistics can be removed either by vacuum or drop
> > subscription. Also, if we go by this logic then there is no harm in
> > retaining the stat entries for tablesync errors. Why have different
> > behavior for apply and tablesync workers?
>
> My understanding is that the subscription worker statistics entry
> corresponds to workers (but not physical workers since the physical
> process is changed after restarting). So if the worker finishes its
> jobs, it is no longer necessary to show errors since further problems
> will not occur after that. Table sync worker’s job finishes when
> completing table copy (unless table sync is performed again by REFRESH
> PUBLICATION) whereas apply worker’s job finishes when the subscription
> is dropped.
>

Actually, I am not very sure how users can use the old error
information after we allowed skipping the conflicting xid. Say, if
they want to add/remove some constraints on the table based on
previous errors then they might want to refer to errors of both the
apply worker and table sync worker.

> Also, I’m concerned about a situation like where a lot of
> table sync failed. In which case, if we don’t drop table sync worker
> statistics after completing its job, we end up having a lot of entries
> in the view unless the subscription is dropped.
>

True, but the same could be said for apply workers where errors can be
accumulated over a period of time.

> >
> > I have another question in this regard. Currently, the reset function
> > seems to be resetting only the first stat entry for a subscription.
> > But can't we have multiple stat entries for a subscription considering
> > the view's cumulative nature?
>
> I might be missing your points but I think that with the current
> patch, the view has multiple entries for a subscription. That is,
> there is one apply worker stats and multiple table sync worker stats
> per subscription.
>

Can't we have multiple entries for one apply worker?

> And pg_stat_reset_subscription() function can reset
> any stats by specifying subscription OID and relation OID.
>

Say, if the user has supplied just subscription OID then isn't it
better to reset all the error entries for that subscription?

-- 
With Regards,
Amit Kapila.

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue