Horton

Re: Skipping logical replication transactions on subscriber side

David G. Johnston <david.g.johnston@gmail.com>

View thread View on postgr.es
From: "David G. Johnston" <david.g.johnston@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, vignesh C <vignesh21@gmail.com>, Greg Nancarrow <gregn4422@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, Alexey Lesovsky <lesovsky@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2022-01-22T16:47:14Z
Lists: pgsql-hackers 
On Sat, Jan 22, 2022 at 9:21 AM David G. Johnston <
david.g.johnston@gmail.com> wrote:

> On Sat, Jan 22, 2022 at 2:41 AM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
>
>>
>> > Additionally, the description for pg_stat_subscription_workers should
>> describe what happens once the transaction represented by last_error_xid
>> has either been successfully processed or skipped.  Does this "last error"
>> stick around until another error happens (which is hopefully very rare) or
>> does it reset to blanks?
>> >
>>
>> It will be reset only on subscription drop, otherwise, it will stick
>> around until another error happens.
>
>
> I really dislike the user experience this provides, and given it is new in
> v15 (and right now this table seems to exist solely to support this
> feature) changing this seems within the realm of possibility. I have to
> imagine these workers have a sense of local state that would just be "no
> errors, no need to touch pg_stat_subscription_workers at the end of this
> transaction's commit".  It would save a local state of the error_xid and if
> a successfully committed transaction has that xid it would clear the
> error.  The skip code path would also check for and see the matching xid
> value and clear the error.  Even if the local state thing doesn't work, one
> catalog lookup per transaction seems like potentially reasonable overhead
> to incur here.
>
>
It shouldn't even need to be that overhead intensive.  Once an error is
encountered the system stops.  By construction it must be told to redo, at
which point the information about "last error" is no longer relevant and
can be removed (for skipping the user/system will have already done
everything with the xid that is needed before the redo is issued).  In the
steady-state it then is simply empty until a new error arises at which
point it becomes populated again; and stays that way until the system goes
into redo mode as instructed by the user via one of several methods.

David J.

Commits

  1. Test ALIGNOF_DOUBLE==4 compatibility under ALIGNOF_DOUBLE==8.

  2. Reorder subskiplsn in pg_subscription to avoid alignment issues.

  3. Add ALTER SUBSCRIPTION ... SKIP.

  4. Optionally disable subscriptions on error.

  5. Update docs of logical replication for commit 8d74fc96db.

  6. Respect permissions within logical replication.

  7. Fix regression test failure caused by commit 8d74fc96db.

  8. Add a view to show the stats of subscription workers.

  9. Add logical change details to logical replication worker errcontext.

  10. Rename LOGICAL_REP_MSG_STREAM_END to LOGICAL_REP_MSG_STREAM_STOP.

  11. Fix typo in protocol.sgml.

  12. Remove unused argument in apply_handle_commit_internal().

  13. Fix replication of in-progress transactions in tablesync worker.

  14. Reorder pg_sequence columns to avoid alignment issue