Horton

Re: Conflict detection for update_deleted in logical replication

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Nisha Moond <nisha.moond412@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-01-30T22:39:28Z
Lists: pgsql-hackers 
On Thu, Jan 23, 2025 at 3:47 AM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Wednesday, January 22, 2025 7:54 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> wrote:
> > On Saturday, January 18, 2025 11:45 AM Zhijie Hou (Fujitsu)
> > <houzj.fnst@fujitsu.com> wrote:
> > > I think invalidating the slot is OK and we could also let the apply
> > > worker to automatic recovery as suggested in [1].
> > >
> > > Here is the V24 patch set. I modified 0004 patch to implement the slot
> > > Invalidation part. Since the automatic recovery could be an
> > > optimization and the discussion is in progress, I didn't implement that part.
> >
> > The implementation is in progress and I will include it in next version.
> >
> > Here is the V25 patch set that includes the following change:
> >
> > 0001
> >
> > * Per off-list discussion with Amit, I added few comments to mention the
> > reason of skipping advancing xid when table sync is in progress and to mention
> > that the advancement will not be delayed if changes are filtered out on
> > publisher via row/table filter.
> >
> > 0004
> >
> > * Fixed a bug that the launcher would advance the slot.xmin when some apply
> >   workers have not yet started.
> >
> > * Fixed a bug that the launcher did not advance the slot.xmin even if one of the
> >   apply worker has stopped conflict retention due to the lag.
> >
> > * Add a retain_conflict_info column in the pg_stat_subscription view to
> >   indicate whether the apply worker is effectively retaining conflict
> >   information. The value is set to true only if retain_conflict_info is enabled
> >   for the associated subscription, and the retention duration for conflict
> >   detection by the apply worker has not exceeded
> >   max_conflict_retention_duration. Thanks Kuroda-san for contributing codes
> >   off-list.
>
> Here is V25 patch set which includes the following changes:
>
> 0004
> * Addressed Nisha's comments[1].
> * Fixed a cfbot failure[2] in the doc.

I have one question about the 0004 patch; it implemented
max_conflict_retntion_duration as a subscription parameter. But the
launcher invalidates the pg_conflict_detection slot only if all
subscriptions with retain_conflict_info stopped retaining dead tuples
due to the max_conflict_retention_duration parameter. Therefore, even
if users set the parameter to a low value to avoid table bloats, it
would not make sense if other subscriptions set it to a larger value.
Is my understanding correct?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.