Horton

RE: Conflict detection for update_deleted in logical replication

Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>

View thread View on postgr.es
From: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, Nisha Moond <nisha.moond412@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-01-23T11:47:05Z
Lists: pgsql-hackers

Attachments

On Wednesday, January 22, 2025 7:54 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> wrote:
> On Saturday, January 18, 2025 11:45 AM Zhijie Hou (Fujitsu)
> <houzj.fnst@fujitsu.com> wrote:
> > I think invalidating the slot is OK and we could also let the apply
> > worker to automatic recovery as suggested in [1].
> >
> > Here is the V24 patch set. I modified 0004 patch to implement the slot
> > Invalidation part. Since the automatic recovery could be an
> > optimization and the discussion is in progress, I didn't implement that part.
> 
> The implementation is in progress and I will include it in next version.
> 
> Here is the V25 patch set that includes the following change:
> 
> 0001
> 
> * Per off-list discussion with Amit, I added few comments to mention the
> reason of skipping advancing xid when table sync is in progress and to mention
> that the advancement will not be delayed if changes are filtered out on
> publisher via row/table filter.
> 
> 0004
> 
> * Fixed a bug that the launcher would advance the slot.xmin when some apply
>   workers have not yet started.
> 
> * Fixed a bug that the launcher did not advance the slot.xmin even if one of the
>   apply worker has stopped conflict retention due to the lag.
> 
> * Add a retain_conflict_info column in the pg_stat_subscription view to
>   indicate whether the apply worker is effectively retaining conflict
>   information. The value is set to true only if retain_conflict_info is enabled
>   for the associated subscription, and the retention duration for conflict
>   detection by the apply worker has not exceeded
>   max_conflict_retention_duration. Thanks Kuroda-san for contributing codes
>   off-list.

Here is V25 patch set which includes the following changes:

0004
* Addressed Nisha's comments[1].
* Fixed a cfbot failure[2] in the doc.

New 0005
* The patch allows the launcher to drop and re-create the invalidated slot, if at
least one apply worker has confirmed that the retention duration is now within
the max_conflict_retention_duration.

[1] https://www.postgresql.org/message-id/CABdArM69ukHVQMmnyKpwbYH8aLhXu-d4fGbSiTkq7BBgRWe0TA%40mail.gmail.com
[2] https://cirrus-ci.com/task/5006366856708096?logs=docs_build#L299

Best Regards,
Hou zj

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.