Horton

Re: Conflict detection for update_deleted in logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, Nisha Moond <nisha.moond412@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-01-16T10:01:55Z
Lists: pgsql-hackers 
On Wed, Jan 15, 2025 at 2:20 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> In the latest version, we implemented a simpler approach that allows the apply
> worker to directly advance the oldest_nonremovable_xid if the waiting time
> exceeds the newly introduced option's limit. I've named this option
> "max_conflict_retention_duration," as it aligns better with the conflict
> detection concept and the "retain_conflict_info" option.
>
> During the last phase (RCI_WAIT_FOR_LOCAL_FLUSH), the apply worker evaluates
> how much time it has spent waiting. If this duration exceeds the
> max_conflict_retention_duration, the worker directly advances the
> oldest_nonremovable_xid and logs a message indicating the forced advancement of
> the non-removable transaction ID.
>
> This approach is a bit like a time-based option that discussed before.
> Compared to the slot invalidation approach, this approach is simpler because we
> can avoid adding 1) new slot invalidation type due to apply lag, 2) new field
> lag_behind in shared memory (MyLogicalRepWorker) to indicate when the lag
> exceeds the limit, and 3) additional logic in the launcher to handle each
> worker's lag status.
>
> In the slot invalidation, user would be able to confirm if the current by
> checking if the slot in pg_replication_slot in invalidated or not, while in the
> simpler approach mentioned, user could only confirm that by checking the LOGs.
>

The user needs to check the LOGs corresponding to all subscriptions on
the node. I see the simplicity of the approach you used but still the
slot_invalidation idea sounds better to me on the grounds that it will
be convenient for users/DBA to know when to rely on the update_missing
type conflict if there is a valid and active slot with the name
'pg_conflict_detection' (or whatever name we decide to give) then
users can rely on the detected conflict. Sawada-San, and others, do
you have any preference on this matter?

Do we want to prohibit the combination copy_data as true and
retain_conflict_info=true?  I understand that with the new parameter
'max_conflict_retention_duration', for large copies slot would anyway
be invalidated but I don't want to give users more ways to see this
slot invalidated in the beginning itself. Similarly during ALTER
SUBSCRIPTION, if the initial synch is in progress, we can disallow
enabling retain_conflict_info. Later, if there is a real demand for
such a combination, we can always enable it.

-- 
With Regards,
Amit Kapila.

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.