Horton

RE: Conflict detection for update_deleted in logical replication

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>

View thread View on postgr.es
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Amit Kapila' <amit.kapila16@gmail.com>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
Cc: Nisha Moond <nisha.moond412@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-01-07T11:22:04Z
Lists: pgsql-hackers

Attachments

Dear Amit, Hou,

> BTW, it is not clear how retaining dead tuples will help the detection
> update_origin_differs. Will it happen when the tuple is inserted or
> updated on the subscriber and then when we try to update the same
> tuple due to remote update, the commit_ts information of the xact is
> not available because the same is already removed by vacuum? This
> should happen for the update case for the new row generated by the
> update operation as that will be used in comparison. Can you please
> show it be a test case even if it is manual?

I've confirmed that retaining dead tuples is helpful for the origin_differ detection.
I considered a workload to prove it.

## Workload

I ran below steps:

1. Setup a publisher. There was a table and it had a tuple.
2. Setup a subscriber. There was a table and the same tuple was inserted by itself (not replicated).
   The subscription option was "copy_data = off", and GUC setting was "track_commit_timestamp=on".
3. Installed "xid_wraparound" extension on the subscriber.
4. Called `SELECT consume_xids(60000000);` on the subscriber side to advance xid.
5. Ran VACUUM FREEZE for all the databases on the subscriber side.
6. Updated the tuple on the publisher side.

The key idea is that commit_ts entries could be removed when the given
transaction ID is frozen. In this workload, 60 millon transaction IDs are consumed
and this is bit larger than vacuum_freeze_min_age.
I.e., step 5 can freeze a tuple which was inserted to the subscriber at step 2.
Then, we run VACUUM FREEZE command for databases to advance pg_database.datfrozenxid.
This command can teach commit_ts module that entries for old transactions can be truncated.

Attached script automate the test.

## Result

When detect_update_deleted of the subscription was set to false,
update_origin_differ was not detected on the subscriber.
In contrast, when detect_update_deleted was true, it was detectable:

```
LOG:  conflict detected on relation "public.foo": conflict=update_origin_differs
DETAIL:  Updating the row that was modified locally in transaction 745 at ...
         Existing local tuple (1, 1); remote tuple (1, 2); replica identity (a)=(1).
```

Based on the fact, we can conclude that the option is also helpful for detecting
origin_differ conflicts.

> Can't it happen for delete_origin_differs as well for the same reason?

Right. I've also tested with almost the same way as above, and I got the same result.
You can confirm by modifying the last statement in the attached script.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.