Horton

Re: Conflict detection for update_deleted in logical replication

shveta malik <shveta.malik@gmail.com>

View thread View on postgr.es
From: shveta malik <shveta.malik@gmail.com>
To: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Nisha Moond <nisha.moond412@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, vignesh C <vignesh21@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, shveta malik <shveta.malik@gmail.com>
Date: 2025-09-05T06:01:27Z
Lists: pgsql-hackers 
On Thu, Sep 4, 2025 at 3:30 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> Hi,
>
> As reported by Robert[1], it is worth adding a test for the race condition in
> the RecordTransactionCommitPrepared() function to reduce the risk of future code
> changes:
>
>         /*
>          * Note it is important to set committs value after marking ourselves as
>          * in the commit critical section (DELAY_CHKPT_IN_COMMIT). This is because
>          * we want to ensure all transactions that have acquired commit timestamp
>          * are finished before we allow the logical replication client to advance
>          * its xid which is used to hold back dead rows for conflict detection.
>          * See comments atop worker.c.
>          */
>          committs = GetCurrentTimestamp();
>
> While writing the test, I noticed a bug that conflict-relevant data could be
> prematurely removed before applying prepared transactions on the publisher that
> are in the commit phase. This occurred because GetOldestActiveTransactionId()
> was called on the publisher, which failed to account for the backend executing
> COMMIT PREPARED, as this backend does not have an xid stored in PGPROC.
>
> Since this issue overlaps with the race condition related to timestamp
> acquisition, I've prepared a bug fix along with a test for the race condition.
> The 0001 patch fixes this issue by introducing a new function that iterates over
> global transactions and identifies prepared transactions during the commit
> phase. 0002 added injection points and tap-test to test the bug and timestamp
> acquisition.
>

Thank You for the patches.Verified 001, works well. Just one minor comment:

1)
+ PGPROC    *proc = GetPGProcByNumber(gxact->pgprocno);
+ PGPROC    *commitproc;
We get first proc and then commitproc. proc is used only to get
databaseId, why don't we get databaseId from commitproc itself?

~~

Few trivial comments for 002:
1)
+# Test that publisher's transactions marked with DELAY_CHKPT_IN_COMMIT prevent
+# concurrently deleted tuples on the subscriber from being removed.

Here shall we also mention something like:
This test also acts as a safeguard to prevent developers from moving
the timestamp acquisition before marking DELAY_CHKPT_IN_COMMIT in
RecordTransactionCommitPrepared.

2)
# This test depends on an injection point to block the transaction commit after
# marking DELAY_CHKPT_IN_COMMIT flag.

Shall we say:
..to block the prepared transaction commit..

3)
+# Create a publisher node. Disable autovacuum to stablized the tests related to
+# manual VACUUM and transaction ID.

to stablized --> to stabilize

4)
+# Confirm that the dead tuple can be removed now
+my ($cmdret, $stdout, $stderr) =
+  $node_subscriber->psql('postgres', qq(VACUUM (verbose) public.tab;));
+
+ok($stderr =~ qr/1 are dead but not yet removable/,
+ 'the deleted column is non-removable');

can be removed now --> cannot be removed now

thanks
Shveta

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.