Horton

Re: Conflict detection for update_deleted in logical replication

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Nisha Moond <nisha.moond412@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2024-12-17T08:25:42Z
Lists: pgsql-hackers 
On Tue, Dec 17, 2024 at 8:54 AM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Monday, December 16, 2024 7:21 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> > So IIUC in step 2) we send the message and get the list of all the
> > transactions which are in the commit phase? What do you exactly mean by a
> > transaction which is in the commit phase?
>
> I was referring to transactions calling RecordTransactionCommit() and have
> entered the commit critical section. In the patch, we checked if the proc has
> marked the new flag DELAY_CHKPT_IN_COMMIT in 'MyProc->delayChkptFlags'.
>
> > Can I assume transactions which are currently running on the publisher?
>
> I think it's a subset of the running transactions. We only get the transactions
> in commit phase with the intention to avoid delays caused by waiting for
> long-running transactions to complete, which can result in the long retention
> of dead tuples.

Ok

> We decided to wait for running(committing) transactions due to the WAL/LSN
> inversion issue[1]. The original idea is to directly return the latest WAL
> write position without checking running transactions. But since there is a gap
> between when we acquire the commit_timestamp and the commit LSN, it's possible
> the transactions might have been assigned an earlier commit timestamp but have
> not yet written the commit WAL record.

Yes, that makes sense.

> > And in step 3) we wait for all the transactions to get committed which we saw
> > running (or in the commit phase) and we anyway don't worry about the newly
> > started transactions as they would not be problematic for us. And in step 4)
> > we would wait for all the flush location to reach "last received WAL
> > position", here my question is what exactly will be the "last received WAL
> > position" I assume it would be the position somewhere after the position of
> > the commit WAL of all the transaction we were interested on the publisher?
>
> Yes, your understanding is correct. It's a position after the position of all
> the interesting transactions. In the patch, we get the latest WAL write
> position(GetXLogWriteRecPtr()) in walsender after all interesting transactions
> have finished and reply it to apply worker.

Got it, thanks.


-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.