Horton

Re: Conflict detection for update_deleted in logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
Cc: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Nisha Moond <nisha.moond412@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-01-01T05:36:15Z
Lists: pgsql-hackers 
On Thu, Dec 19, 2024 at 4:34 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Sunday, December 15, 2024 9:39 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> >
> > 5. The apply worker needs to at least twice get the publisher status message to
> > advance oldest_nonremovable_xid once. It then uses the remote_lsn of the last
> > such message to ensure that it has been applied locally. Such a remote_lsn
> > could be a much later value than required leading to delay in advancing
> > oldest_nonremovable_xid. How about if while first time processing the
> > publisher_status message on walsender, we get the
> > latest_transaction_in_commit by having a function
> > GetLatestTransactionIdInCommit() instead of
> > GetOldestTransactionIdInCommit() and then simply wait till that proc has
> > written commit WAL (aka wait till it clears DELAY_CHKPT_IN_COMMIT)?
> > Then get the latest LSN wrote and send that to apply worker waiting for the
> > publisher_status message. If this is feasible then we should be able to
> > advance oldest_nonremovable_xid with just one publisher_status message.
> > Won't that be an improvement over current? If so, we can even further try to
> > improve it by just using commit_LSN of the transaction returned by
> > GetLatestTransactionIdInCommit(). One idea is that we can try to use
> > MyProc->waitLSN which we are using in synchronous replication for our
> > purpose. See SyncRepWaitForLSN.
>
> I will do more performance tests on this and address if it improves
> the performance.
>

Did you check this idea? Again, thinking about this, I see a downside
to the new proposal. In the new proposal, the walsender needs to
somehow wait for the transactions in the commit which essentially
means that it may lead delay in decoding and sending the decoded WAL.
But it is still worth checking the impact of such a change, if nothing
else, we can add a short comment in the code to suggest such an
improvement is not worthwhile.

-- 
With Regards,
Amit Kapila.

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.