Horton

Re: Conflict detection for update_deleted in logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Nisha Moond <nisha.moond412@gmail.com>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-01-08T09:52:48Z
Lists: pgsql-hackers 
On Wed, Jan 8, 2025 at 3:02 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Dec 19, 2024 at 11:11 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
> >
> > Here is further performance test analysis with v16 patch-set.
> >
> >
> > In the test scenarios already shared on -hackers [1], where pgbench was run only on the publisher node in a pub-sub setup, no performance degradation was observed on either node.
> >
> >
> >
> > In contrast, when pgbench was run only on the subscriber side with detect_update_deleted=on [2], the TPS performance was reduced due to dead tuple accumulation. This performance drop depended on the wal_receiver_status_interval—larger intervals resulted in more dead tuple accumulation on the subscriber node. However, after the improvement in patch v16-0002, which dynamically tunes the status request, the default TPS reduction was limited to only 1%.
> >
> >
> >
> > We performed more benchmarks with the v16-patches where pgbench was run on both the publisher and subscriber, focusing on TPS performance. To summarize the key observations:
> >
> >  - No performance impact on the publisher as dead tuple accumulation does not occur on the publisher.
>
> Nice. It means that frequently getting in-commit-phase transactions by
> the subscriber didn't have a negative impact on the publisher's
> performance.
>
> >
> >  - The performance is reduced on the subscriber side (TPS reduction (~50%) [3] ) due to dead tuple retention for the conflict detection when detect_update_deleted=on.
> >
> >  - Performance reduction happens only on the subscriber side, as workload on the publisher is pretty high and the apply workers must wait for the amount of transactions with earlier timestamps to be applied and flushed before advancing the non-removable XID to remove dead tuples.
>
> Assuming that the performance dip happened due to dead tuple retention
> for the conflict detection, would TPS on other databases also be
> affected?
>

As we use slot->xmin to retain dead tuples, shouldn't the impact be
global (means on all databases)? Or, maybe I am missing something.

> >
> >
> > [3] Test with pgbench run on both publisher and subscriber.
> >
> >
> >
> > Test setup:
> >
> > - Tests performed on pgHead + v16 patches
> >
> > - Created a pub-sub replication system.
> >
> > - Parameters for both instances were:
> >
> >
> >
> >    share_buffers = 30GB
> >
> >    min_wal_size = 10GB
> >
> >    max_wal_size = 20GB
> >
> >    autovacuum = false
>
> Since you disabled autovacuum on the subscriber, dead tuples created
> by non-hot updates are accumulated anyway regardless of
> detect_update_deleted setting, is that right?
>

I think hot-pruning mechanism during the update operation will remove
dead tuples even when autovacuum is disabled.

-- 
With Regards,
Amit Kapila.

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.