Horton

Re: Conflict detection for update_deleted in logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Nisha Moond <nisha.moond412@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2025-02-10T09:15:22Z
Lists: pgsql-hackers 
On Mon, Feb 10, 2025 at 10:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Feb 7, 2025 at 11:17 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> >
> > > I'm not really sure that these behaviors are the expected behavior of
> > > users who set max_conflict_retention_duration to some subscriptions.
> > > Or I might have set the wrong expectation or assumption on this
> > > parameter. I'm fine with a subscription-level
> > > max_conflict_retention_duration if it's clear this option works as
> > > expected by users who want to use it.
> > >
> >
> > It seems you are not convinced to provide this parameter at the
> > subscription level and anyway providing it as GUC will simplify the
> > implementation and it would probably be easier for users to configure
> > rather than giving it at the subscription level for all subscriptions
> > that have set retain_conflict_info set to true. I guess in the future
> > we can provide it at the subscription level as well if there is a
> > clear use case for it. Does that make sense to you?
>
> Would it make sense to introduce a GUC parameter for this value, where
> subscribers can overwrite it for their specific subscriptions, but
> only up to the limit set by the GUC? This would allow flexibility in
> certain cases --subscribers could opt to wait for a shorter duration
> than the GUC value if needed.
>
> Although a concrete use case isn't immediately clear, consider a
> hypothetical scenario: Suppose a subscriber connected to Node1 must
> wait for long period to avoid an incorrect conflict decision. In such
> cases, it would rely on the default high value set by the GUC.
> However, since Node1 is generally responsive and rarely has
> long-running transactions, this long wait would only be necessary in
> rare cases. On the other hand, a subscriber pulling from Node2 may not
> require as stringent conflict detection. If Node2 frequently has
> long-running transactions, waiting too long could lead to excessive
> delays.
>
> The idea here is that the Node1 subscriber can wait for the full
> max_conflict_retention_duration set by the GUC when necessary, while
> the Node2 subscriber can choose a shorter wait time to avoid
> unnecessary delays caused by frequent long transactions.
>

I see that users can have some cases like this where it can be helpful
to provide the option to set max_conflict_retention_duration both at
GUC as well as a subscription parameter. However, I suggest let's go a
bit slower in adding more options for this particular stuff. In the
first version of this work, let's add a GUC and then let it bake for
some time after which we can discuss again adding a subscription
option based on some feedback from the field.

-- 
With Regards,
Amit Kapila.

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.