Horton

RE: Conflict detection for update_deleted in logical replication

Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>

View thread View on postgr.es
From: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>
To: Amit Kapila <amit.kapila16@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: Nisha Moond <nisha.moond412@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, shveta malik <shveta.malik@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2024-12-02T11:33:38Z
Lists: pgsql-hackers

Attachments

On Monday, December 2, 2024 12:14 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Fri, Nov 29, 2024 at 4:05 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > 02. maybe_advance_nonremovable_xid
> >
> > ```
> > +        case RCI_REQUEST_PUBLISHER_STATUS:
> > +            request_publisher_status(data);
> > +            break;
> > ```
> >
> > I think the part is not reachable because the transit
> >
> RCI_REQUEST_PUBLISHER_STATUS->RCI_WAIT_FOR_PUBLISHER_STATU
> S is done in
> > get_candidate_xid()->request_publisher_status().
> > Can we remove this?
> >
> 
> After changing phase to RCI_REQUEST_PUBLISHER_STATUS, we directly
> invoke request_publisher_status, and similarly, after changing phase to
> RCI_WAIT_FOR_LOCAL_FLUSH, we call wait_for_local_flush. Won't it be
> better that in both cases and other similar cases, we instead invoke
> maybe_advance_nonremovable_xid()? This will make
> maybe_advance_nonremovable_xid() the only function with the knowledge to
> take action based on phase rather than spreading the knowledge of
> phase-related actions to various functions. Then we should also add a
> comment at the end in request_publisher_status() where we change the
> phase but don't do anything. The comment should explain the reason for the
> same.

Agreed.

> 
> One more point, it seems on a busy server, the patch won't be able to advance
> nonremovable_xid. We should call
> maybe_advance_nonremovable_xid() at all the places where we call
> send_feedback() and additionally, we should also call it after applying some
> threshold number (say 100) of messages. The latter is to avoid the cases where
> we won't invoke the required functionality on a busy server with a large value of
> sender/receiver timeouts.

Right. I think instead of adding a threshold, we could directly check if it's
time to start next round of xid advancement attempt. Since we already get the
current timestamp in the loop of receiving msg, it would not be expensive to
check the time difference. I have used this approach in V13 and will test the
performance for it.

Here is the V13 patch set which addressed this and pending comments
in [1][2].

[1] https://www.postgresql.org/message-id/TYAPR01MB5692A6AC6EF2E22A2C9D099AF52A2%40TYAPR01MB5692.jpnprd01.prod.outlook.com
[2] https://www.postgresql.org/message-id/CAA4eK1L-J-iJJ6gpzUc9wJY6eGfBbCDmh%2B5fHUZoixnoFbJSNg%40mail.gmail.com

Best Regards,
Hou zj

Commits

  1. Fix intermittent BF failures in 035_conflicts.

  2. Resume conflict-relevant data retention automatically.

  3. Fix intermittent test failure introduced in 6456c6e2c4.

  4. Fix Coverity issue reported in commit a850be2fe.

  5. Add test to prevent premature removal of conflict-relevant data.

  6. Post-commit review fixes for 228c370868.

  7. Add max_retention_duration option to subscriptions.

  8. Detect and report update_deleted conflicts.

  9. Preserve conflict-relevant data during logical replication.

  10. Integrate FullTransactionIds deeper into two-phase code

  11. Improve checks for GUC recovery_target_timeline

  12. Prevent excessive delays before launching new logrep workers.

  13. Keep WAL segments by slot's last saved restart LSN

  14. Rework some code handling pg_subscription data in psql and pg_dump

  15. Use generateClonedIndexStmt to propagate CREATE INDEX to partitions.