Horton

Re: long-standing data loss bug in initial sync of logical replication

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, vignesh C <vignesh21@gmail.com>, Nitin Motiani <nitinmotiani@google.com>, Andres Freund <andres@anarazel.de>, Masahiko Sawada <sawada.mshk@gmail.com>
Date: 2025-03-15T06:24:40Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix typo in test file name added in commit 4909b38af0.

  2. Fix data loss in logical replication.

  3. Avoid invalidating all RelationSyncCache entries on publication rename.

  4. Remove obsolete RECHECK keyword completely

  5. Backport BackgroundPsql perl test module

On Thu, Mar 13, 2025 at 2:12 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
>
> Workload C. DDL is happening on publication but on unrelated table
> ============================================
> We did not run the workload because we expected this could be same results as D.
> 588acf6 is needed to optimize the workload.
>
> -----
>
> Workload D. DDL is happening on the related published table,
>                         and one insert is done per invalidation
> =========================================
> This workload had huge regression same as the master branch. This is expected
> because distributed invalidation messages require all concurrent transactions
> to rebuild relsync caches.
>
> Concurrent txn     | Head (sec)   | Patch (sec)  | Degradation (%)
> ------------------ | ------------ | ------------ | ----------------
> 50                 | 0.013496     | 0.015588     | 15.5034
> 100                | 0.015112     | 0.018868     | 24.8517
> 500                | 0.018483     | 0.038714     | 109.4536
> 1000               | 0.023402     | 0.063735     | 172.3524
> 2000               | 0.031596     | 0.110860     | 250.8720
>

IIUC, workloads C and D will have regression in back branches, and
HEAD will have regression only for workload D. We have avoided
workload C regression in HEAD via commits 7c99dc587a and 3abe9dc188.
We can backpatch those commits if required, but I think it is better
not to do those as scenarios C and D won't be that common, and we
should go ahead with the fix as it is. In the future, if we get any
way to avoid regression due to scenario-D, then we can do that for the
HEAD branch.

Thoughts?

-- 
With Regards,
Amit Kapila.