Re: long-standing data loss bug in initial sync of logical replication

Amit Kapila <amit.kapila16@gmail.com>

From: Amit Kapila <amit.kapila16@gmail.com>

To: Shlok Kyal <shlok.kyal.oss@gmail.com>

Cc: Masahiko Sawada <sawada.mshk@gmail.com>, vignesh C <vignesh21@gmail.com>, Nitin Motiani <nitinmotiani@google.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2025-02-26T09:54:37Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix typo in test file name added in commit 4909b38af0.
- 50b8ad30f754 18.0 landed
- d96206f259d6 17.5 landed
- 9987c94662c2 16.9 landed
- 90bc4523fd47 15.13 landed
- bb1bc9fa962e 14.18 landed
- 4164d6976316 13.21 landed
Fix data loss in logical replication.
- 247ee94150b6 13.21 landed
- 4909b38af034 18.0 landed
- cadaf0ac4637 17.5 landed
- 9a2f8b4f01d5 16.9 landed
- 9f21be08e884 15.13 landed
- 0434033e8bb5 14.18 landed
Avoid invalidating all RelationSyncCache entries on publication rename.
- 3abe9dc18892 18.0 cited
Remove obsolete RECHECK keyword completely
- 7da1bdc2c2f1 18.0 cited
Backport BackgroundPsql perl test module
- 187b8991f70f 16.4 cited

On Wed, Feb 26, 2025 at 9:21 AM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> I have done the performance testing for cases where we distribute a
> small amount of invalidation messages to many concurrently decoded
> transactions.
> Here are results:
>
> Concurrent Txn |    Head (sec)    |    Patch (sec)      | Degradation in %
> ---------------------------------------------------------------------------------------------
> 50                      |   0.2627734      |   0.2654608        | 1.022706256
> 100                    |   0.4801048      |   0.4869254        | 1.420648158
> 500                    |   2.2170336      |   2.2438656        | 1.210265825
> 1000                  |   4.4957402      |   4.5282574        | 0.723289126
> 2000                  |   9.2013082      |   9.21164             | 0.112286207
>
> The steps I followed is:
> 1. Initially logical replication is setup.
> 2. Then we start 'n' number of concurrent transactions.
>     Each txn look like:
>     BEGIN;
>     Insert into t1 values(11);
> 3. Now we add two invalidation which will be distributed each
> transaction by running command:
>     ALTER PUBLICATION regress_pub1 DROP TABLE t1
>     ALTER PUBLICATION regress_pub1 ADD TABLE t1
> 4. Then run an insert for each txn. It will build cache for relation
> in each txn.
> 5. Commit Each transaction.
>
> I have also attached the script.
>

The tests are done using pub-sub setup which has some overhead of
logical replication as well. Can we try this test by fetching changes
via SQL API using pgoutput as plugin to see the impact?

-- 
With Regards,
Amit Kapila.