Re: long-standing data loss bug in initial sync of logical replication

Benoit Lobréau <benoit.lobreau@dalibo.com>

From: Benoit Lobréau <benoit.lobreau@dalibo.com>

To: Amit Kapila <amit.kapila16@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>

Cc: "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>, Shlok Kyal <shlok.kyal.oss@gmail.com>, vignesh C <vignesh21@gmail.com>, Nitin Motiani <nitinmotiani@google.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2025-02-28T08:28:13Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix typo in test file name added in commit 4909b38af0.
- 50b8ad30f754 18.0 landed
- d96206f259d6 17.5 landed
- 9987c94662c2 16.9 landed
- 90bc4523fd47 15.13 landed
- bb1bc9fa962e 14.18 landed
- 4164d6976316 13.21 landed
Fix data loss in logical replication.
- 247ee94150b6 13.21 landed
- 4909b38af034 18.0 landed
- cadaf0ac4637 17.5 landed
- 9a2f8b4f01d5 16.9 landed
- 9f21be08e884 15.13 landed
- 0434033e8bb5 14.18 landed
Avoid invalidating all RelationSyncCache entries on publication rename.
- 3abe9dc18892 18.0 cited
Remove obsolete RECHECK keyword completely
- 7da1bdc2c2f1 18.0 cited
Backport BackgroundPsql perl test module
- 187b8991f70f 16.4 cited

Attachments

cache_inval_bug_v1.ods (application/vnd.oasis.opendocument.spreadsheet)

Hi,

It took me a while but I ran the test on my laptop with 20 runs per 
test. I asked for a dedicated server and will re-run the tests if/when I 
have it.

count of partitions |   Head (sec) |    Fix (sec) |    Degradation (%)
----------------------------------------------------------------------
1000                |       0,0265 |       0,028  |  5,66037735849054
5000                |       0,091  |       0,0945 |  3,84615384615385
10000               |       0,1795 |       0,1815 |  1,11420612813371

  Concurrent Txn |    Head (sec)    |    Patch (sec) | Degradation in %
  ---------------------------------------------------------------------
  50             |   0,1797647      |   0,1920949    |  6,85907744957
  100            |   0,3693029      |   0,3823425    |  3,53086856344
  500            |   1,62265755     |   1,91427485   | 17,97158617972
  1000           |   3,01388635     |   3,57678295   | 18,67676928162
  2000           |   7,0171877      |   6,4713304    |  8,43500897435

I'll try to run test2.pl later (right now it fails).

hope this helps.

-- 
Benoit Lobréau
Consultant
http://dalibo.com