Horton

Re: long-standing data loss bug in initial sync of logical replication

Shlok Kyal <shlok.kyal.oss@gmail.com>

View thread
From: Shlok Kyal <shlok.kyal.oss@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Nitin Motiani <nitinmotiani@google.com>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2024-09-09T05:21:43Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix typo in test file name added in commit 4909b38af0.

  2. Fix data loss in logical replication.

  3. Avoid invalidating all RelationSyncCache entries on publication rename.

  4. Remove obsolete RECHECK keyword completely

  5. Backport BackgroundPsql perl test module

Attachments

On Mon, 2 Sept 2024 at 10:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Aug 30, 2024 at 3:06 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > Next I am planning to test solely on the logical decoding side and
> > will share the results.
> >
>
> Thanks, the next set of proposed tests makes sense to me. It will also
> be useful to generate some worst-case scenarios where the number of
> invalidations is more to see the distribution cost in such cases. For
> example, Truncate/Drop a table with 100 or 1000 partitions.
>
> --
> With Regards,
> Amit Kapila.

Also, I did testing with a table with partitions. To test for the
scenario where the number of invalidations are more than distribution.
Following is the test case:
1. Created a publisher on a single table, say 'tconc_1';
2. Created a second publisher on a partition table say 'tp';
3. Created 'tcount' partitions for the table 'tp'.
4. two sessions are running in parallel, let's say S1 and S2.
5. Begin a transaction in S1.
6. S1: Insert a row in table 'tconc_1'
   S2: BEGIN; TRUNCATE TABLE tp; COMMIT;
   With patch, this will add 'tcount * 3' invalidation messages to
transaction in session 1.
   S1: Insert a row in table 't_conc1'
7. COMMIT the transaction in S1.
8. run 'pg_logical_slot_get_binary_changes' to get the decoding changes.

Performance:
We see a degradation in performance. Results are an average of 5 runs.

count of partitions |   Head (sec) |    Fix (sec) |    Degradation (%)
-------------------------------------------------------------------------------------
1000                     |   0.114           |    0.118     |    3.50877193
5000                     |   0.502           |    0.522     |    3.984063745
10000                   |   1.012           |    1.024     |    1.185770751

I have also attached the test script here. And will also do further testing.

Thanks and Regards,
Shlok Kyal