Horton

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

Shlok Kyal <shlok.kyal.oss@gmail.com>

View thread View on postgr.es
From: Shlok Kyal <shlok.kyal.oss@gmail.com>
To: Peter Smith <smithpb2250@gmail.com>
Cc: Melih Mutlu <m.melihmutlu@gmail.com>, vignesh C <vignesh21@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>, Melanie Plageman <melanieplageman@gmail.com>, "Wei Wang (Fujitsu)" <wangw.fnst@fujitsu.com>, "Yu Shi (Fujitsu)" <shiy.fnst@fujitsu.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, shveta malik <shveta.malik@gmail.com>
Date: 2024-01-10T09:29:22Z
Lists: pgsql-hackers 
Hi,

This patch is not applying on the HEAD. Please rebase and share the
updated patch.

Thanks and Regards
Shlok Kyal

On Wed, 10 Jan 2024 at 14:55, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Oops - now with attachments
>
> On Mon, Aug 21, 2023 at 5:56 PM Peter Smith <smithpb2250@gmail.com> wrote:
>>
>> Hi Melih,
>>
>> Last week we revisited your implementation of design#2. Vignesh rebased it, and then made a few other changes.
>>
>> PSA v28*
>>
>> The patch changes include:
>> * changed the logic slightly by setting recv_immediately(new variable), if this variable is set the main apply worker loop will not wait in this case.
>> * setting the relation state to ready immediately if there are no more incremental changes to be synced.
>> * receive the incremental changes if applicable and set the relation state to ready without waiting.
>> * reuse the worker if the worker is free before trying to start a new table sync worker
>> * restarting the tablesync worker only after wal_retrieve_retry_interval
>>
>> ~
>>
>> FWIW, we just wanted to share with you the performance measurements seen using this design#2 patch set:
>>
>> ======
>>
>> RESULTS (not busy tests)
>>
>> ------
>> 10 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           125     119     140     133
>> HEAD+v28*:      92      93      123     134
>> %improvement:   27%     22%     12%     -1%
>> ------
>> 100 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           1037    843     1109    1155
>> HEAD+v28*:      591     625     2616    2569
>> %improvement:   43%     26%     -136%   -122%
>> ------
>> 1000 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           15874   10047   9919    10338
>> HEAD+v28*:      33673   12199   9094    9896
>> %improvement:   -112%   -21%    8%      4%
>> ------
>> 2000 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           45266   24216   19395   19820
>> HEAD+v28*:      88043   21550   21668   22607
>> %improvement:  -95%     11%     -12%    -14%
>>
>> ~~~
>>
>> Note - the results were varying quite a lot in comparison to the HEAD
>> e.g. HEAD results are very consistent, but the v28* results observed are not
>> HEAD 1000 (2w): 15861, 15777, 16007, 15950, 15886, 15740, 15846, 15740, 15908, 15940
>> v28* 1000 (2w):  34214, 13679, 8792, 33289, 31976, 56071, 57042, 56163, 34058, 11969
>>
>> ------
>> Kind Regards,
>> Peter Smith.
>> Fujitsu Australia

Commits

  1. Refactor to split Apply and Tablesync Workers code.

  2. Fix possible logical replication crash.