Horton

Thread

Commits

  1. Refactor to split Apply and Tablesync Workers code.

  2. Fix possible logical replication crash.

  1. [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-07-05T13:50:20Z

    Hi hackers,

I created a patch to reuse tablesync workers and their replication slots
for more tables that are not synced yet. So that overhead of creating and
dropping workers/replication slots can be reduced.

Current version of logical replication has two steps: tablesync and apply.
In tablesync step, apply worker creates a tablesync worker for each table
and those tablesync workers are killed when they're done with their
associated table. (the number of tablesync workers running at the same time
is limited by "max_sync_workers_per_subscription")
Each tablesync worker also creates a replication slot on publisher during
its lifetime and drops the slot before exiting.

The purpose of this patch is getting rid of the overhead of
creating/killing a new worker (and replication slot) for each table.
It aims to reuse tablesync workers and their replication slots so that
tablesync workers can copy multiple tables from publisher to subscriber
during their lifetime.

The benefits of reusing tablesync workers can be significant if tables are
empty or close to empty.
In an empty table case, spawning tablesync workers and handling replication
slots are where the most time is spent since the actual copy phase takes
too little time.


The changes in the behaviour of tablesync workers with this patch as
follows:
1- After tablesync worker is done with syncing the current table, it takes
a lock and fetches tables in init state
2- it looks for a table that is not already being synced by another worker
from the tables with init state
3- If it founds one, updates its state for the new table and loops back to
beginning to start syncing
4- If no table found, it drops the replication slot and exits


With those changes, I did some benchmarking to see if it improves anything.
This results compares this patch with the latest version of master branch.
"max_sync_workers_per_subscription" is set to 2 as default.
Got some results simply averaging timings from 5 consecutive runs for each
branch.

First, tested logical replication with empty tables.
10 tables
----------------
- master:    286.964 ms
- the patch:    116.852 ms

100 tables
----------------
- master:    2785.328 ms
- the patch:    706.817 ms

10K tables
----------------
- master:    39612.349 ms
- the patch:    12526.981 ms


Also tried replication tables with some data
10 tables loaded with 10MB data
----------------
- master:    1517.714 ms
- the patch:    1399.965 ms

100 tables loaded with 10MB data
----------------
- master:    16327.229 ms
- the patch:    11963.696 ms


Then loaded more data
10 tables loaded with 100MB data
----------------
- master:    13910.189 ms
- the patch:    14770.982 ms

100 tables loaded with 100MB data
----------------
- master:    146281.457 ms
- the patch:    156957.512


If tables are mostly empty, the improvement can be significant - up to 3x
faster logical replication.
With some data loaded, it can still be faster to some extent.
When the table size increases more, the advantage of reusing workers
becomes insignificant.


I would appreciate your comments and suggestions.Thanks in advance for
reviewing.

Best,
Melih

  2. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-06T03:36:13Z

    On Tue, Jul 5, 2022 at 7:20 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> I created a patch to reuse tablesync workers and their replication slots for more tables that are not synced yet. So that overhead of creating and dropping workers/replication slots can be reduced.
>
> Current version of logical replication has two steps: tablesync and apply.
> In tablesync step, apply worker creates a tablesync worker for each table and those tablesync workers are killed when they're done with their associated table. (the number of tablesync workers running at the same time is limited by "max_sync_workers_per_subscription")
> Each tablesync worker also creates a replication slot on publisher during its lifetime and drops the slot before exiting.
>
> The purpose of this patch is getting rid of the overhead of creating/killing a new worker (and replication slot) for each table.
> It aims to reuse tablesync workers and their replication slots so that tablesync workers can copy multiple tables from publisher to subscriber during their lifetime.
>
> The benefits of reusing tablesync workers can be significant if tables are empty or close to empty.
> In an empty table case, spawning tablesync workers and handling replication slots are where the most time is spent since the actual copy phase takes too little time.
>
>
> The changes in the behaviour of tablesync workers with this patch as follows:
> 1- After tablesync worker is done with syncing the current table, it takes a lock and fetches tables in init state
> 2- it looks for a table that is not already being synced by another worker from the tables with init state
> 3- If it founds one, updates its state for the new table and loops back to beginning to start syncing
> 4- If no table found, it drops the replication slot and exits
>

How would you choose the slot name for the table sync, right now it
contains the relid of the table for which it needs to perform sync?
Say, if we ignore to include the appropriate identifier in the slot
name, we won't be able to resue/drop the slot after restart of table
sync worker due to an error.

>
> With those changes, I did some benchmarking to see if it improves anything.
> This results compares this patch with the latest version of master branch. "max_sync_workers_per_subscription" is set to 2 as default.
> Got some results simply averaging timings from 5 consecutive runs for each branch.
>
> First, tested logical replication with empty tables.
> 10 tables
> ----------------
> - master:    286.964 ms
> - the patch:    116.852 ms
>
> 100 tables
> ----------------
> - master:    2785.328 ms
> - the patch:    706.817 ms
>
> 10K tables
> ----------------
> - master:    39612.349 ms
> - the patch:    12526.981 ms
>
>
> Also tried replication tables with some data
> 10 tables loaded with 10MB data
> ----------------
> - master:    1517.714 ms
> - the patch:    1399.965 ms
>
> 100 tables loaded with 10MB data
> ----------------
> - master:    16327.229 ms
> - the patch:    11963.696 ms
>
>
> Then loaded more data
> 10 tables loaded with 100MB data
> ----------------
> - master:    13910.189 ms
> - the patch:    14770.982 ms
>
> 100 tables loaded with 100MB data
> ----------------
> - master:    146281.457 ms
> - the patch:    156957.512
>
>
> If tables are mostly empty, the improvement can be significant - up to 3x faster logical replication.
> With some data loaded, it can still be faster to some extent.
>

These results indicate that it is a good idea, especially for very small tables.

> When the table size increases more, the advantage of reusing workers becomes insignificant.
>

It seems from your results that performance degrades for large
relations. Did you try to investigate the reasons for the same?

-- 
With Regards,
Amit Kapila.

  3. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-07-06T08:17:29Z

    On Wed, Jul 6, 2022 at 9:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> How would you choose the slot name for the table sync, right now it
> contains the relid of the table for which it needs to perform sync?
> Say, if we ignore to include the appropriate identifier in the slot
> name, we won't be able to resue/drop the slot after restart of table
> sync worker due to an error.

I had a quick look into the patch and it seems it is using the worker
array index instead of relid while forming the slot name, and I think
that make sense, because now whichever worker is using that worker
index can reuse the slot created w.r.t that index.

> >
> > With those changes, I did some benchmarking to see if it improves anything.
> > This results compares this patch with the latest version of master branch. "max_sync_workers_per_subscription" is set to 2 as default.
> > Got some results simply averaging timings from 5 consecutive runs for each branch.
> >
> > First, tested logical replication with empty tables.
> > 10 tables
> > ----------------
> > - master:    286.964 ms
> > - the patch:    116.852 ms
> >
> > 100 tables
> > ----------------
> > - master:    2785.328 ms
> > - the patch:    706.817 ms
> >
> > 10K tables
> > ----------------
> > - master:    39612.349 ms
> > - the patch:    12526.981 ms
> >
> >
> > Also tried replication tables with some data
> > 10 tables loaded with 10MB data
> > ----------------
> > - master:    1517.714 ms
> > - the patch:    1399.965 ms
> >
> > 100 tables loaded with 10MB data
> > ----------------
> > - master:    16327.229 ms
> > - the patch:    11963.696 ms
> >
> >
> > Then loaded more data
> > 10 tables loaded with 100MB data
> > ----------------
> > - master:    13910.189 ms
> > - the patch:    14770.982 ms
> >
> > 100 tables loaded with 100MB data
> > ----------------
> > - master:    146281.457 ms
> > - the patch:    156957.512
> >
> >
> > If tables are mostly empty, the improvement can be significant - up to 3x faster logical replication.
> > With some data loaded, it can still be faster to some extent.
> >
>
> These results indicate that it is a good idea, especially for very small tables.
>
> > When the table size increases more, the advantage of reusing workers becomes insignificant.
> >
>
> It seems from your results that performance degrades for large
> relations. Did you try to investigate the reasons for the same?

Yeah, that would be interesting to know that why there is a drop in some cases.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  4. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-06T09:18:43Z

    On Wed, Jul 6, 2022 at 1:47 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Jul 6, 2022 at 9:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > How would you choose the slot name for the table sync, right now it
> > contains the relid of the table for which it needs to perform sync?
> > Say, if we ignore to include the appropriate identifier in the slot
> > name, we won't be able to resue/drop the slot after restart of table
> > sync worker due to an error.
>
> I had a quick look into the patch and it seems it is using the worker
> array index instead of relid while forming the slot name, and I think
> that make sense, because now whichever worker is using that worker
> index can reuse the slot created w.r.t that index.
>

I think that won't work because each time on restart the slot won't be
fixed. Now, it is possible that we may drop the wrong slot if that
state of copying rel is SUBREL_STATE_DATASYNC. Also, it is possible
that while creating a slot, we fail because the same name slot already
exists due to some other worker which has created that slot has been
restarted. Also, what about origin_name, won't that have similar
problems? Also, if the state is already SUBREL_STATE_FINISHEDCOPY, if
the slot is not the same as we have used in the previous run of a
particular worker, it may start WAL streaming from a different point
based on the slot's confirmed_flush_location.

-- 
With Regards,
Amit Kapila.

  5. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-07-06T10:40:10Z

    On Wed, Jul 6, 2022 at 2:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jul 6, 2022 at 1:47 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Jul 6, 2022 at 9:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > How would you choose the slot name for the table sync, right now it
> > > contains the relid of the table for which it needs to perform sync?
> > > Say, if we ignore to include the appropriate identifier in the slot
> > > name, we won't be able to resue/drop the slot after restart of table
> > > sync worker due to an error.
> >
> > I had a quick look into the patch and it seems it is using the worker
> > array index instead of relid while forming the slot name, and I think
> > that make sense, because now whichever worker is using that worker
> > index can reuse the slot created w.r.t that index.
> >
>
> I think that won't work because each time on restart the slot won't be
> fixed. Now, it is possible that we may drop the wrong slot if that
> state of copying rel is SUBREL_STATE_DATASYNC.

So it will drop the previous slot the worker at that index was using,
so it is possible that on that slot some relation was at
SUBREL_STATE_FINISHEDCOPY or so and we will drop that slot.  Because
now relid and replication slot association is not 1-1 so it would be
wrong to drop based on the relstate which is picked by this worker.
In short it makes sense what you have pointed out.

Also, it is possible
> that while creating a slot, we fail because the same name slot already
> exists due to some other worker which has created that slot has been
> restarted. Also, what about origin_name, won't that have similar
> problems? Also, if the state is already SUBREL_STATE_FINISHEDCOPY, if
> the slot is not the same as we have used in the previous run of a
> particular worker, it may start WAL streaming from a different point
> based on the slot's confirmed_flush_location.

Yeah this is also true, when a tablesync worker has to do catch up
after completing the copy then it might stream from the wrong lsn.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  6. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-07-08T16:56:23Z

    Hi Amit and Dilip,

Thanks for the replies.


> > I had a quick look into the patch and it seems it is using the worker
> > array index instead of relid while forming the slot name
>

Yes, I changed the slot names so they include slot index instead of
relation id.
This was needed because I aimed to separate replication slots from
relations.

I think that won't work because each time on restart the slot won't be
> fixed. Now, it is possible that we may drop the wrong slot if that
> state of copying rel is SUBREL_STATE_DATASYNC. Also, it is possible
> that while creating a slot, we fail because the same name slot already
> exists due to some other worker which has created that slot has been
> restarted. Also, what about origin_name, won't that have similar
> problems? Also, if the state is already SUBREL_STATE_FINISHEDCOPY, if
> the slot is not the same as we have used in the previous run of a
> particular worker, it may start WAL streaming from a different point
> based on the slot's confirmed_flush_location.
>

You're right Amit. In case of a failure, tablesync phase of a relation may
continue with different worker and replication slot due to this change in
naming.
Seems like the same replication slot should be used from start to end for a
relation during tablesync. However, creating/dropping replication slots can
be a major overhead in some cases.
It would be nice if these slots are somehow reused.

To overcome this issue, I've been thinking about making some changes in my
patch.
So far, my proposal would be as follows:

Slot naming can be like pg_<sub_id>_<worker_pid> instead of
pg_<sub_id>_<slot_index>. This way each worker can use the same replication
slot during their lifetime.
But if a worker is restarted, then it will switch to a new replication slot
since its pid has changed.

pg_subscription_rel catalog can store replication slot name for each
non-ready relation. Then we can find the slot needed for that particular
relation to complete tablesync.
If a worker syncs a relation without any error, everything works well and
this new replication slot column from the catalog will not be needed.
However if a worker is restarted due to a failure, the previous run of that
worker left its slot behind since it did not exit properly.
And the restarted worker (with a different pid) will see that the relation
is actually in  SUBREL_STATE_FINISHEDCOPY and want to proceed for the
catchup step.
Then the worker can look for that particular relation's replication slot
from pg_subscription_rel catalog (slot name should be there since relation
state is not ready). And tablesync can proceed with that slot.

There might be some cases where some replication slots are left behind. An
example to such cases would be when the slot is removed from
pg_subscription_rel catalog and detached from any relation, but tha slot
actually couldn't be dropped for some reason. For such cases, a slot
cleanup logic is needed. This cleanup can also be done by tablesync workers.
Whenever a tablesync worker is created, it can look for existing
replication slots that do not belong to any relation and any worker (slot
name has pid for that), and drop those slots if it finds any.

What do you think about this new way of handling slots? Do you see any
points of concern?

I'm currently working on adding this change into the patch. And would
appreciate any comment.

Thanks,
Melih

  7. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-07-08T16:59:43Z

    >
> It seems from your results that performance degrades for large
> relations. Did you try to investigate the reasons for the same?
>

I have not tried to investigate the performance degradation for large
relations yet.
Once I'm done with changes for the slot usage, I'll look into this and come
with more findings.

Thanks,
Melih

  8. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-12T12:24:59Z

    On Fri, Jul 8, 2022 at 10:26 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
>> I think that won't work because each time on restart the slot won't be
>> fixed. Now, it is possible that we may drop the wrong slot if that
>> state of copying rel is SUBREL_STATE_DATASYNC. Also, it is possible
>> that while creating a slot, we fail because the same name slot already
>> exists due to some other worker which has created that slot has been
>> restarted. Also, what about origin_name, won't that have similar
>> problems? Also, if the state is already SUBREL_STATE_FINISHEDCOPY, if
>> the slot is not the same as we have used in the previous run of a
>> particular worker, it may start WAL streaming from a different point
>> based on the slot's confirmed_flush_location.
>
>
> You're right Amit. In case of a failure, tablesync phase of a relation may continue with different worker and replication slot due to this change in naming.
> Seems like the same replication slot should be used from start to end for a relation during tablesync. However, creating/dropping replication slots can be a major overhead in some cases.
> It would be nice if these slots are somehow reused.
>
> To overcome this issue, I've been thinking about making some changes in my patch.
> So far, my proposal would be as follows:
>
> Slot naming can be like pg_<sub_id>_<worker_pid> instead of pg_<sub_id>_<slot_index>. This way each worker can use the same replication slot during their lifetime.
> But if a worker is restarted, then it will switch to a new replication slot since its pid has changed.
>

I think using worker_pid also has similar risks of mixing slots from
different workers because after restart same worker_pid could be
assigned to a totally different worker. Can we think of using a unique
64-bit number instead? This will be allocated when each workers
started for the very first time and after that we can refer catalog to
find it as suggested in the idea below.

> pg_subscription_rel catalog can store replication slot name for each non-ready relation. Then we can find the slot needed for that particular relation to complete tablesync.
>

Yeah, this is worth investigating. However, instead of storing the
slot_name, we can store just the unique number (as suggested above).
We should use the same for the origin name as well.

> If a worker syncs a relation without any error, everything works well and this new replication slot column from the catalog will not be needed.
> However if a worker is restarted due to a failure, the previous run of that worker left its slot behind since it did not exit properly.
> And the restarted worker (with a different pid) will see that the relation is actually in  SUBREL_STATE_FINISHEDCOPY and want to proceed for the catchup step.
> Then the worker can look for that particular relation's replication slot from pg_subscription_rel catalog (slot name should be there since relation state is not ready). And tablesync can proceed with that slot.
>
> There might be some cases where some replication slots are left behind. An example to such cases would be when the slot is removed from pg_subscription_rel catalog and detached from any relation, but tha slot actually couldn't be dropped for some reason. For such cases, a slot cleanup logic is needed. This cleanup can also be done by tablesync workers.
> Whenever a tablesync worker is created, it can look for existing replication slots that do not belong to any relation and any worker (slot name has pid for that), and drop those slots if it finds any.
>

This sounds tricky. Why not first drop slot/origin and then detach it
from pg_subscription_rel? On restarts, it is possible that we may
error out after dropping the slot or origin but before updating the
catalog entry but in such case we can ignore missing slot/origin and
detach them from pg_subscription_rel. Also, if we use the unique
number as suggested above, I think even if we don't remove it after
the relation state is ready, it should be okay.

> What do you think about this new way of handling slots? Do you see any points of concern?
>
> I'm currently working on adding this change into the patch. And would appreciate any comment.
>

Thanks for making progress!

-- 
With Regards,
Amit Kapila.

  9. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-07-27T10:26:10Z

    Hi Amit,

I updated the patch in order to prevent the problems that might be caused
by using different replication slots for syncing a table.
As suggested in previous emails, replication slot names are stored in the
catalog. So slot names can be reached later and it is ensured
that same replication slot is used during tablesync step of a table.

With the current version of the patch:
-. "srrelslotname" column is introduced into pg_subscibtion_rel catalog. It
stores the slot name for tablesync

-. Tablesync worker logic is now as follows:
1. Tablesync worker is launched by apply worker for a table.
2. Worker generates a default replication slot name for itself. Slot name
includes subid and worker pid for tracking purposes.
3. If table has a slot name value in the catalog:

i. If the table state is DATASYNC, drop the replication slot from the
catalog and proceed tablesync with a new slot.

ii. If the table state is FINISHEDCOPY, use the replicaton slot from the
catalog, do not create a new slot.

4. Before worker moves to new table, drop any replication slot that are
retrieved from the catalog and used.
5. In case of no table left to sync, drop the replication slot of that sync
worker with worker pid if it exists. (It's possible that a sync worker do
not create a replication slot for itself but uses slots read from the
catalog in each iteration)


I think using worker_pid also has similar risks of mixing slots from
> different workers because after restart same worker_pid could be
> assigned to a totally different worker. Can we think of using a unique
> 64-bit number instead? This will be allocated when each workers
> started for the very first time and after that we can refer catalog to
> find it as suggested in the idea below.
>

I'm not sure how likely to have colliding pid's for different tablesync
workers in the same subscription.
Though ,having pid in slot name makes it easier to track which slot belongs
to which worker. That's why I kept using pid in slot names.
But I think it should be simple to switch to using a unique 64-bit number.
So I can remove pid's from slot names, if you think that it would be
better.


> We should use the same for the origin name as well.
>

I did not really change anything related to origin names. Origin names are
still the same and include relation id. What do you think would be an issue
with origin names in this patch?


> This sounds tricky. Why not first drop slot/origin and then detach it
> from pg_subscription_rel? On restarts, it is possible that we may
> error out after dropping the slot or origin but before updating the
> catalog entry but in such case we can ignore missing slot/origin and
> detach them from pg_subscription_rel. Also, if we use the unique
> number as suggested above, I think even if we don't remove it after
> the relation state is ready, it should be okay.
>

Right, I did not add an additional slot cleanup step. The patch now drops
the slot when we're done with it and then removes it from the catalog.


Thanks,
Melih

  10. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-28T14:01:58Z

    On Wed, Jul 27, 2022 at 3:56 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Amit,
>
> I updated the patch in order to prevent the problems that might be caused by using different replication slots for syncing a table.
> As suggested in previous emails, replication slot names are stored in the catalog. So slot names can be reached later and it is ensured
> that same replication slot is used during tablesync step of a table.
>
> With the current version of the patch:
> -. "srrelslotname" column is introduced into pg_subscibtion_rel catalog. It stores the slot name for tablesync
>
> -. Tablesync worker logic is now as follows:
> 1. Tablesync worker is launched by apply worker for a table.
> 2. Worker generates a default replication slot name for itself. Slot name includes subid and worker pid for tracking purposes.
> 3. If table has a slot name value in the catalog:
>
> i. If the table state is DATASYNC, drop the replication slot from the catalog and proceed tablesync with a new slot.
>
> ii. If the table state is FINISHEDCOPY, use the replicaton slot from the catalog, do not create a new slot.
>
> 4. Before worker moves to new table, drop any replication slot that are retrieved from the catalog and used.
>

Why after step 4, do you need to drop the replication slot? Won't just
clearing the required info from the catalog be sufficient?

> 5. In case of no table left to sync, drop the replication slot of that sync worker with worker pid if it exists. (It's possible that a sync worker do not create a replication slot for itself but uses slots read from the catalog in each iteration)
>
>
>> I think using worker_pid also has similar risks of mixing slots from
>> different workers because after restart same worker_pid could be
>> assigned to a totally different worker. Can we think of using a unique
>> 64-bit number instead? This will be allocated when each workers
>> started for the very first time and after that we can refer catalog to
>> find it as suggested in the idea below.
>
>
> I'm not sure how likely to have colliding pid's for different tablesync workers in the same subscription.
>

Hmm, I think even if there is an iota of a chance which I think is
there, we can't use worker_pid. Assume, that if the same worker_pid is
assigned to another worker once the worker using it got an error out,
the new worker will fail as soon as it will try to create a
replication slot.

> Though ,having pid in slot name makes it easier to track which slot belongs to which worker. That's why I kept using pid in slot names.
> But I think it should be simple to switch to using a unique 64-bit number. So I can remove pid's from slot names, if you think that it would be better.
>

I feel it would be better or maybe we need to think of some other
identifier but one thing we need to think about before using a 64-bit
unique identifier here is how will we retrieve its last used value
after restart of server. We may need to store it in a persistent way
somewhere.
>>
>> We should use the same for the origin name as well.
>
>
> I did not really change anything related to origin names. Origin names are still the same and include relation id. What do you think would be an issue with origin names in this patch?
>

The problems will be similar to the slot name. The origin is used to
track the progress of replication, so, if we use the wrong origin name
after the restart, it can send the wrong start_streaming position to
the publisher.

-- 
With Regards,
Amit Kapila.

  11. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-07-28T16:02:43Z

    >
> Why after step 4, do you need to drop the replication slot? Won't just
> clearing the required info from the catalog be sufficient?
>

The replication slots that we read from the catalog will not be used for
anything else after we're done with syncing the table which the rep slot
belongs to.
It's removed from the catalog when the sync is completed and it basically
becomes a slot that is not linked to any table or worker. That's why I
think it should be dropped rather than left behind.

Note that if a worker dies and its replication slot continues to exist,
that slot will only be used to complete the sync process of the one table
that the dead worker was syncing but couldn't finish.
When that particular table is synced and becomes ready, the replication
slot has no use anymore.


> Hmm, I think even if there is an iota of a chance which I think is
> there, we can't use worker_pid. Assume, that if the same worker_pid is
> assigned to another worker once the worker using it got an error out,
> the new worker will fail as soon as it will try to create a
> replication slot.
>

Right. If something like that happens, worker will fail without doing
anything. Then a new one will be launched and that one will continue to
do the work.
The worst case might be having conflicting pid over and over again while
also having replication slots whose name includes one of those pids still
exist.
It seems unlikely but possible, yes.


> I feel it would be better or maybe we need to think of some other
> identifier but one thing we need to think about before using a 64-bit
> unique identifier here is how will we retrieve its last used value
> after restart of server. We may need to store it in a persistent way
> somewhere.
>

We might consider storing this info in a catalog again. Since this last
used value will be different for each subscription, pg_subscription can be
a good place to keep that.


> The problems will be similar to the slot name. The origin is used to
> track the progress of replication, so, if we use the wrong origin name
> after the restart, it can send the wrong start_streaming position to
> the publisher.
>

I understand. But origin naming logic is still the same. Its format is like
pg_<subid>_<relid> .
I did not need to change this since it seems to me origins should belong to
only one table. The patch does not reuse origins.
So I don't think this change introduces an issue with origin. What do you
think?

Thanks,
Melih

  12. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-29T10:17:44Z

    On Thu, Jul 28, 2022 at 9:32 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>>
>> Why after step 4, do you need to drop the replication slot? Won't just
>> clearing the required info from the catalog be sufficient?
>
>
> The replication slots that we read from the catalog will not be used for anything else after we're done with syncing the table which the rep slot belongs to.
> It's removed from the catalog when the sync is completed and it basically becomes a slot that is not linked to any table or worker. That's why I think it should be dropped rather than left behind.
>
> Note that if a worker dies and its replication slot continues to exist, that slot will only be used to complete the sync process of the one table that the dead worker was syncing but couldn't finish.
> When that particular table is synced and becomes ready, the replication slot has no use anymore.
>

Why can't it be used to sync the other tables if any?

>>
>> Hmm, I think even if there is an iota of a chance which I think is
>> there, we can't use worker_pid. Assume, that if the same worker_pid is
>> assigned to another worker once the worker using it got an error out,
>> the new worker will fail as soon as it will try to create a
>> replication slot.
>
>
> Right. If something like that happens, worker will fail without doing anything. Then a new one will be launched and that one will continue to do the work.
> The worst case might be having conflicting pid over and over again while also having replication slots whose name includes one of those pids still exist.
> It seems unlikely but possible, yes.
>
>>
>> I feel it would be better or maybe we need to think of some other
>> identifier but one thing we need to think about before using a 64-bit
>> unique identifier here is how will we retrieve its last used value
>> after restart of server. We may need to store it in a persistent way
>> somewhere.
>
>
> We might consider storing this info in a catalog again. Since this last used value will be different for each subscription, pg_subscription can be a good place to keep that.
>

This sounds reasonable. Let's do this unless we get some better idea.

>>
>> The problems will be similar to the slot name. The origin is used to
>> track the progress of replication, so, if we use the wrong origin name
>> after the restart, it can send the wrong start_streaming position to
>> the publisher.
>
>
> I understand. But origin naming logic is still the same. Its format is like pg_<subid>_<relid> .
> I did not need to change this since it seems to me origins should belong to only one table. The patch does not reuse origins.
> So I don't think this change introduces an issue with origin. What do you think?
>

There is no such restriction that origins should belong to only one
table. What makes you think like that?

-- 
With Regards,
Amit Kapila.

  13. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-08-05T13:55:09Z

    Hi Amit,

>> Why after step 4, do you need to drop the replication slot? Won't just
> >> clearing the required info from the catalog be sufficient?
> >
> >
> > The replication slots that we read from the catalog will not be used for
> anything else after we're done with syncing the table which the rep slot
> belongs to.
> > It's removed from the catalog when the sync is completed and it
> basically becomes a slot that is not linked to any table or worker. That's
> why I think it should be dropped rather than left behind.
> >
> > Note that if a worker dies and its replication slot continues to exist,
> that slot will only be used to complete the sync process of the one table
> that the dead worker was syncing but couldn't finish.
> > When that particular table is synced and becomes ready, the replication
> slot has no use anymore.
> >
>
> Why can't it be used to sync the other tables if any?
>

It can be used. But I thought it would be better not to, for example in the
following case:
Let's say a sync worker starts with a table in INIT state. The worker
creates a new replication slot to sync that table.
When sync of the table is completed, it will move to the next one. This
time the new table may be in FINISHEDCOPY state, so the worker may need to
use the new table's existing replication slot.
Before the worker will move to the next table again, there will be two
replication slots used by the worker. We might want to keep one and drop
the other.
At this point, I thought it would be better to keep the replication slot
created by this worker in the first place. I think it's easier to track
slots this way since we know how to generate the rep slots name.
Otherwise we would need to store the replication slot name somewhere too.



> This sounds reasonable. Let's do this unless we get some better idea.
>

I updated the patch to use an unique id for replication slot names and
store the last used id in the catalog.
Can you look into it again please?


There is no such restriction that origins should belong to only one
> table. What makes you think like that?
>

I did not reuse origins since I didn't think it would significantly improve
the performance as reusing replication slots does.
So I just kept the origins as they were, even if it was possible to reuse
them. Does that make sense?

Best,
Melih

  14. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-06T13:01:08Z

    On Fri, Aug 5, 2022 at 7:25 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>>
>> Why can't it be used to sync the other tables if any?
>
>
> It can be used. But I thought it would be better not to, for example in the following case:
> Let's say a sync worker starts with a table in INIT state. The worker creates a new replication slot to sync that table.
> When sync of the table is completed, it will move to the next one. This time the new table may be in FINISHEDCOPY state, so the worker may need to use the new table's existing replication slot.
> Before the worker will move to the next table again, there will be two replication slots used by the worker. We might want to keep one and drop the other.
> At this point, I thought it would be better to keep the replication slot created by this worker in the first place. I think it's easier to track slots this way since we know how to generate the rep slots name.
> Otherwise we would need to store the replication slot name somewhere too.
>

I think there is some basic flaw in slot reuse design. Currently, we
copy the table by starting a repeatable read transaction (BEGIN READ
ONLY ISOLATION LEVEL REPEATABLE READ) and create a slot that
establishes a snapshot which is first used for copy and then LSN
returned by it is used in the catchup phase after the copy is done.
The patch won't establish such a snapshot before a table copy as it
won't create a slot each time. If this understanding is correct, I
think we need to use ExportSnapshot/ImportSnapshot functionality to
achieve it or do something else to avoid the problem mentioned.

>
>>
>> This sounds reasonable. Let's do this unless we get some better idea.
>
>
>> There is no such restriction that origins should belong to only one
>> table. What makes you think like that?
>
>
> I did not reuse origins since I didn't think it would significantly improve the performance as reusing replication slots does.
> So I just kept the origins as they were, even if it was possible to reuse them. Does that make sense?
>

For small tables, it could have a visible performance difference as it
involves database write operations to each time create and drop the
origin. But if we don't want to reuse then also you need to set its
origin_lsn appropriately. Currently (without this patch), after
creating the slot, we directly use the origin_lsn returned by
create_slot API whereas now it won't be the same case as the patch
doesn't create a slot every time.

-- 
With Regards,
Amit Kapila.

  15. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-08-15T11:26:26Z

    Hi Amit,

Amit Kapila <amit.kapila16@gmail.com>, 6 Ağu 2022 Cmt, 16:01 tarihinde şunu
yazdı:

> I think there is some basic flaw in slot reuse design. Currently, we
> copy the table by starting a repeatable read transaction (BEGIN READ
> ONLY ISOLATION LEVEL REPEATABLE READ) and create a slot that
> establishes a snapshot which is first used for copy and then LSN
> returned by it is used in the catchup phase after the copy is done.
> The patch won't establish such a snapshot before a table copy as it
> won't create a slot each time. If this understanding is correct, I
> think we need to use ExportSnapshot/ImportSnapshot functionality to
> achieve it or do something else to avoid the problem mentioned.
>

I did not really think about the snapshot created by replication slot while
making this change. Thanks for pointing it out.
I've been thinking about how to fix this issue. There are some points I'm
still not sure about.
If the worker will not create a new replication slot, which snapshot should
we actually export and then import?
At the line where the worker was supposed to create replication slot but
now will reuse an existing slot instead, calling pg_export_snapshot() can
export the snapshot instead of CREATE_REPLICATION_SLOT.
However, importing that snapshot into the current transaction may not make
any difference since we exported that snapshot from the same transaction. I
think this wouldn't make sense.
How else an export/import snapshot logic can be placed in this change?

LSN also should be set accurately. The current change does not handle LSN
properly.
I see that CREATE_REPLICATION_SLOT returns consistent_point which indicates
the earliest location which streaming can start from. And this
consistent_point is used as origin_startpos.
If that's the case, would it make sense to use "confirmed_flush_lsn" of the
replication slot in case the slot is being reused?
Since confirmed_flush_lsn can be considered as the safest, earliest
location which streaming can start from, I think it would work.

And at this point, with the correct LSN, I'm wondering whether this
export/import logic is really necessary if the worker does not create a
replication slot. What do you think?


For small tables, it could have a visible performance difference as it
> involves database write operations to each time create and drop the
> origin. But if we don't want to reuse then also you need to set its
> origin_lsn appropriately. Currently (without this patch), after
> creating the slot, we directly use the origin_lsn returned by
> create_slot API whereas now it won't be the same case as the patch
> doesn't create a slot every time.
>

Correct. For this issue, please consider the LSN logic explained above.


Thanks,
Melih

  16. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-07T04:35:59Z

    On Mon, Aug 15, 2022 at 4:56 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Amit,
>
> Amit Kapila <amit.kapila16@gmail.com>, 6 Ağu 2022 Cmt, 16:01 tarihinde şunu yazdı:
>>
>> I think there is some basic flaw in slot reuse design. Currently, we
>> copy the table by starting a repeatable read transaction (BEGIN READ
>> ONLY ISOLATION LEVEL REPEATABLE READ) and create a slot that
>> establishes a snapshot which is first used for copy and then LSN
>> returned by it is used in the catchup phase after the copy is done.
>> The patch won't establish such a snapshot before a table copy as it
>> won't create a slot each time. If this understanding is correct, I
>> think we need to use ExportSnapshot/ImportSnapshot functionality to
>> achieve it or do something else to avoid the problem mentioned.
>
>
> I did not really think about the snapshot created by replication slot while making this change. Thanks for pointing it out.
> I've been thinking about how to fix this issue. There are some points I'm still not sure about.
> If the worker will not create a new replication slot, which snapshot should we actually export and then import?
>

Can we (export/import) use the snapshot we used the first time when a
slot is created for future transactions that copy other tables?
Because if we can do that then I think we can use the same LSN as
returned for the slot when it was created for all other table syncs.

-- 
With Regards,
Amit Kapila.

  17. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Ian Lawrence Barwick <barwick@gmail.com> — 2022-11-04T02:47:15Z

    2022年8月5日(金) 22:55 Melih Mutlu <m.melihmutlu@gmail.com>:
>
> Hi Amit,
>
>> >> Why after step 4, do you need to drop the replication slot? Won't just
>> >> clearing the required info from the catalog be sufficient?
>> >
>> >
>> > The replication slots that we read from the catalog will not be used for anything else after we're done with syncing the table which the rep slot belongs to.
>> > It's removed from the catalog when the sync is completed and it basically becomes a slot that is not linked to any table or worker. That's why I think it should be dropped rather than left behind.
>> >
>> > Note that if a worker dies and its replication slot continues to exist, that slot will only be used to complete the sync process of the one table that the dead worker was syncing but couldn't finish.
>> > When that particular table is synced and becomes ready, the replication slot has no use anymore.
>> >
>>
>> Why can't it be used to sync the other tables if any?
>
>
> It can be used. But I thought it would be better not to, for example in the following case:
> Let's say a sync worker starts with a table in INIT state. The worker creates a new replication slot to sync that table.
> When sync of the table is completed, it will move to the next one. This time the new table may be in FINISHEDCOPY state, so the worker may need to use the new table's existing replication slot.
> Before the worker will move to the next table again, there will be two replication slots used by the worker. We might want to keep one and drop the other.
> At this point, I thought it would be better to keep the replication slot created by this worker in the first place. I think it's easier to track slots this way since we know how to generate the rep slots name.
> Otherwise we would need to store the replication slot name somewhere too.
>
>
>>
>> This sounds reasonable. Let's do this unless we get some better idea.
>
>
> I updated the patch to use an unique id for replication slot names and store the last used id in the catalog.
> Can you look into it again please?
>
>
>> There is no such restriction that origins should belong to only one
>> table. What makes you think like that?
>
>
> I did not reuse origins since I didn't think it would significantly improve the performance as reusing replication slots does.
> So I just kept the origins as they were, even if it was possible to reuse them. Does that make sense?

Hi

cfbot reports the patch no longer applies [1].  As CommitFest 2022-11 is
currently underway, this would be an excellent time to update the patch.

[1] http://cfbot.cputube.org/patch_40_3784.log

Thanks

Ian Barwick

  18. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-12-05T13:00:12Z

    Hi hackers,

I've been working on/struggling with this patch for a while. But I haven't
updated this thread regularly.
So sharing what I did with this patch so far.

> Amit Kapila <amit.kapila16@gmail.com>, 6 Ağu 2022 Cmt, 16:01 tarihinde
> şunu yazdı:
> >>
> >> I think there is some basic flaw in slot reuse design. Currently, we
> >> copy the table by starting a repeatable read transaction (BEGIN READ
> >> ONLY ISOLATION LEVEL REPEATABLE READ) and create a slot that
> >> establishes a snapshot which is first used for copy and then LSN
> >> returned by it is used in the catchup phase after the copy is done.
> >> The patch won't establish such a snapshot before a table copy as it
> >> won't create a slot each time. If this understanding is correct, I
> >> think we need to use ExportSnapshot/ImportSnapshot functionality to
> >> achieve it or do something else to avoid the problem mentioned.
>

This issue that Amit mentioned causes some problems such as duplicated rows
in the subscriber.
Basically, with this patch, tablesync worker creates a replication slot
only in its first run. To ensure table copy and sync are consistent with
each other, the worker needs the correct snapshot and LSN which both are
returned by slot create operation.
Since this patch does not create a rep. slot in each table copy and instead
reuses the one created in the beginning, we do not get a new snapshot and
LSN for each table anymore. Snapshot gets lost right after the transaction
is committed, but  the patch continues to use the same LSN for next tables
without the proper snapshot.
In the end, for example, the worker might first copy some rows, then apply
changes from rep. slot and inserts those rows again for the tables in
later iterations.

I discussed some possible ways to resolve this with Amit offline:
1- Copy all tables in one transaction so that we wouldn't need to deal with
snapshots.
Not easy to keep track of the progress. If the transaction fails, we would
need to start all over again.

2- Don't lose the first snapshot (by keeping a transaction open with the
snapshot imported or some other way) and use the same snapshot and LSN for
all tables.
I'm not sure about the side effects of keeping a transaction open that long
or using a snapshot that might be too old after some time.
Still seems like it might work.

3- For each table, get a new snapshot and LSN by using an existing
replication slot.
Even though this approach wouldn't create a new replication slot, preparing
the slot for snapshot and then taking the snapshot may be costly.
Need some numbers here to see how much this approach would improve the
performance.

I decided to go with approach 3 (creating a new snapshot with an existing
replication slot) for now since it would require less change in the
tablesync worker logic than the other options would.
To achieve this, this patch introduces a new command for Streaming
Replication Protocol.
The new REPLICATION_SLOT_SNAPSHOT command basically mimics how
CREATE_REPLICATION_SLOT creates a snapshot, but without actually creating a
new replication slot.
Later the tablesync worker calls this command if it decides not to create a
new rep. slot, uses the snapshot created and LSN returned by the command.

Also;
After the changes discussed here [1], concurrent replication origin drops
by apply worker and tablesync workers may hold each other on wait due to
locks taken by replorigin_drop_by_name.
I see that this harms the performance of logical replication quite a bit in
terms of speed.
Even though reusing replication origins was discussed in this thread
before, the patch didn't include any change to do so.
The updated version of the patch now also reuses replication origins too.
Seems like even only changes to reuse origins by itself improves the
performance significantly.

Attached two patches:
0001: adds REPLICATION_SLOT_SNAPSHOT command for replication protocol.
0002: Reuses workers/replication slots and origins for tablesync

I would appreciate any feedback/review/thought on the approach and both
patches.
I will also share some numbers to compare performances of the patch and
master branch soon.

[1]
https://www.postgresql.org/message-id/flat/20220714115155.GA5439%40depesz.com

Best,
--
Melih Mutlu
Microsoft

  19. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-12-05T13:25:20Z

    On Mon, Dec 5, 2022 at 6:30 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Attached two patches:
> 0001: adds REPLICATION_SLOT_SNAPSHOT command for replication protocol.
> 0002: Reuses workers/replication slots and origins for tablesync
>
> I would appreciate any feedback/review/thought on the approach and both patches.
> I will also share some numbers to compare performances of the patch and master branch soon.
>

It would be interesting to see the numbers differently for resue of
replication slots and origins. This will let us know how much each of
those optimizations helps with the reuse of workers.

-- 
With Regards,
Amit Kapila.

  20. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-12-15T12:03:16Z

    Hi,

Attached new versions of the patch with some changes/fixes.

Here also some numbers to compare the performance of log. rep. with this
patch against the current master branch.

My method of benchmarking is the same with what I did earlier in this
thread. (on a different environment, so not compare the result from this
email with the ones from earlier emails)

> With those changes, I did some benchmarking to see if it improves anything.
> This results compares this patch with the latest version of master branch.
> "max_sync_workers_per_subscription" is set to 2 as default.
> Got some results simply averaging timings from 5 consecutive runs for each
> branch.


Since this patch is expected to improve log. rep. of empty/close-to-empty
tables, started with measuring performance with empty tables.

            |  10 tables      |  100 tables        |  1000 tables
------------------------------------------------------------------------------
master |  283.430 ms  |  22739.107 ms  |  105226.177 ms
------------------------------------------------------------------------------
 patch  |  189.139 ms  |  1554.802 ms    |  23091.434 ms

After the changes discussed here [1], concurrent replication origin drops
> by apply worker and tablesync workers may hold each other on wait due to
> locks taken by replorigin_drop_by_name.
> I see that this harms the performance of logical replication quite a bit
> in terms of speed.
> [1]
> https://www.postgresql.org/message-id/flat/20220714115155.GA5439%40depesz.com


Firstly, as I mentioned, replication origin drops made things worse for the
master branch.
Locks start being a more serious issue when the number of tables increases.
The patch reuses the origin so does not need to drop them in each
iteration. That's why the difference between the master and the patch is
more significant now than it was when I first sent the patch.

To just show that the improvement is not only the result of reuse of
origins, but also reuse of rep. slots and workers, I just reverted those
commits which causes the origin drop issue.

              |  10 tables      |  100 tables        |  1000 tables
-----------------------------------------------------------------------------
reverted |  270.012 ms  |  2483.907 ms   |  31660.758 ms
-----------------------------------------------------------------------------
 patch    |  189.139 ms  |  1554.802 ms   |  23091.434 ms

With this patch, logical replication is still faster, even if we wouldn't
have an issue with rep. origin drops.

Also here are some numbers with 10 tables loaded with some data :

             |     10 MB          |     100 MB
----------------------------------------------------------
master  |  2868.524 ms   |  14281.711 ms
----------------------------------------------------------
 patch   |  1750.226 ms   |  14592.800 ms

The difference between the master and the patch is getting close when the
size of tables increase, as expected.


I would appreciate any feedback/thought on the approach/patch/numbers etc.

Thanks,
-- 
Melih Mutlu
Microsoft

  21. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-12-16T02:46:05Z

    On Thu, Dec 15, 2022 at 5:33 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Also here are some numbers with 10 tables loaded with some data :
>
>              |     10 MB          |     100 MB
> ----------------------------------------------------------
> master  |  2868.524 ms   |  14281.711 ms
> ----------------------------------------------------------
>  patch   |  1750.226 ms   |  14592.800 ms
>
> The difference between the master and the patch is getting close when the size of tables increase, as expected.
>

Right, but when the size is 100MB, it seems to be taking a bit more
time. Do we want to evaluate with different sizes to see how it looks?
Other than that all the numbers are good.

-- 
With Regards,
Amit Kapila.

  22. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2022-12-20T14:44:36Z

    Hi Amit,

Amit Kapila <amit.kapila16@gmail.com>, 16 Ara 2022 Cum, 05:46 tarihinde
şunu yazdı:

> Right, but when the size is 100MB, it seems to be taking a bit more
> time. Do we want to evaluate with different sizes to see how it looks?
> Other than that all the numbers are good.
>

I did a similar testing with again 100MB and also 1GB this time.

             |     100 MB           |     1 GB
----------------------------------------------------------
master  |  14761.425 ms   |  160932.982 ms
----------------------------------------------------------
 patch   |  14398.408 ms   |  160593.078 ms

This time, it seems like the patch seems slightly faster than the master.
Not sure if we can say the patch slows things down (or speeds up) if the
size of tables increases.
The difference may be something arbitrary or caused by other factors. What
do you think?

I also wondered what happens when "max_sync_workers_per_subscription" is
set to 1.
Which means tablesync will be done sequentially in both cases but the patch
will use only one worker and one replication slot during the whole
tablesync process.
Here are the numbers for that case:

             |     100 MB          |     1 GB
----------------------------------------------------------
master  |  27751.463 ms  |  312424.999 ms
----------------------------------------------------------
 patch   |  27342.760 ms  |  310021.767 ms

Best,
-- 
Melih Mutlu
Microsoft

  23. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2022-12-21T12:05:34Z

    On Tue, Dec 20, 2022 at 8:14 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com>, 16 Ara 2022 Cum, 05:46 tarihinde şunu yazdı:
>>
>> Right, but when the size is 100MB, it seems to be taking a bit more
>> time. Do we want to evaluate with different sizes to see how it looks?
>> Other than that all the numbers are good.
>
>
> I did a similar testing with again 100MB and also 1GB this time.
>
>              |     100 MB           |     1 GB
> ----------------------------------------------------------
> master  |  14761.425 ms   |  160932.982 ms
> ----------------------------------------------------------
>  patch   |  14398.408 ms   |  160593.078 ms
>
> This time, it seems like the patch seems slightly faster than the master.
> Not sure if we can say the patch slows things down (or speeds up) if the size of tables increases.
> The difference may be something arbitrary or caused by other factors. What do you think?
>

Yes, I agree with you as I also can't see an obvious reason for any
slowdown with this patch's idea.

-- 
With Regards,
Amit Kapila.

  24. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-01-03T14:53:05Z

    Hi hackers,

Sending an updated version of this patch to get rid of compiler warnings.

I would highly appreciate any feedback.

Thanks,
-- 
Melih Mutlu
Microsoft

  25. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-01-11T08:31:12Z

    Hi hackers,

Rebased the patch to resolve conflicts.

Best,
-- 
Melih Mutlu
Microsoft

  26. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2023-01-17T07:46:06Z

    On Wed, Jan 11, 2023 4:31 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> 
> Hi hackers,
> 
> Rebased the patch to resolve conflicts.
> 

Thanks for your patch. Here are some comments.

0001 patch
============
1. walsender.c
+	/* Create a tuple to send consisten WAL location */

"consisten" should be "consistent" I think.

2. logical.c
+	if (need_full_snapshot)
+	{
+		LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+
+		SpinLockAcquire(&slot->mutex);
+		slot->effective_catalog_xmin = xmin_horizon;
+		slot->data.catalog_xmin = xmin_horizon;
+		slot->effective_xmin = xmin_horizon;
+		SpinLockRelease(&slot->mutex);
+
+		xmin_horizon = GetOldestSafeDecodingTransactionId(!need_full_snapshot);
+		ReplicationSlotsComputeRequiredXmin(true);
+
+		LWLockRelease(ProcArrayLock);
+	}

It seems that we should first get the safe decoding xid, then inform the slot
machinery about the new limit, right? Otherwise the limit will be
InvalidTransactionId and that seems inconsistent with the comment.

3. doc/src/sgml/protocol.sgml
+       is used in the currenct transaction. This command is currently only supported
+       for logical replication.
+       slots.

We don't need to put "slots" in a new line.


0002 patch
============
1.
In pg_subscription_rel.h, I think the type of "srrelslotname" can be changed to
NameData, see "subslotname" in pg_subscription.h.

2.
+				 * Find the logical replication sync worker if exists store
+				 * the slot number for dropping associated replication slots
+				 * later.

Should we add comma after "if exists"?

3.
+	PG_FINALLY();
+	{
+		pfree(cmd.data);
+	}
+	PG_END_TRY();
+	\
+		return tablelist;
+}

Do we need the backslash?

4.
+	/*
+	 * Advance to the LSN got from walrcv_create_slot. This is WAL
+	 * logged for the purpose of recovery. Locks are to prevent the
+	 * replication origin from vanishing while advancing.

"walrcv_create_slot" should be changed to
"walrcv_create_slot/walrcv_slot_snapshot" I think.

5.
+			/* Replication drop might still exist. Try to drop */
+			replorigin_drop_by_name(originname, true, false);

Should "Replication drop" be "Replication origin"?

6.
I saw an assertion failure in the following case, could you please look into it?
The backtrace is attached.

-- pub
CREATE TABLE tbl1 (a int, b text);
CREATE TABLE tbl2 (a int primary key, b text);
create publication pub for table tbl1, tbl2;
insert into tbl1 values (1, 'a');
insert into tbl1 values (1, 'a');

-- sub
CREATE TABLE tbl1 (a int primary key, b text);
CREATE TABLE tbl2 (a int primary key, b text);
create subscription sub connection 'dbname=postgres port=5432' publication pub;

Subscriber log:
2023-01-17 14:47:10.054 CST [1980841] LOG:  logical replication apply worker for subscription "sub" has started
2023-01-17 14:47:10.060 CST [1980843] LOG:  logical replication table synchronization worker for subscription "sub", table "tbl1" has started
2023-01-17 14:47:10.070 CST [1980845] LOG:  logical replication table synchronization worker for subscription "sub", table "tbl2" has started
2023-01-17 14:47:10.073 CST [1980843] ERROR:  duplicate key value violates unique constraint "tbl1_pkey"
2023-01-17 14:47:10.073 CST [1980843] DETAIL:  Key (a)=(1) already exists.
2023-01-17 14:47:10.073 CST [1980843] CONTEXT:  COPY tbl1, line 2
2023-01-17 14:47:10.074 CST [1980821] LOG:  background worker "logical replication worker" (PID 1980843) exited with exit code 1
2023-01-17 14:47:10.083 CST [1980845] LOG:  logical replication table synchronization worker for subscription "sub", table "tbl2" has finished
2023-01-17 14:47:10.083 CST [1980845] LOG:  logical replication table synchronization worker for subscription "sub" has moved to sync table "tbl1".
TRAP: failed Assert("node != InvalidRepOriginId"), File: "origin.c", Line: 892, PID: 1980845

Regards,
Shi yu

  27. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2023-01-17T11:15:38Z

    On Wed, Jan 11, 2023 4:31 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> Rebased the patch to resolve conflicts.

Thanks for your patch set.

Here are some comments:

v3-0001* patch
===============

1. typedefs.list
I think we also need to add "walrcv_slot_snapshot_fn" to this file.

v7-0002* patch
===============
1. About function ReplicationOriginNameForLogicalRep()
Do we need to modify the API of this function? I think the original API could
also meet the current needs. Since this is not a static function, I think it
seems better to keep the original API if there is no reason. Please let me know
if I'm missing something.

-----

2. Comment atop the function GetSubscriptionRelReplicationSlot
+/*
+ * Get replication slot name of subscription table.
+ *
+ * Returns null if the subscription table does not have a replication slot.
+ */

Since this function always returns NULL, I think it would be better to say the
value in "slotname" here instead of the function's return value.

If you agree with this, please also kindly modify the comment atop the function
GetSubscriptionRelOrigin.

-----

3. typo
+			 * At this point, there shouldn't be any existing replication
+			 * origin wit the same name.

wit -> with

-----

4. In function CreateSubscription
+	values[Anum_pg_subscription_sublastusedid - 1] = Int64GetDatum(1);

I think it might be better to initialize this field to NULL or 0 here.
Because in the patch, we always ignore the initialized value when launching
the sync worker in the function process_syncing_tables_for_apply. And I think
we could document in pg-doc that this value means that no tables have been
synced yet.

-----

5. New member "created_slot" in structure LogicalRepWorker
+	/*
+	 * Indicates if the sync worker created a replication slot or it reuses an
+	 * existing one created by another worker.
+	 */
+	bool		created_slot;

I think the second half of the sentence looks inaccurate.
Because I think this flag could be false even when we reuse an existing slot
created by another worker. Assuming the first run for the worker tries to sync
a table which is synced by another sync worker before, and the relstate is set
to SUBREL_STATE_FINISHEDCOPY by another sync worker, I think this flag will not
be set to true. (see function LogicalRepSyncTableStart)

So, what if we simplify the description here and just say that this worker
already has it's default slot?

If I'm not missing something and you agree with this, please also kindly modify
the relevant comment atop the if-statement (!MyLogicalRepWorker->created_slot)
in the function LogicalRepSyncTableStart.

Regards,
Wang Wei

  28. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-01-23T13:00:01Z

    Hi,

Thanks for your review.
Attached updated versions of the patches.

wangw.fnst@fujitsu.com <wangw.fnst@fujitsu.com>, 17 Oca 2023 Sal, 14:15
tarihinde şunu yazdı:

> On Wed, Jan 11, 2023 4:31 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> v3-0001* patch
> ===============
>
> 1. typedefs.list
> I think we also need to add "walrcv_slot_snapshot_fn" to this file.
>

Done.


> v7-0002* patch
> ===============
> 1. About function ReplicationOriginNameForLogicalRep()
> Do we need to modify the API of this function? I think the original API
> could
> also meet the current needs. Since this is not a static function, I think
> it
> seems better to keep the original API if there is no reason. Please let me
> know
> if I'm missing something.
>

You're right.
I still need to modify ReplicationOriginNameForLogicalRep. Origin names are
not tied to relations anymore, so their name doesn't need to
include relation id.
But I didn't really need to change the function signature. I reverted that
part of the change in the updated version of the patch.


> 2. Comment atop the function GetSubscriptionRelReplicationSlot
>

Done


> 3. typo
> +                        * At this point, there shouldn't be any existing
> replication
> +                        * origin wit the same name.
>

Done.


> 4. In function CreateSubscription
> +       values[Anum_pg_subscription_sublastusedid - 1] = Int64GetDatum(1);
>
> I think it might be better to initialize this field to NULL or 0 here.
> Because in the patch, we always ignore the initialized value when launching
> the sync worker in the function process_syncing_tables_for_apply. And I
> think
> we could document in pg-doc that this value means that no tables have been
> synced yet.
>

I changed it to start from 0 and added a line into the related doc to
indicate that 0 means that no table has been synced yet.


> 5. New member "created_slot" in structure LogicalRepWorker
> +       /*
> +        * Indicates if the sync worker created a replication slot or it
> reuses an
> +        * existing one created by another worker.
> +        */
> +       bool            created_slot;
>
> I think the second half of the sentence looks inaccurate.
> Because I think this flag could be false even when we reuse an existing
> slot
> created by another worker. Assuming the first run for the worker tries to
> sync
> a table which is synced by another sync worker before, and the relstate is
> set
> to SUBREL_STATE_FINISHEDCOPY by another sync worker, I think this flag
> will not
> be set to true. (see function LogicalRepSyncTableStart)
>
> So, what if we simplify the description here and just say that this worker
> already has it's default slot?
>
> If I'm not missing something and you agree with this, please also kindly
> modify
> the relevant comment atop the if-statement
> (!MyLogicalRepWorker->created_slot)
> in the function LogicalRepSyncTableStart.
>

This "created_slot" indicates whether the current worker has created a
replication slot for its own use. If so, created_slot will be true,
otherwise false.
Let's say the tablesync worker has not created its own slot yet in its
previous runs or this is its first run. And the worker decides to reuse an
existing replication slot (which created by another tablesync worker). Then
created_slot is expected to be false. Because this particular
tablesync worker has not created its own slot yet in either of its runs.

In your example, the worker is in its first run and begin to sync a table
whose state is FINISHEDCOPY. If the table's state is FINISHEDCOPY then the
table should already have a replication slot created for its own sync
process. The worker will want to reuse that existing replication slot for
this particular table and it will not create a new replication slot. So
created_slot will be false, because the worker has not actually created any
replication slot yet.

Basically, created_slot is set to true only if "walrcv_create_slot" is
called by the tablesync worker any time during its lifetime. Otherwise,
it's possible that the worker can use existing replication slots for each
table it syncs. (e.g. if all the tables that the worker has synced were in
FINISHEDCOPY  state, then the worker will not need to create a new slot).

Does it make sense now? Maybe I need to improve comments to make it clearer.

Best,
-- 
Melih Mutlu
Microsoft

  29. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-01-25T13:02:11Z

    On Mon, Jan 23, 2023 at 6:30 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> Thanks for your review.
> Attached updated versions of the patches.
>

Hello,
I am still in the process of reviewing the patch, before that I tried
to run below test:

--publisher
create table tab1(id int , name varchar);
create table tab2(id int primary key , name varchar);
create table tab3(id int primary key , name varchar);
Insert into tab1 values(10, 'a');
Insert into tab1 values(20, 'b');
Insert into tab1 values(30, 'c');

Insert into tab2 values(10, 'a');
Insert into tab2 values(20, 'b');
Insert into tab2 values(30, 'c');

Insert into tab3 values(10, 'a');
Insert into tab3 values(20, 'b');
Insert into tab3 values(30, 'c');

create publication mypub for table tab1, tab2, tab3;

--subscriber
create table tab1(id int , name varchar);
create table tab2(id int primary key , name varchar);
create table tab3(id int primary key , name varchar);
create subscription mysub connection 'dbname=postgres host=localhost
user=shveta port=5432' publication mypub;

--I see initial data copied, but new catalog columns srrelslotname
and srreloriginname are not updated:
postgres=# select sublastusedid from pg_subscription;
 sublastusedid
---------------
             2

postgres=# select * from pg_subscription_rel;
 srsubid | srrelid | srsubstate | srsublsn  | srrelslotname | srreloriginname
---------+---------+------------+-----------+---------------+-----------------
   16409 |   16384 | r          | 0/15219E0 |               |
   16409 |   16389 | r          | 0/15219E0 |               |
   16409 |   16396 | r          | 0/15219E0 |               |

When are these supposed to be updated? I thought the slotname created
will be updated here. Am I missing something here?

Also the v8 patch does not apply on HEAD, giving merge conflicts.

thanks
Shveta

  30. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-01-26T14:23:22Z

    Hi Shveta,

Thanks for reviewing.

shveta malik <shveta.malik@gmail.com>, 25 Oca 2023 Çar, 16:02 tarihinde
şunu yazdı:

> On Mon, Jan 23, 2023 at 6:30 PM Melih Mutlu <m.melihmutlu@gmail.com>
> wrote:
> --I see initial data copied, but new catalog columns srrelslotname
> and srreloriginname are not updated:
> postgres=# select sublastusedid from pg_subscription;
>  sublastusedid
> ---------------
>              2
>
> postgres=# select * from pg_subscription_rel;
>  srsubid | srrelid | srsubstate | srsublsn  | srrelslotname |
> srreloriginname
>
> ---------+---------+------------+-----------+---------------+-----------------
>    16409 |   16384 | r          | 0/15219E0 |               |
>    16409 |   16389 | r          | 0/15219E0 |               |
>    16409 |   16396 | r          | 0/15219E0 |               |
>
> When are these supposed to be updated? I thought the slotname created
> will be updated here. Am I missing something here?
>

If a relation is currently being synced by a tablesync worker and uses a
replication slot/origin for that operation, then srrelslotname and
srreloriginname fields will have values.
When a relation is done with its replication slot/origin, their info gets
removed from related catalog row, so that slot/origin can be reused for
another table or dropped if not needed anymore.
In your case, all relations are in READY state so it's expected that
srrelslotname and srreloriginname are empty. READY relations do not need a
replication slot/origin anymore.

Tables are probably synced so quickly that you're missing the moments when
a tablesync worker copies a relation and stores its rep. slot/origin in the
catalog.
If initial sync is long enough, then you should be able to see the columns
get updated. I follow [1] to make it longer and test if the patch really
updates the catalog.



> Also the v8 patch does not apply on HEAD, giving merge conflicts.
>

Rebased and resolved conflicts. Please check the new version

---
[1]
publisher:
SELECT 'CREATE TABLE table_'||i||'(i int);' FROM generate_series(1, 100)
g(i) \gexec
SELECT 'INSERT INTO table_'||i||' SELECT x FROM generate_series(1, 10000)
x' FROM generate_series(1, 100) g(i) \gexec
CREATE PUBLICATION mypub FOR ALL TABLES;

subscriber:
SELECT 'CREATE TABLE table_'||i||'(i int);' FROM generate_series(1, 100)
g(i) \gexec
CREATE SUBSCRIPTION mysub CONNECTION 'dbname=postgres port=5432 '
PUBLICATION mypub;
select * from pg_subscription_rel where srrelslotname <> ''; \watch 0.5


Thanks,
-- 
Melih Mutlu
Microsoft

  31. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-01-27T10:11:56Z

    On Thu, Jan 26, 2023 at 7:53 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> If a relation is currently being synced by a tablesync worker and uses a replication slot/origin for that operation, then srrelslotname and srreloriginname fields will have values.
> When a relation is done with its replication slot/origin, their info gets removed from related catalog row, so that slot/origin can be reused for another table or dropped if not needed anymore.
> In your case, all relations are in READY state so it's expected that srrelslotname and srreloriginname are empty. READY relations do not need a replication slot/origin anymore.
>
> Tables are probably synced so quickly that you're missing the moments when a tablesync worker copies a relation and stores its rep. slot/origin in the catalog.
> If initial sync is long enough, then you should be able to see the columns get updated. I follow [1] to make it longer and test if the patch really updates the catalog.
>

Thank You for the details. It is clear now.
>

>
> Rebased and resolved conflicts. Please check the new version
>
Please find my suggestions on v9:

1.
--Can we please add a few more points to the Summary to make it more clear.
a) something telling that reusability of workers is for tables under
one subscription and not across multiple subscriptions.
b) Since we are reusing both workers and slots, can we add:
--when do we actually end up spawning a new worker
--when do we actually end up creating a new slot in a worker rather
than using existing one.
--if we reuse existing slots, what happens to the snapshot?


2.
+       The last used ID for tablesync workers. This ID is used to
+       create replication slots. The last used ID needs to be stored
+       to make logical replication can safely proceed after any interruption.
+       If sublastusedid is 0, then no table has been synced yet.

--typo:
 to make logical replication can safely proceed ==> to make logical
replication safely proceed

--Also, does it sound better:
The last used ID for tablesync workers. It acts as an unique
identifier for replication slots
which are created by table-sync workers. The last used ID needs to be
persisted...


3.
is_first_run;
move_to_next_rel;
--Do you think one variable is enough here as we do not get any extra
info by using 2 variables? Can we have 1 which is more generic like
'ready_to_reuse'. Otherwise, please let me know if we must use 2.


4.
/* missin_ok = true, since the origin might be already dropped. */
typo: missing_ok


5. GetReplicationSlotNamesBySubId:
errmsg("not tuple returned."));

Can we have a better error msg:
                ereport(ERROR,
                        errmsg("could not receive list of slots
associated with subscription %d, error: %s", subid, res->err));

6.
static void
clean_sync_worker(void)

--can we please add introductory comment for this function.

7.
    /*
     * Pick the table for the next run if there is not another worker
     * already picked that table.
     */
Pick the table for the next run if it is not already picked up by
another worker.

8.
process_syncing_tables_for_sync()

/* Cleanup before next run or ending the worker. */
--can we please improve this comment:
if there is no more work left for this worker, stop the worker
gracefully, else do clean-up and make it ready for the next
relation/run.

9.
LogicalRepSyncTableStart:
         * Read previous slot name from the catalog, if exists.
         */
        prev_slotname = (char *) palloc0(NAMEDATALEN);
Do we need to free this at the end?


10.
                if (strlen(prev_slotname) == 0)
                {
                        elog(ERROR, "Replication slot could not be
found for relation %u",
                                 MyLogicalRepWorker->relid);
                }
shall we mention subid also in error msg.

I am reviewing further...
thanks
Shveta

  32. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-01-27T14:23:03Z

    On Thu, 26 Jan 2023 at 19:53, Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Shveta,
>
> Thanks for reviewing.
>
> shveta malik <shveta.malik@gmail.com>, 25 Oca 2023 Çar, 16:02 tarihinde şunu yazdı:
>>
>> On Mon, Jan 23, 2023 at 6:30 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>> --I see initial data copied, but new catalog columns srrelslotname
>> and srreloriginname are not updated:
>> postgres=# select sublastusedid from pg_subscription;
>>  sublastusedid
>> ---------------
>>              2
>>
>> postgres=# select * from pg_subscription_rel;
>>  srsubid | srrelid | srsubstate | srsublsn  | srrelslotname | srreloriginname
>> ---------+---------+------------+-----------+---------------+-----------------
>>    16409 |   16384 | r          | 0/15219E0 |               |
>>    16409 |   16389 | r          | 0/15219E0 |               |
>>    16409 |   16396 | r          | 0/15219E0 |               |
>>
>> When are these supposed to be updated? I thought the slotname created
>> will be updated here. Am I missing something here?
>
>
> If a relation is currently being synced by a tablesync worker and uses a replication slot/origin for that operation, then srrelslotname and srreloriginname fields will have values.
> When a relation is done with its replication slot/origin, their info gets removed from related catalog row, so that slot/origin can be reused for another table or dropped if not needed anymore.
> In your case, all relations are in READY state so it's expected that srrelslotname and srreloriginname are empty. READY relations do not need a replication slot/origin anymore.
>
> Tables are probably synced so quickly that you're missing the moments when a tablesync worker copies a relation and stores its rep. slot/origin in the catalog.
> If initial sync is long enough, then you should be able to see the columns get updated. I follow [1] to make it longer and test if the patch really updates the catalog.
>
>
>>
>> Also the v8 patch does not apply on HEAD, giving merge conflicts.
>
>
> Rebased and resolved conflicts. Please check the new version

CFBot shows some compilation errors as in [1], please post an updated
version for the same:
[14:38:38.392] [827/1808] Compiling C object
src/backend/postgres_lib.a.p/replication_logical_tablesync.c.o
[14:38:38.392] ../src/backend/replication/logical/tablesync.c: In
function ‘LogicalRepSyncTableStart’:
[14:38:38.392] ../src/backend/replication/logical/tablesync.c:1629:3:
warning: implicit declaration of function ‘walrcv_slot_snapshot’
[-Wimplicit-function-declaration]
[14:38:38.392] 1629 | walrcv_slot_snapshot(LogRepWorkerWalRcvConn,
slotname, &options, origin_startpos);
[14:38:38.392] | ^~~~~~~~~~~~~~~~~~~~

[14:38:45.125] FAILED: src/backend/postgres
[14:38:45.125] cc @src/backend/postgres.rsp
[14:38:45.125] /usr/bin/ld:
src/backend/postgres_lib.a.p/replication_logical_tablesync.c.o: in
function `LogicalRepSyncTableStart':
[14:38:45.125] /tmp/cirrus-ci-build/build/../src/backend/replication/logical/tablesync.c:1629:
undefined reference to `walrcv_slot_snapshot'
[14:38:45.125] collect2: error: ld returned 1 exit status

[1] - https://cirrus-ci.com/task/4897131543134208?logs=build#L1236

Regards,
Vignesh

  33. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-01-31T09:59:28Z

    On Fri, Jan 27, 2023 at 3:41 PM shveta malik <shveta.malik@gmail.com> wrote:
>

>
> I am reviewing further...
> thanks
> Shveta

Few more comments:

v4-0001:

1)
REPLICATION_SLOT_SNAPSHOT
--Do we need 'CREATE' prefix with it i.e. CREATE_REPLICATION_SNAPSHOT
(or some other brief one with CREATE?).  'REPLICATION_SLOT_SNAPSHOT'
does not look like a command/action and thus is confusing.

2)
is used in the currenct transaction. This command is currently only supported
for logical replication.
slots.
--typo: currenct-->current
--slots can be moved to previous line

3)
/*
 * Signal that we don't need the timeout mechanism. We're just creating
 * the replication slot and don't yet accept feedback messages or send
 * keepalives. As we possibly need to wait for further WAL the walsender
 * would otherwise possibly be killed too soon.
 */
We're just creating the replication slot --> We're just creating the
replication snapshot


4)
I see XactReadOnly check in CreateReplicationSlot, do we need the same
in ReplicationSlotSnapshot() as well?


===============
v9-0002:

5)
  /* We are safe to drop the replication trackin origin after this
--typo: tracking

6)
                slot->data.catalog_xmin = xmin_horizon;
                slot->effective_xmin = xmin_horizon;
                SpinLockRelease(&slot->mutex);
                xmin_horizon =
GetOldestSafeDecodingTransactionId(!need_full_snapshot);
                ReplicationSlotsComputeRequiredXmin(true);

--do we need to set xmin_horizon in slot after
'GetOldestSafeDecodingTransactionId' call, otherwise it will be set to
InvalidId in slot. Is that intentional? I see that we do set this
correct xmin_horizon in builder->initial_xmin_horizon but the slot is
carrying Invalid one.

thanks
Shveta

  34. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2023-01-31T10:27:26Z

    On Mon, Jan 23, 2023 21:00 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> Hi,
>
> Thanks for your review. 
> Attached updated versions of the patches.

Thanks for updating the patch set.

> > 5. New member "created_slot" in structure LogicalRepWorker
> > +       /*
> > +        * Indicates if the sync worker created a replication slot or it reuses an
> > +        * existing one created by another worker.
> > +        */
> > +       bool            created_slot;
> >
> > I think the second half of the sentence looks inaccurate.
> > Because I think this flag could be false even when we reuse an existing slot
> > created by another worker. Assuming the first run for the worker tries to sync
> > a table which is synced by another sync worker before, and the relstate is set
> > to SUBREL_STATE_FINISHEDCOPY by another sync worker, I think this flag will
> not
> > be set to true. (see function LogicalRepSyncTableStart)
> >
> > So, what if we simplify the description here and just say that this worker
> > already has it's default slot?
> >
> > If I'm not missing something and you agree with this, please also kindly modify
> > the relevant comment atop the if-statement (!MyLogicalRepWorker-
> >created_slot)
> > in the function LogicalRepSyncTableStart.
> 
> This "created_slot" indicates whether the current worker has created a
> replication slot for its own use. If so, created_slot will be true, otherwise false.
> Let's say the tablesync worker has not created its own slot yet in its previous
> runs or this is its first run. And the worker decides to reuse an existing
> replication slot (which created by another tablesync worker). Then created_slot
> is expected to be false. Because this particular tablesync worker has not created
> its own slot yet in either of its runs.
>
> In your example, the worker is in its first run and begin to sync a table whose
> state is FINISHEDCOPY. If the table's state is FINISHEDCOPY then the table
> should already have a replication slot created for its own sync process. The
> worker will want to reuse that existing replication slot for this particular table
> and it will not create a new replication slot. So created_slot will be false, because
> the worker has not actually created any replication slot yet.
> 
> Basically, created_slot is set to true only if "walrcv_create_slot" is called by the
> tablesync worker any time during its lifetime. Otherwise, it's possible that the
> worker can use existing replication slots for each table it syncs. (e.g. if all the
> tables that the worker has synced were in FINISHEDCOPY  state, then the
> worker will not need to create a new slot).
> 
> Does it make sense now? Maybe I need to improve comments to make it
> clearer.

Yes, I think it makes sense. Thanks for the detailed explanation.
I think I misunderstood the second half of the comment. I previously thought it
meant that it was also true when reusing an existing slot.

I found one typo in v9-0002, but it seems already mentioned by Shi in [1].#5
before. Maybe you can have a look at that email for this and some other
comments.

Regards,
Wang Wei

  35. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2023-01-31T10:40:03Z

    On Tues, Jan 31, 2023 18:27 PM I wrote:
> I found one typo in v9-0002, but it seems already mentioned by Shi in [1].#5
> before. Maybe you can have a look at that email for this and some other
> comments.

Sorry, I forgot to add the link to the email. Please refer to [1].

[1] - https://www.postgresql.org/message-id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpnprd01.prod.outlook.com

Regards,
Wang Wei

  36. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-01-31T11:00:48Z

    On Tue, Jan 31, 2023 at 3:57 PM wangw.fnst@fujitsu.com
<wangw.fnst@fujitsu.com> wrote:
>
> On Mon, Jan 23, 2023 21:00 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> > Hi,
> >
> > Thanks for your review.
> > Attached updated versions of the patches.
>
> Thanks for updating the patch set.
>
> > > 5. New member "created_slot" in structure LogicalRepWorker
> > > +       /*
> > > +        * Indicates if the sync worker created a replication slot or it reuses an
> > > +        * existing one created by another worker.
> > > +        */
> > > +       bool            created_slot;
> > >

> Yes, I think it makes sense. Thanks for the detailed explanation.
> I think I misunderstood the second half of the comment. I previously thought it
> meant that it was also true when reusing an existing slot.
>

I agree with Wang-san that the comment is confusing, I too
misunderstood it initially during my first run of the code. Maybe it
can be improved.
'Indicates if the sync worker created a replication slot for itself;
set to false if sync worker reuses an existing one created by another
worker'

thanks
Shveta

  37. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-01T11:35:44Z

    Hi,

Please see attached patches for the below changes.

shveta malik <shveta.malik@gmail.com>, 27 Oca 2023 Cum, 13:12 tarihinde
şunu yazdı:

> On Thu, Jan 26, 2023 at 7:53 PM Melih Mutlu <m.melihmutlu@gmail.com>
> wrote:
> 1.
> --Can we please add a few more points to the Summary to make it more clear.
> a) something telling that reusability of workers is for tables under
> one subscription and not across multiple subscriptions.
> b) Since we are reusing both workers and slots, can we add:
> --when do we actually end up spawning a new worker
> --when do we actually end up creating a new slot in a worker rather
> than using existing one.
> --if we reuse existing slots, what happens to the snapshot?
>

I modified the commit message if that's what you mean by the Summary.


> 2.
> +       The last used ID for tablesync workers. This ID is used to
> +       create replication slots. The last used ID needs to be stored
> +       to make logical replication can safely proceed after any
> interruption.
> +       If sublastusedid is 0, then no table has been synced yet.
>
> --typo:
>  to make logical replication can safely proceed ==> to make logical
> replication safely proceed
>

Done


> 3.
> is_first_run;
> move_to_next_rel;
> --Do you think one variable is enough here as we do not get any extra
> info by using 2 variables? Can we have 1 which is more generic like
> 'ready_to_reuse'. Otherwise, please let me know if we must use 2.
>

Right. Removed is_first_run and renamed move_to_next_rel as ready_to_reuse.


> 4.
> /* missin_ok = true, since the origin might be already dropped. */
> typo: missing_ok
>

Done.


> 5. GetReplicationSlotNamesBySubId:
> errmsg("not tuple returned."));
>
> Can we have a better error msg:
>                 ereport(ERROR,
>                         errmsg("could not receive list of slots
> associated with subscription %d, error: %s", subid, res->err));
>

Done.


> 6.
> static void
> clean_sync_worker(void)
>
> --can we please add introductory comment for this function.
>

Done.


> 7.
>     /*
>      * Pick the table for the next run if there is not another worker
>      * already picked that table.
>      */
> Pick the table for the next run if it is not already picked up by
> another worker.
>

Done.


> 8.
> process_syncing_tables_for_sync()
>
> /* Cleanup before next run or ending the worker. */
> --can we please improve this comment:
> if there is no more work left for this worker, stop the worker
> gracefully, else do clean-up and make it ready for the next
> relation/run.
>

Done


> 9.
> LogicalRepSyncTableStart:
>          * Read previous slot name from the catalog, if exists.
>          */
>         prev_slotname = (char *) palloc0(NAMEDATALEN);
> Do we need to free this at the end?
>

Pfree'd prev_slotname after we're done with it.


> 10.
>                 if (strlen(prev_slotname) == 0)
>                 {
>                         elog(ERROR, "Replication slot could not be
> found for relation %u",
>                                  MyLogicalRepWorker->relid);
>                 }
> shall we mention subid also in error msg.
>

Done.

Thanks for reviewing,
-- 
Melih Mutlu
Microsoft

  38. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-01T11:44:22Z

    Hi,

I mistakenly attached v9 in my previous email.
Please see attached v6 and v10 for the previous and below changes.

shveta malik <shveta.malik@gmail.com>, 31 Oca 2023 Sal, 12:59 tarihinde
şunu yazdı:

> On Fri, Jan 27, 2023 at 3:41 PM shveta malik <shveta.malik@gmail.com>
> wrote:
> 1)
> REPLICATION_SLOT_SNAPSHOT
> --Do we need 'CREATE' prefix with it i.e. CREATE_REPLICATION_SNAPSHOT
> (or some other brief one with CREATE?).  'REPLICATION_SLOT_SNAPSHOT'
> does not look like a command/action and thus is confusing.
>

Renamed it as CREATE_REPLICATION_SNAPSHOT


> 2)
> is used in the currenct transaction. This command is currently only
> supported
> for logical replication.
> slots.
> --typo: currenct-->current
> --slots can be moved to previous line
>

Done.


> 3)
> /*
>  * Signal that we don't need the timeout mechanism. We're just creating
>  * the replication slot and don't yet accept feedback messages or send
>  * keepalives. As we possibly need to wait for further WAL the walsender
>  * would otherwise possibly be killed too soon.
>  */
> We're just creating the replication slot --> We're just creating the
> replication snapshot
>

Done.


> 4)
> I see XactReadOnly check in CreateReplicationSlot, do we need the same
> in ReplicationSlotSnapshot() as well?
>

Added this check too.


> ===============
> v9-0002:
>
> 5)
>   /* We are safe to drop the replication trackin origin after this
> --typo: tracking
>

Done.


> 6)
>                 slot->data.catalog_xmin = xmin_horizon;
>                 slot->effective_xmin = xmin_horizon;
>                 SpinLockRelease(&slot->mutex);
>                 xmin_horizon =
> GetOldestSafeDecodingTransactionId(!need_full_snapshot);
>                 ReplicationSlotsComputeRequiredXmin(true);
>
> --do we need to set xmin_horizon in slot after
> 'GetOldestSafeDecodingTransactionId' call, otherwise it will be set to
> InvalidId in slot. Is that intentional? I see that we do set this
> correct xmin_horizon in builder->initial_xmin_horizon but the slot is
> carrying Invalid one.
>

I think you're right. Moved GetOldestSafeDecodingTransactionId call before
xmin_horizon assignment.

Thanks,
-- 
Melih Mutlu
Microsoft

  39. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-01T12:00:50Z

    On Wed, Feb 1, 2023 at 5:05 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> Please see attached patches for the below changes.
>

> Thanks for reviewing,
> --
> Melih Mutlu
> Microsoft

Hello Melih,
Thank you for making the changes.

I have few more comments:
1)
src/backend/replication/logical/worker.c: (errmsg("logical replication
table synchronization worker for subscription \"%s\", table \"%s\" has
started",
src/backend/replication/logical/worker.c: (errmsg("logical replication
table synchronization worker for subscription \"%s\" has moved to sync
table \"%s\".",
src/backend/replication/logical/tablesync.c: (errmsg("logical
replication table synchronization worker for subscription \"%s\",
table \"%s\" has finished",

In above can we have rep_slot_id as well in trace message, else it is
not clear which worker moved to next relation. We may have:
logical replication table synchronization worker_%d for subscription
\"%s\" has moved to sync table, rep_slot_id,....

Overall we need to improve the tracing. I will give my suggestions on
this later (in detail).

2) I found a crash in the previous patch (v9), but have not tested it
on the latest yet. Crash happens when all the replication slots are
consumed and we are trying to create new. I tweaked the settings like
below so that it can be reproduced easily:
max_sync_workers_per_subscription=3
max_replication_slots = 2
and then ran the test case shared by you. I think there is some memory
corruption happening. (I did test in debug mode, have not tried in
release mode). I tried to put some traces to identify the root-cause.
I observed that worker_1 keeps on moving from 1 table to another table
correctly, but at some point, it gets corrupted i.e. origin-name
obtained for it is wrong and it tries to advance that and since that
origin does not exist, it  asserts and then something else crashes.
>From log: (new trace lines added by me are prefixed by shveta, also
tweaked code to have my comment 1 fixed to have clarity on worker-id).

form below traces, it is clear that worker_1 was moving from one
relation to another, always getting correct origin 'pg_16688_1', but
at the end it got 'pg_16688_49' which does not exist. Second part of
trace shows who updated 'pg_16688_49', it was done by worker_49 which
even did not get chance to create this origin due to max_rep_slot
reached.
==============================
..............
2023-02-01 16:01:38.041 IST [9243] LOG:  logical replication table
synchronization worker_1 for subscription "mysub", table "table_93"
has finished
2023-02-01 16:01:38.047 IST [9243] LOG:  logical replication table
synchronization worker_1 for subscription "mysub" has moved to sync
table "table_98".
2023-02-01 16:01:38.055 IST [9243] LOG:  shveta-
LogicalRepSyncTableStart- worker_1 get-origin-id2 originid:2,
originname:pg_16688_1
2023-02-01 16:01:38.055 IST [9243] LOG:  shveta-
LogicalRepSyncTableStart- Worker_1 reusing
slot:pg_16688_sync_1_7195132648087016333, originid:2,
originname:pg_16688_1
2023-02-01 16:01:38.094 IST [9243] LOG:  shveta-
LogicalRepSyncTableStart- worker_1 updated-origin2
originname:pg_16688_1
2023-02-01 16:01:38.096 IST [9243] LOG:  logical replication table
synchronization worker_1 for subscription "mysub", table "table_98"
has finished
2023-02-01 16:01:38.096 IST [9243] LOG:  logical replication table
synchronization worker_1 for subscription "mysub" has moved to sync
table "table_60".
2023-02-01 16:01:38.099 IST [9243] LOG:  shveta-
LogicalRepSyncTableStart- worker_1 get-origin originid:0,
originname:pg_16688_49
2023-02-01 16:01:38.099 IST [9243] LOG:  could not drop replication
slot "pg_16688_sync_49_7195132648087016333" on publisher: ERROR:
replication slot "pg_16688_sync_49_7195132648087016333" does not exist
2023-02-01 16:01:38.103 IST [9243] LOG:  shveta-
LogicalRepSyncTableStart- Worker_1 reusing
slot:pg_16688_sync_1_7195132648087016333, originid:0,
originname:pg_16688_49
TRAP: failed Assert("node != InvalidRepOriginId"), File: "origin.c",
Line: 892, PID: 9243
postgres: logical replication worker for subscription 16688 sync 16384
(ExceptionalCondition+0xbb)[0x56019194d3b7]
postgres: logical replication worker for subscription 16688 sync 16384
(replorigin_advance+0x6d)[0x5601916b53d4]
postgres: logical replication worker for subscription 16688 sync 16384
(LogicalRepSyncTableStart+0xbb4)[0x5601916cb648]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x5d25e2)[0x5601916d35e2]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x5d282c)[0x5601916d382c]
postgres: logical replication worker for subscription 16688 sync 16384
(ApplyWorkerMain+0x17b)[0x5601916d4078]
postgres: logical replication worker for subscription 16688 sync 16384
(StartBackgroundWorker+0x248)[0x56019167f943]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x589ad3)[0x56019168aad3]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x589ee3)[0x56019168aee3]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x588d8d)[0x560191689d8d]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x584604)[0x560191685604]
postgres: logical replication worker for subscription 16688 sync 16384
(PostmasterMain+0x14f1)[0x560191684f1e]
postgres: logical replication worker for subscription 16688 sync 16384
(+0x446e05)[0x560191547e05]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f048cc58d90]
==============================

'pg_16688_49' updated by worker_49:
2023-02-01 16:01:37.083 IST [9348] LOG:  shveta-
LogicalRepSyncTableStart- worker_49 get-origin originid:0,
originname:pg_16688_49
2023-02-01 16:01:37.083 IST [9348] LOG:  shveta-
LogicalRepSyncTableStart- worker_49 updated-origin
originname:pg_16688_49
2023-02-01 16:01:37.083 IST [9348] LOG:  shveta-
LogicalRepSyncTableStart- worker_49 get-origin-id2 originid:0,
originname:pg_16688_49
2023-02-01 16:01:37.083 IST [9348] ERROR:  could not create
replication slot "pg_16688_sync_49_7195132648087016333": ERROR:  all
replication slots are in use
        HINT:  Free one or increase max_replication_slots.
==============================


Rest of the workers keep on exiting and getting recreated since they
could not create slot. The last_used_id kept on increasing on every
restart of subscriber until we kill it. In my case it reached 2k+.

thanks
Shveta

  40. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-01T12:07:25Z

    Hi,

wangw.fnst@fujitsu.com <wangw.fnst@fujitsu.com>, 31 Oca 2023 Sal, 13:40
tarihinde şunu yazdı:

> Sorry, I forgot to add the link to the email. Please refer to [1].
>
> [1] -
> https://www.postgresql.org/message-id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpnprd01.prod.outlook.com


Thanks for pointing out this review. I somehow skipped that, sorry.

Please see attached patches.

shiy.fnst@fujitsu.com <shiy.fnst@fujitsu.com>, 17 Oca 2023 Sal, 10:46
tarihinde şunu yazdı:

> On Wed, Jan 11, 2023 4:31 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> 0001 patch
> ============
> 1. walsender.c
> +       /* Create a tuple to send consisten WAL location */
>
> "consisten" should be "consistent" I think.
>

Done.


> 2. logical.c
> +       if (need_full_snapshot)
> +       {
> +               LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
> +
> +               SpinLockAcquire(&slot->mutex);
> +               slot->effective_catalog_xmin = xmin_horizon;
> +               slot->data.catalog_xmin = xmin_horizon;
> +               slot->effective_xmin = xmin_horizon;
> +               SpinLockRelease(&slot->mutex);
> +
> +               xmin_horizon =
> GetOldestSafeDecodingTransactionId(!need_full_snapshot);
> +               ReplicationSlotsComputeRequiredXmin(true);
> +
> +               LWLockRelease(ProcArrayLock);
> +       }
>
> It seems that we should first get the safe decoding xid, then inform the
> slot
> machinery about the new limit, right? Otherwise the limit will be
> InvalidTransactionId and that seems inconsistent with the comment.
>

You're right. Moved that call before assigning xmin_horizon.


> 3. doc/src/sgml/protocol.sgml
> +       is used in the currenct transaction. This command is currently
> only supported
> +       for logical replication.
> +       slots.
>
> We don't need to put "slots" in a new line.
>

Done.


> 0002 patch
> ============
> 1.
> In pg_subscription_rel.h, I think the type of "srrelslotname" can be
> changed to
> NameData, see "subslotname" in pg_subscription.h.
>
> 2.
> +                                * Find the logical replication sync
> worker if exists store
> +                                * the slot number for dropping associated
> replication slots
> +                                * later.
>
> Should we add comma after "if exists"?
>

Done.

3.
> +       PG_FINALLY();
> +       {
> +               pfree(cmd.data);
> +       }
> +       PG_END_TRY();
> +       \
> +               return tablelist;
> +}
>
> Do we need the backslash?
>

Removed it.


> 4.
> +       /*
> +        * Advance to the LSN got from walrcv_create_slot. This is WAL
> +        * logged for the purpose of recovery. Locks are to prevent the
> +        * replication origin from vanishing while advancing.
>
> "walrcv_create_slot" should be changed to
> "walrcv_create_slot/walrcv_slot_snapshot" I think.


Right, done.


>
>
5.
> +                       /* Replication drop might still exist. Try to drop
> */
> +                       replorigin_drop_by_name(originname, true, false);
>
> Should "Replication drop" be "Replication origin"?
>

Done.


> 6.
> I saw an assertion failure in the following case, could you please look
> into it?
> The backtrace is attached.
>
> -- pub
> CREATE TABLE tbl1 (a int, b text);
> CREATE TABLE tbl2 (a int primary key, b text);
> create publication pub for table tbl1, tbl2;
> insert into tbl1 values (1, 'a');
> insert into tbl1 values (1, 'a');
>
> -- sub
> CREATE TABLE tbl1 (a int primary key, b text);
> CREATE TABLE tbl2 (a int primary key, b text);
> create subscription sub connection 'dbname=postgres port=5432' publication
> pub;
>
> Subscriber log:
> 2023-01-17 14:47:10.054 CST [1980841] LOG:  logical replication apply
> worker for subscription "sub" has started
> 2023-01-17 14:47:10.060 CST [1980843] LOG:  logical replication table
> synchronization worker for subscription "sub", table "tbl1" has started
> 2023-01-17 14:47:10.070 CST [1980845] LOG:  logical replication table
> synchronization worker for subscription "sub", table "tbl2" has started
> 2023-01-17 14:47:10.073 CST [1980843] ERROR:  duplicate key value violates
> unique constraint "tbl1_pkey"
> 2023-01-17 14:47:10.073 CST [1980843] DETAIL:  Key (a)=(1) already exists.
> 2023-01-17 14:47:10.073 CST [1980843] CONTEXT:  COPY tbl1, line 2
> 2023-01-17 14:47:10.074 CST [1980821] LOG:  background worker "logical
> replication worker" (PID 1980843) exited with exit code 1
> 2023-01-17 14:47:10.083 CST [1980845] LOG:  logical replication table
> synchronization worker for subscription "sub", table "tbl2" has finished
> 2023-01-17 14:47:10.083 CST [1980845] LOG:  logical replication table
> synchronization worker for subscription "sub" has moved to sync table
> "tbl1".
> TRAP: failed Assert("node != InvalidRepOriginId"), File: "origin.c", Line:
> 892, PID: 1980845
>

I'm not able to reproduce this yet. Will look into it further.

Thanks,
-- 
Melih Mutlu
Microsoft

  41. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-01T12:12:19Z

    Hi Shveta,

shveta malik <shveta.malik@gmail.com>, 1 Şub 2023 Çar, 15:01 tarihinde şunu
yazdı:

> On Wed, Feb 1, 2023 at 5:05 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> 2) I found a crash in the previous patch (v9), but have not tested it
> on the latest yet. Crash happens when all the replication slots are
> consumed and we are trying to create new. I tweaked the settings like
> below so that it can be reproduced easily:
> max_sync_workers_per_subscription=3
> max_replication_slots = 2
> and then ran the test case shared by you. I think there is some memory
> corruption happening. (I did test in debug mode, have not tried in
> release mode). I tried to put some traces to identify the root-cause.
> I observed that worker_1 keeps on moving from 1 table to another table
> correctly, but at some point, it gets corrupted i.e. origin-name
> obtained for it is wrong and it tries to advance that and since that
> origin does not exist, it  asserts and then something else crashes.
> From log: (new trace lines added by me are prefixed by shveta, also
> tweaked code to have my comment 1 fixed to have clarity on worker-id).
>
> form below traces, it is clear that worker_1 was moving from one
> relation to another, always getting correct origin 'pg_16688_1', but
> at the end it got 'pg_16688_49' which does not exist. Second part of
> trace shows who updated 'pg_16688_49', it was done by worker_49 which
> even did not get chance to create this origin due to max_rep_slot
> reached.
>

Thanks for investigating this error. I think it's the same error as the one
Shi reported earlier. [1]
I couldn't reproduce it yet but will apply your tweaks and try again.
Looking into this.

[1]
https://www.postgresql.org/message-id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpnprd01.prod.outlook.com


Thanks,
-- 
Melih Mutlu
Microsoft

  42. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-01T12:57:56Z

    On Wed, Feb 1, 2023 at 5:42 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:

>
>
> Thanks for investigating this error. I think it's the same error as the one Shi reported earlier. [1]
> I couldn't reproduce it yet but will apply your tweaks and try again.
> Looking into this.
>
> [1] https://www.postgresql.org/message-id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpnprd01.prod.outlook.com

I tried Shi-san's testcase earlier but I too could not reproduce it,
so I assumed that it is fixed in one of your patches already and thus
thought that the current issue is a new one.

thanks
Shveta

  43. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-02T03:48:03Z

    On Wed, Feb 1, 2023 at 5:42 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Shveta,
>
> shveta malik <shveta.malik@gmail.com>, 1 Şub 2023 Çar, 15:01 tarihinde şunu yazdı:
>>
>> On Wed, Feb 1, 2023 at 5:05 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>> 2) I found a crash in the previous patch (v9), but have not tested it
>> on the latest yet. Crash happens when all the replication slots are
>> consumed and we are trying to create new. I tweaked the settings like
>> below so that it can be reproduced easily:
>> max_sync_workers_per_subscription=3
>> max_replication_slots = 2
>> and then ran the test case shared by you. I think there is some memory
>> corruption happening. (I did test in debug mode, have not tried in
>> release mode). I tried to put some traces to identify the root-cause.
>> I observed that worker_1 keeps on moving from 1 table to another table
>> correctly, but at some point, it gets corrupted i.e. origin-name
>> obtained for it is wrong and it tries to advance that and since that
>> origin does not exist, it  asserts and then something else crashes.
>> From log: (new trace lines added by me are prefixed by shveta, also
>> tweaked code to have my comment 1 fixed to have clarity on worker-id).
>>
>> form below traces, it is clear that worker_1 was moving from one
>> relation to another, always getting correct origin 'pg_16688_1', but
>> at the end it got 'pg_16688_49' which does not exist. Second part of
>> trace shows who updated 'pg_16688_49', it was done by worker_49 which
>> even did not get chance to create this origin due to max_rep_slot
>> reached.
>
>
> Thanks for investigating this error. I think it's the same error as the one Shi reported earlier. [1]
> I couldn't reproduce it yet but will apply your tweaks and try again.
> Looking into this.
>
> [1] https://www.postgresql.org/message-id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpnprd01.prod.outlook.com
>

Hi Melih,
I think I am able to identify the root cause. It is not memory
corruption, but the way origin-names are stored in system-catalog
mapped to a particular relation-id before even those are created.

After adding few more logs:

[4227] LOG:  shveta- LogicalRepSyncTableStart- worker_49 constructed
originname :pg_16684_49, relid:16540
[4227] LOG:  shveta- LogicalRepSyncTableStart- worker_49
updated-origin in system catalog:pg_16684_49,
slot:pg_16684_sync_49_7195149685251088378, relid:16540
[4227] LOG:  shveta- LogicalRepSyncTableStart- worker_49
get-origin-id2 originid:0, originname:pg_16684_49
[4227] ERROR:  could not create replication slot
"pg_16684_sync_49_7195149685251088378": ERROR:  all replication slots
are in use
        HINT:  Free one or increase max_replication_slots.


[4428] LOG:  shveta- LogicalRepSyncTableStart- worker_148 constructed
originname :pg_16684_49, relid:16540
[4428] LOG:  could not drop replication slot
"pg_16684_sync_49_7195149685251088378" on publisher: ERROR:
replication slot "pg_16684_sync_49_7195149      685251088378" does not
exist
[4428] LOG:  shveta- LogicalRepSyncTableStart- worker_148 drop-origin
originname:pg_16684_49
[4428] LOG:  shveta- LogicalRepSyncTableStart- worker_148
updated-origin:pg_16684_49,
slot:pg_16684_sync_148_7195149685251088378, relid:16540

So from above, worker_49 came and picked up relid:16540, constructed
origin-name and slot-name and updated in system-catalog and then it
tried to actually create that slot and origin but since max-slot count
was reached, it failed and exited, but did not do cleanup from system
catalog for that relid.

Then worker_148 came and also picked up table 16540 since it was not
completed/started by previous worker, but this time it found that
origin and slot entry present in system-catalog against this relid, so
it picked the same names and started processing, but since those do
not exist, it asserted while advancing the origin.

The assert is only reproduced when an already running worker (say
worker_1) who has 'created=true' set, gets to sync the relid for which
a previously failed worker has tried and updated origin-name w/o
creating it. In such a case worker_1 (with created=true) will try to
reuse the origin and thus will try to advance it and will end up
asserting. That is why you might not see the assertion always. The
condition 'created' is set to true for that worker and it goes to
reuse the origin updated by the previous worker.

So to fix this, I think either we update origin and slot entries in
the system catalog after the creation has passed or we clean-up the
system catalog in case of failure. What do you think?

thanks
Shveta

  44. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-02T09:05:37Z

    On Thu, Feb 2, 2023 at 9:18 AM shveta malik <shveta.malik@gmail.com> wrote:
>
>
> Hi Melih,
> I think I am able to identify the root cause. It is not memory
> corruption, but the way origin-names are stored in system-catalog
> mapped to a particular relation-id before even those are created.
>

Apart from the problem mentioned in my earlier email, I think there is
one more issue here as seen by the same assert causing testcase. The
'lastusedid' stored in system-catalog kept on increasing w/o even slot
and origin getting created. 2 workers worked well with
max_replication_slots=2 and then since all slots were consumed 3rd one
could not create any slot and exited but it increased lastusedid. Then
another worker came, incremented lastusedId in system-catalog and
failed to create slot and exited and so on. This makes lastUsedId
incremented continuously until you kill the subscriber or free any
slot used previously. If you keep subscriber running long enough, it
will make lastUsedId go beyond its limit.
Shouldn't lastUsedId be incremented only after making sure that worker
has created a slot and origin corresponding to that particular
rep_slot_id (derived using lastUsedId). Please let me know if my
understanding is not correct.

thanks
Shveta

  45. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-02T11:31:09Z

    On Wed, Feb 1, 2023 at 5:37 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
> Please see attached patches.
>
> Thanks,
> --
> Melih Mutlu
> Microsoft

Hi Melih,

Few suggestions on v10-0002-Reuse patch

1)
        for (int64 i = 1; i <= lastusedid; i++)
        {
                char            originname_to_drop[NAMEDATALEN] = {0};
                snprintf(originname_to_drop,
sizeof(originname_to_drop), "pg_%u_%lld", subid, (long long) i);
         .......
          }

--Is it better to use the function
'ReplicationOriginNameForLogicalRep' here instead of sprintf, just to
be consistent everywhere to construct origin-name?


2)
pa_launch_parallel_worker:
launched = logicalrep_worker_launch(MyLogicalRepWorker->dbid,
                                                            MySubscription->oid,

MySubscription->name,

MyLogicalRepWorker->userid,
                                                             InvalidOid,

dsm_segment_handle(winfo->dsm_seg),
                                                            0);

--Can we please define 'InvalidRepSlotId' macro and pass it here as
the last arg to make it more readable.
#define InvalidRepSlotId 0
Same in ApplyLauncherMain where we are passing 0 as last arg to
logicalrep_worker_launch.

3)
We are safe to drop the replication trackin origin after this
--typo: trackin -->tracking

4)
process_syncing_tables_for_sync:
if (MyLogicalRepWorker->slot_name && strcmp(syncslotname,
MyLogicalRepWorker->slot_name) != 0)
{
          ..............
ReplicationOriginNameForLogicalRep(MyLogicalRepWorker->subid,

MyLogicalRepWorker->relid,
                                                              originname,

sizeof(originname));

/* Drop replication origin */
replorigin_drop_by_name(originname, true, false);
}

--Are we passing missing_ok as true (second arg) intentionally here in
replorigin_drop_by_name? Once we fix the issue reported  in my earlier
email (ASSERT), do you think it makes sense to  pass missing_ok as
false here?

5)
process_syncing_tables_for_sync:
                foreach(lc, rstates)
                {

                        rstate = (SubscriptionRelState *)
palloc(sizeof(SubscriptionRelState));
                        memcpy(rstate, lfirst(lc),
sizeof(SubscriptionRelState));
                        /*
                         * Pick the table for the next run if it is
not already picked up
                         * by another worker.
                         */
                        LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
                        if (rstate->state != SUBREL_STATE_SYNCDONE &&

!logicalrep_worker_find(MySubscription->oid, rstate->relid, false))

                        {
                           .........
                        }
                        LWLockRelease(LogicalRepWorkerLock);
                }

--Do we need to palloc for each relation separately? Shall we do it
once outside the loop and reuse it? Also pfree is not done for rstate
here.



6)
LogicalRepSyncTableStart:
1385         slotname = (char *) palloc(NAMEDATALEN);
1413         prev_slotname = (char *) palloc(NAMEDATALEN);
1481         slotname = prev_slotname;
1502         pfree(prev_slotname);
1512         UpdateSubscriptionRel(MyLogicalRepWorker->subid,
1513
MyLogicalRepWorker->relid,
1514
MyLogicalRepWorker->relstate,
1515
MyLogicalRepWorker->relstate_lsn,
1516                                                   slotname,
1517                                                   originname);

Can you please review the above flow (I have given line# along with),
I think it could be problematic. We alloced prev_slotname, assigned it
to slotname, freed prev_slotname and used slotname after freeing the
prev_slotname.
Also slotname is allocated some memory too, that is not freed.

Reviewing further....

JFYI, I get below while applying patch:

========================
shveta@shveta-vm:~/repos/postgres1/postgres$ git am
~/Desktop/shared/reuse/v10-0002-Reuse-Logical-Replication-Background-worker.patch
Applying: Reuse Logical Replication Background worker
.git/rebase-apply/patch:142: trailing whitespace.
        values[Anum_pg_subscription_rel_srrelslotname - 1] =
.git/rebase-apply/patch:692: indent with spaces.
                    errmsg("could not receive list of slots associated
with the subscription %u, error: %s",
.git/rebase-apply/patch:1055: trailing whitespace.
        /*
.git/rebase-apply/patch:1057: trailing whitespace.
         * relations.
.git/rebase-apply/patch:1059: trailing whitespace.
         * and origin. Then stop the worker gracefully.
warning: 5 lines add whitespace errors.
 ========================



thanks
Shveta

  46. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-03T06:20:27Z

    On Thu, Feb 2, 2023 at 5:01 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> Reviewing further....
>

Few more comments for v10-0002 and v7-0001:

1)
+ * need_full_snapshot
+ * if true, create a snapshot able to read all tables,
+ * otherwise do not create any snapshot.
+ *
CreateDecodingContext(..,CreateDecodingContext,..)

--Is the comment correct? Shall we have same comment here as that of
'CreateDecodingContext'
 * need_full_snapshot -- if true, must obtain a snapshot able to read all
 *              tables; if false, one that can read only catalogs is acceptable.
This function is not going to create a snapshot anyways. It is just a
pre-step and then the caller needs to call 'SnapBuild' functions to
build a snapshot. Here need_full_snapshot decides whether we need all
tables or only catalog tables changes only and thus the comment change
is needed.

==========

2)

Can we please add more logging:

2a)
when lastusedId is incremented and updated in pg_* table
ereport(DEBUG2,
(errmsg("[subid:%d] Incremented lastusedid
to:%ld",MySubscription->oid, MySubscription->lastusedid)));


Comments for LogicalRepSyncTableStart():

2b ) After every UpdateSubscriptionRel:

ereport(DEBUG2,
(errmsg("[subid:%d] LogicalRepSyncWorker_%ld updated origin to %s and
slot to %s for relid %d",
MyLogicalRepWorker->subid, MyLogicalRepWorker->rep_slot_id,
originname, slotname, MyLogicalRepWorker->relid)));


2c )
After walrcv_create_slot:

ereport(DEBUG2,
(errmsg("[subid:%d] LogicalRepSyncWorker_%ld created slot %s",
MyLogicalRepWorker->subid, MyLogicalRepWorker->rep_slot_id, slotname)));


2d)
After replorigin_create:

ereport(DEBUG2,
(errmsg("[subid:%d] LogicalRepSyncWorker_%ld created origin %s [id: %d]",
MyLogicalRepWorker->subid, MyLogicalRepWorker->rep_slot_id,
originname, originid)));


2e)
When it goes to reuse flow (i.e. before walrcv_slot_snapshot), if
needed we can dump newly obtained origin_startpos also:

ereport(DEBUG2,
(errmsg("[subid:%d] LogicalRepSyncWorker_%ld reusing slot %s and origin %s",
MyLogicalRepWorker->subid, MyLogicalRepWorker->rep_slot_id, slotname,
originname)));


2f)
Also in existing comment:

+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" has moved to sync table \"%s\".",
+ MySubscription->name, get_rel_name(MyLogicalRepWorker->relid))));

we can add relid also along with relname. relid is the one stored in
pg_subscription_rel and thus it becomes easy to map. Also we can
change 'logical replication table synchronization worker' to
'LogicalRepSyncWorker_%ld'.
Same change needed in other similar log lines where we say that worker
started and finished.


Please feel free to change the above log lines as you find
appropriate. I have given just a sample sort of thing.

thanks
Shveta

  47. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-03T06:34:58Z

    On Fri, Feb 3, 2023 at 11:50 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Thu, Feb 2, 2023 at 5:01 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
>
> 2e)
> When it goes to reuse flow (i.e. before walrcv_slot_snapshot), if
> needed we can dump newly obtained origin_startpos also:
>
> ereport(DEBUG2,
> (errmsg("[subid:%d] LogicalRepSyncWorker_%ld reusing slot %s and origin %s",
> MyLogicalRepWorker->subid, MyLogicalRepWorker->rep_slot_id, slotname,
> originname)));
>

One addition, I think it will be good to add relid as well in above so
that we can get info as in we are reusing old slot for which relid.
Once we have all the above in log-file, it makes it very easy to
diagnose reuse-sync worker related problems just by looking at the
logfile.

thanks
Shveta

  48. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-03T10:19:23Z

    On Thu, Feb 2, 2023 at 5:01 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> Reviewing further....
>

Hi Melih,

int64 rep_slot_id;
int64 lastusedid;
int64 sublastusedid

--Should all of the above be unsigned integers?

thanks
Shveta

  49. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2023-02-07T02:48:47Z

    On Thu, Feb 2, 2023 11:48 AM shveta malik <shveta.malik@gmail.com> wrote:
> 
> On Wed, Feb 1, 2023 at 5:42 PM Melih Mutlu <m.melihmutlu@gmail.com>
> wrote:
> >
> > Hi Shveta,
> >
> > shveta malik <shveta.malik@gmail.com>, 1 Şub 2023 Çar, 15:01 tarihinde
> şunu yazdı:
> >>
> >> On Wed, Feb 1, 2023 at 5:05 PM Melih Mutlu <m.melihmutlu@gmail.com>
> wrote:
> >> 2) I found a crash in the previous patch (v9), but have not tested it
> >> on the latest yet. Crash happens when all the replication slots are
> >> consumed and we are trying to create new. I tweaked the settings like
> >> below so that it can be reproduced easily:
> >> max_sync_workers_per_subscription=3
> >> max_replication_slots = 2
> >> and then ran the test case shared by you. I think there is some memory
> >> corruption happening. (I did test in debug mode, have not tried in
> >> release mode). I tried to put some traces to identify the root-cause.
> >> I observed that worker_1 keeps on moving from 1 table to another table
> >> correctly, but at some point, it gets corrupted i.e. origin-name
> >> obtained for it is wrong and it tries to advance that and since that
> >> origin does not exist, it  asserts and then something else crashes.
> >> From log: (new trace lines added by me are prefixed by shveta, also
> >> tweaked code to have my comment 1 fixed to have clarity on worker-id).
> >>
> >> form below traces, it is clear that worker_1 was moving from one
> >> relation to another, always getting correct origin 'pg_16688_1', but
> >> at the end it got 'pg_16688_49' which does not exist. Second part of
> >> trace shows who updated 'pg_16688_49', it was done by worker_49
> which
> >> even did not get chance to create this origin due to max_rep_slot
> >> reached.
> >
> >
> > Thanks for investigating this error. I think it's the same error as the one Shi
> reported earlier. [1]
> > I couldn't reproduce it yet but will apply your tweaks and try again.
> > Looking into this.
> >
> > [1] https://www.postgresql.org/message-
> id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpn
> prd01.prod.outlook.com
> >
> 
> Hi Melih,
> I think I am able to identify the root cause. It is not memory
> corruption, but the way origin-names are stored in system-catalog
> mapped to a particular relation-id before even those are created.
> 
> After adding few more logs:
> 
> [4227] LOG:  shveta- LogicalRepSyncTableStart- worker_49 constructed
> originname :pg_16684_49, relid:16540
> [4227] LOG:  shveta- LogicalRepSyncTableStart- worker_49
> updated-origin in system catalog:pg_16684_49,
> slot:pg_16684_sync_49_7195149685251088378, relid:16540
> [4227] LOG:  shveta- LogicalRepSyncTableStart- worker_49
> get-origin-id2 originid:0, originname:pg_16684_49
> [4227] ERROR:  could not create replication slot
> "pg_16684_sync_49_7195149685251088378": ERROR:  all replication slots
> are in use
>         HINT:  Free one or increase max_replication_slots.
> 
> 
> [4428] LOG:  shveta- LogicalRepSyncTableStart- worker_148 constructed
> originname :pg_16684_49, relid:16540
> [4428] LOG:  could not drop replication slot
> "pg_16684_sync_49_7195149685251088378" on publisher: ERROR:
> replication slot "pg_16684_sync_49_7195149      685251088378" does not
> exist
> [4428] LOG:  shveta- LogicalRepSyncTableStart- worker_148 drop-origin
> originname:pg_16684_49
> [4428] LOG:  shveta- LogicalRepSyncTableStart- worker_148
> updated-origin:pg_16684_49,
> slot:pg_16684_sync_148_7195149685251088378, relid:16540
> 
> So from above, worker_49 came and picked up relid:16540, constructed
> origin-name and slot-name and updated in system-catalog and then it
> tried to actually create that slot and origin but since max-slot count
> was reached, it failed and exited, but did not do cleanup from system
> catalog for that relid.
> 
> Then worker_148 came and also picked up table 16540 since it was not
> completed/started by previous worker, but this time it found that
> origin and slot entry present in system-catalog against this relid, so
> it picked the same names and started processing, but since those do
> not exist, it asserted while advancing the origin.
> 
> The assert is only reproduced when an already running worker (say
> worker_1) who has 'created=true' set, gets to sync the relid for which
> a previously failed worker has tried and updated origin-name w/o
> creating it. In such a case worker_1 (with created=true) will try to
> reuse the origin and thus will try to advance it and will end up
> asserting. That is why you might not see the assertion always. The
> condition 'created' is set to true for that worker and it goes to
> reuse the origin updated by the previous worker.
> 
> So to fix this, I think either we update origin and slot entries in
> the system catalog after the creation has passed or we clean-up the
> system catalog in case of failure. What do you think?
> 

I think the first way seems better.

I reproduced the problem I reported before with latest patch (v7-0001,
v10-0002), and looked into this problem. It is caused by a similar reason. Here
is some analysis for the problem I reported [1].#6.

First, a tablesync worker (worker-1) started for "tbl1", its originname is
"pg_16398_1". And it exited because of unique constraint. In
LogicalRepSyncTableStart(), originname in pg_subscription_rel is updated when
updating table state to DATASYNC, and the origin is created when updating table
state to FINISHEDCOPY. So when it exited with state DATASYNC , the origin is not
created but the originname has been updated in pg_subscription_rel.

Then a tablesync worker (worker-2) started for "tbl2", its originname is
"pg_16398_2". After tablesync of "tbl2" finished, this worker moved to sync
table "tbl1". In LogicalRepSyncTableStart(), it got the originname of "tbl1" -
"pg_16398_1", by calling ReplicationOriginNameForLogicalRep(), and tried to drop
the origin (although it is not actually created before). After that, it called
replorigin_by_name to get the originid whose name is "pg_16398_1" and the result
is InvalidOid. Origin won't be created in this case because the sync worker has
created a replication slot (when it synced tbl2), so the originid was still
invalid and it caused an assertion failure when calling replorigin_advance().

It seems we don't need to drop previous origin in worker-2 because the previous
origin was not created in worker-1. I think one way to fix it is to not update
originname of pg_subscription_rel when setting state to DATASYNC, and only do
that when setting state to FINISHEDCOPY. If so, the originname in
pg_subscription_rel will be set at the same time the origin is created.
(Besides, the slotname seems need to be updated when setting state to DATASYNC,
because the previous slot might have been created successfully and we need to get
the previous slotname and drop that.)

[1] https://www.postgresql.org/message-id/OSZPR01MB631013C833C98E826B3CFCB9FDC69%40OSZPR01MB6310.jpnprd01.prod.outlook.com

Regards,
Shi yu

  50. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2023-02-07T07:28:33Z

    On Wed, Feb 1, 2023 20:07 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> Thanks for pointing out this review. I somehow skipped that, sorry.
> 
> Please see attached patches.

Thanks for updating the patch set.
Here are some comments.

1. In the function ApplyWorkerMain.
+			/* This is main apply worker */
+			run_apply_worker(&options, myslotname, originname, sizeof(originname), &origin_startpos);

I think we need to keep the worker name as "leader apply worker" in the comment
like the current HEAD.

---

2. In the function LogicalRepApplyLoop.
+				 * can be reused, we need to take care of memory contexts here
+				 * before moving to sync a table.
+				 */
+				if (MyLogicalRepWorker->ready_to_reuse)
+				{
+					MemoryContextResetAndDeleteChildren(ApplyMessageContext);
+					MemoryContextSwitchTo(TopMemoryContext);
+					return;
+				}

I think in this case we also need to pop the error context stack before
returning. Otherwise, I think we might use the wrong callback
(apply error_callback) after we return from this function.

---

3. About the function UpdateSubscriptionRelReplicationSlot.
This newly introduced function UpdateSubscriptionRelReplicationSlot does not
seem to be invoked. Do we need this function?

Regards,
Wang Wei

  51. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    shveta malik <shveta.malik@gmail.com> — 2023-02-08T13:48:51Z

    On Tue, Feb 7, 2023 at 8:18 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
>
> On Thu, Feb 2, 2023 11:48 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> >
> > So to fix this, I think either we update origin and slot entries in
> > the system catalog after the creation has passed or we clean-up the
> > system catalog in case of failure. What do you think?
> >
>
> I think the first way seems better.

Yes, I agree.

>
> I reproduced the problem I reported before with latest patch (v7-0001,
> v10-0002), and looked into this problem. It is caused by a similar reason. Here
> is some analysis for the problem I reported [1].#6.
>
> First, a tablesync worker (worker-1) started for "tbl1", its originname is
> "pg_16398_1". And it exited because of unique constraint. In
> LogicalRepSyncTableStart(), originname in pg_subscription_rel is updated when
> updating table state to DATASYNC, and the origin is created when updating table
> state to FINISHEDCOPY. So when it exited with state DATASYNC , the origin is not
> created but the originname has been updated in pg_subscription_rel.
>
> Then a tablesync worker (worker-2) started for "tbl2", its originname is
> "pg_16398_2". After tablesync of "tbl2" finished, this worker moved to sync
> table "tbl1". In LogicalRepSyncTableStart(), it got the originname of "tbl1" -
> "pg_16398_1", by calling ReplicationOriginNameForLogicalRep(), and tried to drop
> the origin (although it is not actually created before). After that, it called
> replorigin_by_name to get the originid whose name is "pg_16398_1" and the result
> is InvalidOid. Origin won't be created in this case because the sync worker has
> created a replication slot (when it synced tbl2), so the originid was still
> invalid and it caused an assertion failure when calling replorigin_advance().
>
> It seems we don't need to drop previous origin in worker-2 because the previous
> origin was not created in worker-1. I think one way to fix it is to not update
> originname of pg_subscription_rel when setting state to DATASYNC, and only do
> that when setting state to FINISHEDCOPY. If so, the originname in
> pg_subscription_rel will be set at the same time the origin is created.

+1. Update of system-catalog needs to be done carefully and only when
origin is created.

thanks
Shveta

  52. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2023-02-14T03:36:52Z

    On Thur, Feb 7, 2023 15:29 PM I wrote:
> On Wed, Feb 1, 2023 20:07 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> > Thanks for pointing out this review. I somehow skipped that, sorry.
> >
> > Please see attached patches.
> 
> Thanks for updating the patch set.
> Here are some comments.

Hi, here are some more comments on patch v7-0001*:

1. The new comments atop the function CreateDecodingContext
+ * need_full_snapshot
+ * 		if true, create a snapshot able to read all tables,
+ * 		otherwise do not create any snapshot.

I think if 'need_full_snapshot' is false, it means we will create a snapshot
that can read only catalogs. (see SnapBuild->building_full_snapshot)

===

2. This are two questions I'm not sure about.
2a.
Because pg-doc has the following description in [1]: (option "SNAPSHOT 'use'")
```
'use' will use the snapshot for the current transaction executing the command.
This option must be used in a transaction, and CREATE_REPLICATION_SLOT must be
the first command run in that transaction.
```
So I think in the function CreateDecodingContext, when "need_full_snapshot" is
true, we seem to need the following check, just like in the function
CreateInitDecodingContext:
```
	if (IsTransactionState() &&
		GetTopTransactionIdIfAny() != InvalidTransactionId)
		ereport(ERROR,
				(errcode(ERRCODE_ACTIVE_SQL_TRANSACTION),
				 errmsg("cannot create logical replication slot in transaction that has performed writes")));
```

2b.
It seems that we also need to invoke the function
CheckLogicalDecodingRequirements in the new function CreateReplicationSnapshot,
just like the function CreateReplicationSlot and the function
StartLogicalReplication.

Is there any reason not to do these two checks? Please let me know if I missed
something.

===

3. The invocation of startup_cb_wrapper in the function CreateDecodingContext.
I think we should change the third input parameter to true when invoke function 
startup_cb_wrapper for CREATE_REPLICATION_SNAPSHOT. BTW, after applying patch
v10-0002*, these settings will be inconsistent when sync workers use
"CREATE_REPLICATION_SLOT" and "CREATE_REPLICATION_SNAPSHOT" to take snapshots.
This input parameter (true) will let us disable streaming and two-phase
transactions in function pgoutput_startup. See the last paragraph of the commit
message for 4648243 for more details.

[1] - https://www.postgresql.org/docs/devel/protocol-replication.html

Regards,
Wang Wei

  53. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-16T11:37:19Z

    Hi Shveta and Shi,

Thanks for your investigations.

shveta malik <shveta.malik@gmail.com>, 8 Şub 2023 Çar, 16:49 tarihinde şunu
yazdı:

> On Tue, Feb 7, 2023 at 8:18 AM shiy.fnst@fujitsu.com
> <shiy.fnst@fujitsu.com> wrote:
> > I reproduced the problem I reported before with latest patch (v7-0001,
> > v10-0002), and looked into this problem. It is caused by a similar
> reason. Here
> > is some analysis for the problem I reported [1].#6.
> >
> > First, a tablesync worker (worker-1) started for "tbl1", its originname
> is
> > "pg_16398_1". And it exited because of unique constraint. In
> > LogicalRepSyncTableStart(), originname in pg_subscription_rel is updated
> when
> > updating table state to DATASYNC, and the origin is created when
> updating table
> > state to FINISHEDCOPY. So when it exited with state DATASYNC , the
> origin is not
> > created but the originname has been updated in pg_subscription_rel.
> >
> > Then a tablesync worker (worker-2) started for "tbl2", its originname is
> > "pg_16398_2". After tablesync of "tbl2" finished, this worker moved to
> sync
> > table "tbl1". In LogicalRepSyncTableStart(), it got the originname of
> "tbl1" -
> > "pg_16398_1", by calling ReplicationOriginNameForLogicalRep(), and tried
> to drop
> > the origin (although it is not actually created before). After that, it
> called
> > replorigin_by_name to get the originid whose name is "pg_16398_1" and
> the result
> > is InvalidOid. Origin won't be created in this case because the sync
> worker has
> > created a replication slot (when it synced tbl2), so the originid was
> still
> > invalid and it caused an assertion failure when calling
> replorigin_advance().
> >
> > It seems we don't need to drop previous origin in worker-2 because the
> previous
> > origin was not created in worker-1. I think one way to fix it is to not
> update
> > originname of pg_subscription_rel when setting state to DATASYNC, and
> only do
> > that when setting state to FINISHEDCOPY. If so, the originname in
> > pg_subscription_rel will be set at the same time the origin is created.
>
> +1. Update of system-catalog needs to be done carefully and only when
> origin is created.
>

I see that setting originname in the catalog before actually creating it
causes issues. My concern with setting originname when setting the state to
FINISHEDCOPY is that if worker waits until FINISHEDCOPY to update
slot/origin name but it fails somewhere before reaching FINISHEDCOPY and
after creating slot/origin, then this new created slot/origin will be left
behind. It wouldn't be possible to find and drop them since their names are
not stored in the catalog. Eventually, this might also cause hitting
the max_replication_slots limit in case of such failures between origin
creation and FINISHEDCOPY.

One fix I can think is to update the catalog right after creating a new
origin. But this would also require commiting the current transaction to
actually persist the originname. I guess this action of commiting the
transaction in the middle of initial sync could hurt the copy process.

What do you think?

Also; working on an updated patch to address your other comments. Thanks
again.

Best,
-- 
Melih Mutlu
Microsoft

  54. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-22T12:51:35Z

    Hi,

Melih Mutlu <m.melihmutlu@gmail.com>, 16 Şub 2023 Per, 14:37 tarihinde şunu
yazdı:

> I see that setting originname in the catalog before actually creating it
> causes issues. My concern with setting originname when setting the state to
> FINISHEDCOPY is that if worker waits until FINISHEDCOPY to update
> slot/origin name but it fails somewhere before reaching FINISHEDCOPY and
> after creating slot/origin, then this new created slot/origin will be left
> behind. It wouldn't be possible to find and drop them since their names are
> not stored in the catalog. Eventually, this might also cause hitting
> the max_replication_slots limit in case of such failures between origin
> creation and FINISHEDCOPY.
>
> One fix I can think is to update the catalog right after creating a new
> origin. But this would also require commiting the current transaction to
> actually persist the originname. I guess this action of commiting the
> transaction in the middle of initial sync could hurt the copy process.
>

Here are more thoughts on this:
I still believe that updating originname when setting state
to FINISHEDCOPY is not a good idea since any failure
before FINISHEDCOPY prevent us to store originname in the catalog. If an
origin or slot is not in the catalog, it's not easily possible to find and
drop origins/slot that are left behind. And we definitely do not want to
keep unnecessary origins/slots since we would hit max_replication_slots
limit.
It's better to be safe and update origin/slot names when setting state
to DATASYNC. At this point, the worker must be sure that it writes correct
origin/slot names into the catalog.
Following part actually cleans up the catalog if a table is left behind in
DATASYNC state and its slot and origin cannot be used for sync.

ReplicationSlotDropAtPubNode(LogRepWorkerWalRcvConn, prev_slotname, true);
>
> StartTransactionCommand();
> /* Replication origin might still exist. Try to drop */
> replorigin_drop_by_name(originname, true, false);
>
> /*
> * Remove replication slot and origin name from the relation's
> * catalog record
> */
> UpdateSubscriptionRel(MyLogicalRepWorker->subid,
>  MyLogicalRepWorker->relid,
>  MyLogicalRepWorker->relstate,
>  MyLogicalRepWorker->relstate_lsn,
>  NULL,
>  NULL);
>

The patch needs to refresh the origin name before it begins copying the
table. It will try to read from the catalog but won't find any slot/origin
since they are cleaned. Then, it will move on with the correct origin name
which is the one created/will be created for the current sync worker.

I tested refetching originname and seems like it fixes the errors you
reported.

Thanks,
-- 
Melih Mutlu
Microsoft

  55. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-22T12:56:00Z

    Hi Shveta,

Thanks for reviewing.
Please see attached patches.

shveta malik <shveta.malik@gmail.com>, 2 Şub 2023 Per, 14:31 tarihinde şunu
yazdı:

> On Wed, Feb 1, 2023 at 5:37 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>         for (int64 i = 1; i <= lastusedid; i++)
>         {
>                 char            originname_to_drop[NAMEDATALEN] = {0};
>                 snprintf(originname_to_drop,
> sizeof(originname_to_drop), "pg_%u_%lld", subid, (long long) i);
>          .......
>           }
>
> --Is it better to use the function
> 'ReplicationOriginNameForLogicalRep' here instead of sprintf, just to
> be consistent everywhere to construct origin-name?
>

ReplicationOriginNameForLogicalRep creates a slot name with current
"lastusedid" and doesn't accept that id as parameter. Here the patch needs
to check all possible id's.


> /* Drop replication origin */
> replorigin_drop_by_name(originname, true, false);
> }
>
> --Are we passing missing_ok as true (second arg) intentionally here in
> replorigin_drop_by_name? Once we fix the issue reported  in my earlier
> email (ASSERT), do you think it makes sense to  pass missing_ok as
> false here?
>

Yes, missing_ok is intentional. The user might be concurrently refreshing
the sub or the apply worker might already drop the origin at that point.
So, missing_ok is set to true.
This is also how origin drops before the worker exits are handled on HEAD
too. I only followed the same approach.


> --Do we need to palloc for each relation separately? Shall we do it
> once outside the loop and reuse it? Also pfree is not done for rstate
> here.
>

Removed palloc from the loop. No need to pfree here since the memory
context will be deleted with the next CommitTransactionCommand call.


> Can you please review the above flow (I have given line# along with),
> I think it could be problematic. We alloced prev_slotname, assigned it
> to slotname, freed prev_slotname and used slotname after freeing the
> prev_slotname.
> Also slotname is allocated some memory too, that is not freed.
>

Right, I used memcpy instead of assigning prev_slotname to slotname.
slotname is returned in the end and pfree'd later [1]

I also addressed your other reviews that I didn't explicitly mention in
this email. I simply applied the changes you pointed out. Also added some
more logs as well. I hope it's more useful now.

[1]
https://github.com/postgres/postgres/blob/master/src/backend/replication/logical/worker.c#L4359


Thanks,
-- 
Melih Mutlu
Microsoft

  56. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-02-22T13:04:00Z

    Hi Wang,

Thanks for reviewing.
Please see updated patches. [1]

wangw.fnst@fujitsu.com <wangw.fnst@fujitsu.com>, 7 Şub 2023 Sal, 10:28
tarihinde şunu yazdı:

> 1. In the function ApplyWorkerMain.
> I think we need to keep the worker name as "leader apply worker" in the
> comment
> like the current HEAD.
>

Done.


> I think in this case we also need to pop the error context stack before
> returning. Otherwise, I think we might use the wrong callback
> (apply error_callback) after we return from this function.
>

Done.


> 3. About the function UpdateSubscriptionRelReplicationSlot.
> This newly introduced function UpdateSubscriptionRelReplicationSlot does
> not
> seem to be invoked. Do we need this function?


Removed.

I think if 'need_full_snapshot' is false, it means we will create a snapshot
> that can read only catalogs. (see SnapBuild->building_full_snapshot)


Fixed.

```
> 'use' will use the snapshot for the current transaction executing the
> command.
> This option must be used in a transaction, and CREATE_REPLICATION_SLOT
> must be
> the first command run in that transaction.
> ```

So I think in the function CreateDecodingContext, when "need_full_snapshot"
> is
> true, we seem to need the following check, just like in the function
> CreateInitDecodingContext:

```
>         if (IsTransactionState() &&
>                 GetTopTransactionIdIfAny() != InvalidTransactionId)
>                 ereport(ERROR,
>                                 (errcode(ERRCODE_ACTIVE_SQL_TRANSACTION),
>                                  errmsg("cannot create logical replication
> slot in transaction that has performed writes")));
> ```


You're right to "use" the snapshot, it must be the first command in the
transaction. And that check happens here [2]. CreateReplicationSnapshot has
also similar check.
I think the check you're referring to is needed to actually create a
replication slot and it performs whether the snapshot will be "used" or
"exported". This is not the case for CreateReplicationSnapshot.

It seems that we also need to invoke the function
> CheckLogicalDecodingRequirements in the new function
> CreateReplicationSnapshot,
> just like the function CreateReplicationSlot and the function
> StartLogicalReplication.


Added this check.

3. The invocation of startup_cb_wrapper in the function
> CreateDecodingContext.
> I think we should change the third input parameter to true when invoke
> function
> startup_cb_wrapper for CREATE_REPLICATION_SNAPSHOT. BTW, after applying
> patch
> v10-0002*, these settings will be inconsistent when sync workers use
> "CREATE_REPLICATION_SLOT" and "CREATE_REPLICATION_SNAPSHOT" to take
> snapshots.
> This input parameter (true) will let us disable streaming and two-phase
> transactions in function pgoutput_startup. See the last paragraph of the
> commit
> message for 4648243 for more details.


I'm not sure if "is_init" should be set to true. CreateDecodingContext only
creates a context for an already existing logical slot and does not
initialize new one.
I think inconsistencies between "CREATE_REPLICATION_SLOT" and
"CREATE_REPLICATION_SNAPSHOT" are expected since one creates a new
replication slot and the other does not.
CreateDecodingContext is also used in other places as well. Not sure how
this change would affect those places. I'll look into this more. Please let
me know if I'm missing something.


[1]
https://www.postgresql.org/message-id/CAGPVpCQmEE8BygXr%3DHi2N2t2kOE%3DPJwofn9TX0J9J4crjoXarQ%40mail.gmail.com
[2]
https://github.com/postgres/postgres/blob/master/src/backend/replication/walsender.c#L1108

Thanks,
-- 
Melih Mutlu
Microsoft

  57. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melanie Plageman <melanieplageman@gmail.com> — 2023-02-27T00:10:33Z

    On Wed, Feb 22, 2023 at 8:04 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Wang,
>
> Thanks for reviewing.
> Please see updated patches. [1]

This is cool! Thanks for working on this.
I had a chance to review your patchset and I had some thoughts and
questions.

I notice that you've added a new user-facing option to make a snapshot.
I think functionality to independently make a snapshot for use elsewhere
has been discussed in the past for the implementation of different
features (e.g. [1] pg_dump but they ended up using replication slots for
this I think?), but I'm not quite sure I understand all the implications
for providing a user-visible create snapshot command. Where can it be
used? When can the snapshot be used? In your patch's case, you know that
you can use the snapshot you are creating, but I just wonder if any
restrictions or caveats need be taken for its general use.

For the worker reuse portion of the code, could it be a separate patch
in the set? It could be independently committable and would be easier to
review (separate from repl slot reuse).

Given table sync worker reuse, I think it is worth considering a more
explicit structure for the table sync worker code now -- i.e. having a
TableSyncWorkerMain() function. Though they still do the
LogicalRepApplyLoop(), much of what else they do is different than the
apply leader.

Apply worker leader does:

ApplyWorkerMain()
    walrcv_startstreaming()
    LogicalRepApplyLoop()
        launch table sync workers
    walrcv_endstreaming()
    proc_exit()

Table Sync workers master:

ApplyWorkerMain()
    start_table_sync()
        walrcv_create_slot()
        copy_table()
    walrcv_startstreaming()
    start_apply()
        LogicalRepApplyLoop()
        walrcv_endstreaming()
        proc_exit()

Now table sync workers need to loop back and do start_table_sync() again
for their new table.
You have done this in ApplyWorkerMain(). But I think that this could be
a separate main function since their main loop is effectively totally
different now than an apply worker leader.

Something like:

TableSyncWorkerMain()
    TableSyncWorkerLoop()
        start_table_sync()
        walrcv_startstreaming()
        LogicalRepApplyLoop()
        walrcv_endstreaming()
        wait_for_new_rel_assignment()
    proc_exit()

You mainly have this structure, but it is a bit hidden and some of the
shared functions that previously may have made sense for table sync
worker and apply workers to share don't really make sense to share
anymore.

The main thing that table sync workers and apply workers share is the
logic in LogicalRepApplyLoop() (table sync workers use when they do
catchup), so perhaps we should make the other code separate?

Also on the topic of worker reuse, I was wondering if having workers
find their own next table assignment (as you have done in
process_syncing_tables_for_sync()) makes sense.

The way the whole system would work now (with your patch applied), as I
understand it, the apply leader would loop through the subscription rel
states and launch workers up to max_sync_workers_per_subscription for
every candidate table needing sync. The apply leader will continue to do
this, even though none of those workers would exit unless they die
unexpectedly. So, once it reaches max_sync_workers_per_subscription, it
won't launch any more workers.

When one of these sync workers is finished with a table (it is synced
and caught up), it will search through the subscription rel states
itself looking for a candidate table to work on.

It seems it would be common for workers to be looking through the
subscription rel states at the same time, and I don't really see how you
prevent races in who is claiming a relation to work on. Though you take
a shared lock on the LogicalRepWorkerLock, what if in between
logicalrep_worker_find() and updating my MyLogicalRepWorker->relid,
someone else also updates their relid to that relid. I don't think you
can update LogicalRepWorker->relid with only a shared lock.

I wonder if it is not better to have the apply leader, in
process_syncing_tables_for_apply(), first check for an existing worker
for the rel, then check for an available worker without an assignment,
then launch a worker?

Workers could then sleep after finishing their assignment and wait for
the leader to give them a new assignment.

Given an exclusive lock on LogicalRepWorkerLock, it may be okay for
workers to find their own table assignments from the subscriptionrel --
and perhaps this will be much more efficient from a CPU perspective. It
feels just a bit weird to have the code doing that buried in
process_syncing_tables_for_sync(). It seems like it should at least
return out to a main table sync worker loop in which workers loop
through finding a table and assigning it to themselves, syncing the
table, and catching the table up.

- Melanie

[1] https://www.postgresql.org/message-id/flat/CA%2BU5nMLRjGtpskUkYSzZOEYZ_8OMc02k%2BO6FDi4una3mB4rS1w%40mail.gmail.com#45692f75a1e79d4ce2d4f6a0e3ccb853

  58. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Gregory Stark (as CFM) <stark.cfm@gmail.com> — 2023-04-04T14:51:46Z

    On Sun, 26 Feb 2023 at 19:11, Melanie Plageman
<melanieplageman@gmail.com> wrote:
>
> This is cool! Thanks for working on this.
> I had a chance to review your patchset and I had some thoughts and
> questions.

It looks like this patch got a pretty solid review from Melanie
Plageman in February just before the CF started. It was never set to
Waiting on Author but I think that may be the right state for it.

-- 
Gregory Stark
As Commitfest Manager

  59. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-05-08T15:41:26Z

    Hi Melanie,

Thanks for reviewing.

Melanie Plageman <melanieplageman@gmail.com>, 27 Şub 2023 Pzt, 03:10
tarihinde şunu yazdı:
>
> I notice that you've added a new user-facing option to make a snapshot.
> I think functionality to independently make a snapshot for use elsewhere
> has been discussed in the past for the implementation of different
> features (e.g. [1] pg_dump but they ended up using replication slots for
> this I think?), but I'm not quite sure I understand all the implications
> for providing a user-visible create snapshot command. Where can it be
> used? When can the snapshot be used? In your patch's case, you know that
> you can use the snapshot you are creating, but I just wonder if any
> restrictions or caveats need be taken for its general use.


I can't say a use-case, other than this patch, that needs this user-facing
command. The main reason why I added this command as it is in the patch is
because that's already how other required communication between publisher
and subscriber is done for other operations in logical replication. Even
though it may sound similar to the case in pg_dump discussion, I think the
main difference is that calling CREATE_REPLICATION_SNAPSHOT creates a
snapshot and imports it to wherever it's called (i.e. the same transaction
which invoked CREATE_REPLICATION_SNAPSHOT ), and not used anywhere else.
But I agree that this part of the patch needs more thoughts and reviews.
Honestly, I'm not also sure if this is the ideal way to fix the "snapshot
issue" introduced by reusing the same replication slot.

>
> For the worker reuse portion of the code, could it be a separate patch
> in the set? It could be independently committable and would be easier to
> review (separate from repl slot reuse).

I did this, please see the patch 0001.

>
> You mainly have this structure, but it is a bit hidden and some of the
> shared functions that previously may have made sense for table sync
> worker and apply workers to share don't really make sense to share
> anymore.
>
> The main thing that table sync workers and apply workers share is the
> logic in LogicalRepApplyLoop() (table sync workers use when they do
> catchup), so perhaps we should make the other code separate?

You're right that apply and tablesync worker's paths are unnecessarily
intertwined. With the reusing workers/replication slots logic, I guess it
became worse.
I tried to change the structure to something similar to what you explained.
Tablesync workers have different starting point now and it simply runs as
follows:

TableSyncWorkerMain()
    loop:
        start_table_sync()
        walrcv_startstreaming()
        LogicalRepApplyLoop()
        check if there is a table with INIT state
        if there is such table: // reuse case
            clean_sync_worker()
       else: // exit case
            walrcv_endstreaming()
            ReplicationSlotDropAtPubNode()
            replorigin_drop_by_name
            break
    proc_exit()

> It seems it would be common for workers to be looking through the
> subscription rel states at the same time, and I don't really see how you
> prevent races in who is claiming a relation to work on. Though you take
> a shared lock on the LogicalRepWorkerLock, what if in between
> logicalrep_worker_find() and updating my MyLogicalRepWorker->relid,
> someone else also updates their relid to that relid. I don't think you
> can update LogicalRepWorker->relid with only a shared lock.
>
>
> I wonder if it is not better to have the apply leader, in
> process_syncing_tables_for_apply(), first check for an existing worker
> for the rel, then check for an available worker without an assignment,
> then launch a worker?
>
> Workers could then sleep after finishing their assignment and wait for
> the leader to give them a new assignment.

I'm not sure if we should rely on a single apply worker for the assignment
of several tablesync workers. I suspect that moving the assignment
responsibility to the apply worker may bring some overhead. But I agree
that shared lock on LogicalRepWorkerLock is not good. Changed it to
exclusive lock.

>
> Given an exclusive lock on LogicalRepWorkerLock, it may be okay for
> workers to find their own table assignments from the subscriptionrel --
> and perhaps this will be much more efficient from a CPU perspective. It
> feels just a bit weird to have the code doing that buried in
> process_syncing_tables_for_sync(). It seems like it should at least
> return out to a main table sync worker loop in which workers loop
> through finding a table and assigning it to themselves, syncing the
> table, and catching the table up.

Right, it shouldn't be process_syncing_tables_for_sync()'s responsibility.
I moved it into the TableSyncWorkerMain loop.


Also;
I did some benchmarking like I did a couple of times previously [1].
Here are the recent numbers:

With empty tables:
+--------+------------+-------------+--------------+
|        |  10 tables | 100 tables  | 1000 tables  |
+--------+------------+-------------+--------------+
| master | 296.689 ms | 2579.154 ms | 41753.043 ms |
+--------+------------+-------------+--------------+
| patch  | 210.580 ms | 1724.230 ms | 36247.061 ms |
+--------+------------+-------------+--------------+

With 10 tables loaded with some data:
+--------+------------+-------------+--------------+
|        |    1 MB    | 10 MB       | 100 MB       |
+--------+------------+-------------+--------------+
| master | 568.072 ms | 2074.557 ms | 16995.399 ms |
+--------+------------+-------------+--------------+
| patch  | 470.700 ms | 1923.386 ms | 16980.686 ms |
+--------+------------+-------------+--------------+

It seems that even though master has improved since the last time I did a
similar experiment, the patch still improves the time spent in table sync
for empty/small tables.
Also there is a decrease in the performance of the patch, compared with the
previous results [1]. Some portion of it might be caused by switching from
shared locks to exclusive locks. I'll look into that a bit more though.



[1]
https://www.postgresql.org/message-id/CAGPVpCQdZ_oj-QFcTOhTrUTs-NCKrrZ%3DZNCNPR1qe27rXV-iYw%40mail.gmail.com

Best,
-- 
Melih Mutlu
Microsoft

  60. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-05-24T02:59:18Z

    Hi, and thanks for the patch! It is an interesting idea.

I have not yet fully read this thread, so below are only my first
impressions after looking at patch 0001. Sorry if some of these were
already discussed earlier.

TBH the patch "reuse-workers" logic seemed more complicated than I had
imagined it might be.

1.
IIUC with patch 0001, each/every tablesync worker (a.k.a. TSW) when it
finishes dealing with one table then goes looking to find if there is
some relation that it can process next. So now every TSW has a loop
where it will fight with every other available TSW over who will get
to process the next relation.

Somehow this seems all backwards to me. Isn't it strange for the TSW
to be the one deciding what relation it would deal with next?

IMO it seems more natural to simply return the finished TSW to some
kind of "pool" of available workers and the main Apply process can
just grab a TSW from that pool instead of launching a brand new one in
the existing function process_syncing_tables_for_apply(). Or, maybe
those "available" workers can be returned to a pool maintained within
the launcher.c code, which logicalrep_worker_launch() can draw from
instead of launching a whole new process?

(I need to read the earlier posts to see if these options were already
discussed and rejected)

~~

2.
AFAIK the thing that identifies a  tablesync worker is the fact that
only TSW will have a 'relid'.

But it feels very awkward to me to have a TSW marked as "available"
and yet that LogicalRepWorker must still have some OLD relid field
value lurking (otherwise it will forget that it is a "tablesync"
worker!).

IMO perhaps it is time now to introduce some enum 'type' to the
LogicalRepWorker. Then an "in_use" type=TSW would have a valid 'relid'
whereas an "available" type=TSW would have relid == InvalidOid.

~~

3.
Maybe I am mistaken, but it seems the benchmark results posted are
only using quite a small/default values for
"max_sync_workers_per_subscription", so I wondered how those results
are affected by increasing that GUC. I think having only very few
workers would cause more sequential processing, so conveniently the
effect of the patch avoiding re-launch might be seen in the best
possible light. OTOH, using more TSW in the first place might reduce
the overall tablesync time because the subscriber can do more work in
parallel.

So I'm not quite sure what the goal is here. E.g. if the user doesn't
care much about how long tablesync phase takes then there is maybe no
need for this patch at all. OTOH, I thought if a user does care about
the subscription startup time, won't those users be opting for a much
larger "max_sync_workers_per_subscription" in the first place?
Therefore shouldn't the benchmarking be using a larger number too?

======

Here are a few other random things noticed while looking at patch 0001:

1. Commit message

1a. typo /sequantially/sequentially/

1b. Saying "killed" and "killing" seemed a bit extreme and implies
somebody else is killing the process. But I think mostly tablesync is
just ending by a normal proc exit, so maybe reword this a bit.

~~~

2. It seemed odd that some -- clearly tablesync specific -- functions
are in the worker.c instead of in tablesync.c.

2a. e.g. clean_sync_worker

2b. e.g. sync_worker_exit

~~~

3. process_syncing_tables_for_sync

+ /*
+ * Sync worker is cleaned at this point. It's ready to sync next table,
+ * if needed.
+ */
+ SpinLockAcquire(&MyLogicalRepWorker->relmutex);
+ MyLogicalRepWorker->ready_to_reuse = true;
  SpinLockRelease(&MyLogicalRepWorker->relmutex);
+ }
+
+ SpinLockRelease(&MyLogicalRepWorker->relmutex);

Isn't there a double release of that mutex happening there?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  61. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-05-25T08:59:26Z

    Hi,


Peter Smith <smithpb2250@gmail.com>, 24 May 2023 Çar, 05:59 tarihinde şunu
yazdı:

> Hi, and thanks for the patch! It is an interesting idea.
>
> I have not yet fully read this thread, so below are only my first
> impressions after looking at patch 0001. Sorry if some of these were
> already discussed earlier.
>
> TBH the patch "reuse-workers" logic seemed more complicated than I had
> imagined it might be.
>

If you mean patch 0001 by the patch "reuse-workers", most of the complexity
comes with some refactoring to split apply worker and tablesync worker
paths. [1]
If you mean the whole patch set, then I believe it's because reusing
replication slots also requires having a proper snapshot each time the
worker moves to a new table. [2]


>
> 1.
> IIUC with patch 0001, each/every tablesync worker (a.k.a. TSW) when it
> finishes dealing with one table then goes looking to find if there is
> some relation that it can process next. So now every TSW has a loop
> where it will fight with every other available TSW over who will get
> to process the next relation.
>
> Somehow this seems all backwards to me. Isn't it strange for the TSW
> to be the one deciding what relation it would deal with next?
>
> IMO it seems more natural to simply return the finished TSW to some
> kind of "pool" of available workers and the main Apply process can
> just grab a TSW from that pool instead of launching a brand new one in
> the existing function process_syncing_tables_for_apply(). Or, maybe
> those "available" workers can be returned to a pool maintained within
> the launcher.c code, which logicalrep_worker_launch() can draw from
> instead of launching a whole new process?
>
> (I need to read the earlier posts to see if these options were already
> discussed and rejected)
>

I think ([3]) relying on a single apply worker for the assignment of
several tablesync workers might bring some overhead, it's possible that
some tablesync workers wait in idle until the apply worker assigns them
something. OTOH yes, the current approach makes tablesync workers race for
a new table to sync.
TBF both ways might be worth discussing/investigating more, before deciding
which way to go.


> 2.
> AFAIK the thing that identifies a  tablesync worker is the fact that
> only TSW will have a 'relid'.
>
> But it feels very awkward to me to have a TSW marked as "available"
> and yet that LogicalRepWorker must still have some OLD relid field
> value lurking (otherwise it will forget that it is a "tablesync"
> worker!).
>
> IMO perhaps it is time now to introduce some enum 'type' to the
> LogicalRepWorker. Then an "in_use" type=TSW would have a valid 'relid'
> whereas an "available" type=TSW would have relid == InvalidOid.
>

Hmm, relid will be immediately updated when the worker moves to a new
table. And the time between finishing sync of a table and finding a new
table to sync should be minimal. I'm not sure how having an old relid for
such a small amount of time can do any harm.


> 3.
> Maybe I am mistaken, but it seems the benchmark results posted are
> only using quite a small/default values for
> "max_sync_workers_per_subscription", so I wondered how those results
> are affected by increasing that GUC. I think having only very few
> workers would cause more sequential processing, so conveniently the
> effect of the patch avoiding re-launch might be seen in the best
> possible light. OTOH, using more TSW in the first place might reduce
> the overall tablesync time because the subscriber can do more work in
> parallel.



So I'm not quite sure what the goal is here. E.g. if the user doesn't

care much about how long tablesync phase takes then there is maybe no
> need for this patch at all. OTOH, I thought if a user does care about
> the subscription startup time, won't those users be opting for a much
> larger "max_sync_workers_per_subscription" in the first place?
> Therefore shouldn't the benchmarking be using a larger number too?


Regardless of how many tablesync workers there are, reusing workers will
speed things up if a worker has a chance to sync more than one table.
Increasing the number of tablesync workers, of course, improves the
tablesync performance. But if it doesn't make 100% parallel ( meaning that
# of sync workers != # of tables to sync), then reusing workers can bring
an additional improvement.

Here are some benchmarks similar to earlier, but with 100 tables and
different number of workers:

+--------+-------------+-------------+-------------+------------+
|        | 2 workers   | 4 workers   | 6 workers   | 8 workers  |
+--------+-------------+-------------+-------------+------------+
| master | 2579.154 ms | 1383.153 ms | 1001.559 ms | 911.758 ms |
+--------+-------------+-------------+-------------+------------+
| patch  | 1724.230 ms | 853.894 ms  | 601.176 ms  | 496.395 ms |
+--------+-------------+-------------+-------------+------------+

So yes, increasing the number of workers makes it faster. But reusing
workers can still improve more.



[1]
https://www.postgresql.org/message-id/CAAKRu_YKGyF%2BsvRQqe1th-mG9xLdzneWgh9H1z1DtypBkawkkw%40mail.gmail.com
[2]
https://www.postgresql.org/message-id/CAGPVpCRWEVhXa7ovrhuSQofx4to7o22oU9iKtrOgAOtz_%3DY6vg%40mail.gmail.com
[3]
https://www.postgresql.org/message-id/CAGPVpCRzD-ZZEc9ienhyrVpCzd9AJ7fxE--OFFJBnBg3E0438w%40mail.gmail.com


Best,
-- 
Melih Mutlu
Microsoft

  62. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-05-26T07:29:38Z

    On Thu, May 25, 2023 at 6:59 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
>
> Peter Smith <smithpb2250@gmail.com>, 24 May 2023 Çar, 05:59 tarihinde şunu yazdı:
>>
>> Hi, and thanks for the patch! It is an interesting idea.
>>
>> I have not yet fully read this thread, so below are only my first
>> impressions after looking at patch 0001. Sorry if some of these were
>> already discussed earlier.
>>
>> TBH the patch "reuse-workers" logic seemed more complicated than I had
>> imagined it might be.
>
>
> If you mean patch 0001 by the patch "reuse-workers", most of the complexity comes with some refactoring to split apply worker and tablesync worker paths. [1]
> If you mean the whole patch set, then I believe it's because reusing replication slots also requires having a proper snapshot each time the worker moves to a new table. [2]
>

Yes, I was mostly referring to the same as point 1 below about patch
0001. I guess I just found the concept of mixing A) launching TSW (via
apply worker) with B) reassigning TSW to another relation (by the TSW
battling with its peers) to be a bit difficult to understand. I
thought most of the refactoring seemed to arise from choosing to do it
that way.

>>
>>
>> 1.
>> IIUC with patch 0001, each/every tablesync worker (a.k.a. TSW) when it
>> finishes dealing with one table then goes looking to find if there is
>> some relation that it can process next. So now every TSW has a loop
>> where it will fight with every other available TSW over who will get
>> to process the next relation.
>>
>> Somehow this seems all backwards to me. Isn't it strange for the TSW
>> to be the one deciding what relation it would deal with next?
>>
>> IMO it seems more natural to simply return the finished TSW to some
>> kind of "pool" of available workers and the main Apply process can
>> just grab a TSW from that pool instead of launching a brand new one in
>> the existing function process_syncing_tables_for_apply(). Or, maybe
>> those "available" workers can be returned to a pool maintained within
>> the launcher.c code, which logicalrep_worker_launch() can draw from
>> instead of launching a whole new process?
>>
>> (I need to read the earlier posts to see if these options were already
>> discussed and rejected)
>
>
> I think ([3]) relying on a single apply worker for the assignment of several tablesync workers might bring some overhead, it's possible that some tablesync workers wait in idle until the apply worker assigns them something. OTOH yes, the current approach makes tablesync workers race for a new table to sync.

Yes, it might be slower than the 'patched' code because "available"
workers might be momentarily idle while they wait to be re-assigned to
the next relation. We would need to try it to find out.

> TBF both ways might be worth discussing/investigating more, before deciding which way to go.

+1. I think it would be nice to see POC of both ways for benchmark
comparison because IMO performance is not the only consideration --
unless there is an obvious winner, then they need to be judged also by
the complexity of the logic, the amount of code that needed to be
refactored, etc.

>
>>
>> 2.
>> AFAIK the thing that identifies a  tablesync worker is the fact that
>> only TSW will have a 'relid'.
>>
>> But it feels very awkward to me to have a TSW marked as "available"
>> and yet that LogicalRepWorker must still have some OLD relid field
>> value lurking (otherwise it will forget that it is a "tablesync"
>> worker!).
>>
>> IMO perhaps it is time now to introduce some enum 'type' to the
>> LogicalRepWorker. Then an "in_use" type=TSW would have a valid 'relid'
>> whereas an "available" type=TSW would have relid == InvalidOid.
>
>
> Hmm, relid will be immediately updated when the worker moves to a new table. And the time between finishing sync of a table and finding a new table to sync should be minimal. I'm not sure how having an old relid for such a small amount of time can do any harm.

There is no "harm", but it just didn't feel right to make the
LogicalRepWorker to transition through some meaningless state
("available" for re-use but still assigned some relid), just because
it was easy to do it that way. I think it is more natural for the
'relid' to be valid only when it is valid for the worker and to be
InvalidOid when it is not valid. --- Maybe this gripe would become
more apparent if the implementation use the "free-list" idea because
then you would have a lot of bogus relids assigned to the workers of
that list for longer than just a moment.

>
>>
>> 3.
>> Maybe I am mistaken, but it seems the benchmark results posted are
>> only using quite a small/default values for
>> "max_sync_workers_per_subscription", so I wondered how those results
>> are affected by increasing that GUC. I think having only very few
>> workers would cause more sequential processing, so conveniently the
>> effect of the patch avoiding re-launch might be seen in the best
>> possible light. OTOH, using more TSW in the first place might reduce
>> the overall tablesync time because the subscriber can do more work in
>> parallel.
>>
>>
>>
>> So I'm not quite sure what the goal is here. E.g. if the user doesn't
>>
>> care much about how long tablesync phase takes then there is maybe no
>> need for this patch at all. OTOH, I thought if a user does care about
>> the subscription startup time, won't those users be opting for a much
>> larger "max_sync_workers_per_subscription" in the first place?
>> Therefore shouldn't the benchmarking be using a larger number too?
>
>
> Regardless of how many tablesync workers there are, reusing workers will speed things up if a worker has a chance to sync more than one table. Increasing the number of tablesync workers, of course, improves the tablesync performance. But if it doesn't make 100% parallel ( meaning that # of sync workers != # of tables to sync), then reusing workers can bring an additional improvement.
>
> Here are some benchmarks similar to earlier, but with 100 tables and different number of workers:
>
> +--------+-------------+-------------+-------------+------------+
> |        | 2 workers   | 4 workers   | 6 workers   | 8 workers  |
> +--------+-------------+-------------+-------------+------------+
> | master | 2579.154 ms | 1383.153 ms | 1001.559 ms | 911.758 ms |
> +--------+-------------+-------------+-------------+------------+
> | patch  | 1724.230 ms | 853.894 ms  | 601.176 ms  | 496.395 ms |
> +--------+-------------+-------------+-------------+------------+
>
> So yes, increasing the number of workers makes it faster. But reusing workers can still improve more.
>

Thanks for the benchmark results! There is no denying they seem pretty
good numbers.

But it is difficult to get an overall picture of the behaviour. Mostly
when benchmarks were posted you hold one variable fixed and show only
one other varying. It always leaves me wondering -- what about not
empty tables, or what about different numbers of tables etc. Is it
possible to make some script to gather a bigger set of results so we
can see everything at once? Perhaps then it will become clear there is
some "sweet spot" where the patch is really good but beyond that it
degrades (actually, who knows what it might show).

For example:

=== empty tables

workers:2     workers:4     workers:8     workers:16
tables:10     tables:10     tables:10     tables:10
data:0        data:0        data:0        data:0
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:100    tables:100    tables:100    tables:100
data:0        data:0        data:0        data:0
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:1000   tables:1000   tables:1000   tables:1000
data:0        data:0        data:0        data:0
master/patch  master/patch  master/patch  master/patch

=== 1M data

workers:2     workers:4     workers:8     workers:16
tables:10     tables:10     tables:10     tables:10
data:1M       data:1M       data:1M       data:1M
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:100    tables:100    tables:100    tables:100
data:1M       data:1M       data:1M       data:1M
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:1000   tables:1000   tables:1000   tables:1000
data:1M       data:1M       data:1M       data:1M
master/patch  master/patch  master/patch  master/patch

=== 10M data

workers:2     workers:4     workers:8     workers:16
tables:10     tables:10     tables:10     tables:10
data:10M      data:10M      data:10M      data:10M
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:100    tables:100    tables:100    tables:100
data:10M      data:10M      data:10M      data:10M
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:1000   tables:1000   tables:1000   tables:1000
data:10M      data:10M      data:10M      data:10M
master/patch  master/patch  master/patch  master/patch

== 100M data

workers:2     workers:4     workers:8     workers:16
tables:10     tables:10     tables:10     tables:10
data:100M     data:100M     data:100M     data:100M
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:100    tables:100    tables:100    tables:100
data:100M     data:100M     data:100M     data:100M
master/patch  master/patch  master/patch  master/patch

workers:2     workers:4     workers:8     workers:16
tables:1000   tables:1000   tables:1000   tables:1000
data:100M     data:100M     data:100M     data:100M
master/patch  master/patch  master/patch  master/patch

------
Kind Regards,
Peter Smith
Fujitsu Australia

  63. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-06-01T10:54:02Z

    Hi,

I rebased the patch and addressed the following reviews.

Peter Smith <smithpb2250@gmail.com>, 24 May 2023 Çar, 05:59 tarihinde
şunu yazdı:
> Here are a few other random things noticed while looking at patch 0001:
>
> 1. Commit message
>
> 1a. typo /sequantially/sequentially/
>
> 1b. Saying "killed" and "killing" seemed a bit extreme and implies
> somebody else is killing the process. But I think mostly tablesync is
> just ending by a normal proc exit, so maybe reword this a bit.
>

Fixed the type and reworded a bit.

>
> 2. It seemed odd that some -- clearly tablesync specific -- functions
> are in the worker.c instead of in tablesync.c.
>
> 2a. e.g. clean_sync_worker
>
> 2b. e.g. sync_worker_exit
>

Honestly, the distinction between worker.c and tablesync.c is not that
clear to me. Both seem like they're doing some work for tablesync and
apply.
But yes, you're right. Those functions fit better to tablesync.c. Moved them.

>
> 3. process_syncing_tables_for_sync
>
> + /*
> + * Sync worker is cleaned at this point. It's ready to sync next table,
> + * if needed.
> + */
> + SpinLockAcquire(&MyLogicalRepWorker->relmutex);
> + MyLogicalRepWorker->ready_to_reuse = true;
>   SpinLockRelease(&MyLogicalRepWorker->relmutex);
> + }
> +
> + SpinLockRelease(&MyLogicalRepWorker->relmutex);
>
> Isn't there a double release of that mutex happening there?

Fixed.

Thanks,
-- 
Melih Mutlu
Microsoft

  64. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-06-01T11:22:30Z

    Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 26 May 2023 Cum, 10:30 tarihinde
şunu yazdı:
>
> On Thu, May 25, 2023 at 6:59 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> Yes, I was mostly referring to the same as point 1 below about patch
> 0001. I guess I just found the concept of mixing A) launching TSW (via
> apply worker) with B) reassigning TSW to another relation (by the TSW
> battling with its peers) to be a bit difficult to understand. I
> thought most of the refactoring seemed to arise from choosing to do it
> that way.

No, the refactoring is not related to the way of assigning a new
table. In fact, the patch did not include such refactoring a couple
versions earlier [1] and was still assigning tables the same way. It
was suggested here [2]. Then, I made the patch 0001 which includes
some refactoring and only reuses the worker and nothing else. Also I
find it more understandable this way, maybe it's a bit subjective.

I feel that logical replication related files are getting more and
more complex and hard to understand with each change. IMHO, even
without reusing anything, those need some refactoring anyway. But for
this patch, refactoring some places made it simpler to reuse workers
and/or replication slots, regardless of how tables are assigned to
TSW's.

> +1. I think it would be nice to see POC of both ways for benchmark
> comparison because IMO performance is not the only consideration --
> unless there is an obvious winner, then they need to be judged also by
> the complexity of the logic, the amount of code that needed to be
> refactored, etc.

Will try to do that. But, like I mentioned above, I don't think that
such a change would reduce the complexity or number of lines changed.

> But it is difficult to get an overall picture of the behaviour. Mostly
> when benchmarks were posted you hold one variable fixed and show only
> one other varying. It always leaves me wondering -- what about not
> empty tables, or what about different numbers of tables etc. Is it
> possible to make some script to gather a bigger set of results so we
> can see everything at once? Perhaps then it will become clear there is
> some "sweet spot" where the patch is really good but beyond that it
> degrades (actually, who knows what it might show).

I actually shared the benchmarks with different numbers of tables and
sizes. But those were all with 2 workers. I guess you want a similar
benchmark with different numbers of workers.
Will work on this and share soon.



[1] https://www.postgresql.org/message-id/CAGPVpCQmEE8BygXr%3DHi2N2t2kOE%3DPJwofn9TX0J9J4crjoXarQ%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAAKRu_YKGyF%2BsvRQqe1th-mG9xLdzneWgh9H1z1DtypBkawkkw%40mail.gmail.com

Thanks,
-- 
Melih Mutlu
Microsoft

  65. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-06-01T12:22:21Z

    On Thu, Jun 1, 2023 at 7:22 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Peter,
>
> Peter Smith <smithpb2250@gmail.com>, 26 May 2023 Cum, 10:30 tarihinde
> şunu yazdı:
> >
> > On Thu, May 25, 2023 at 6:59 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> > Yes, I was mostly referring to the same as point 1 below about patch
> > 0001. I guess I just found the concept of mixing A) launching TSW (via
> > apply worker) with B) reassigning TSW to another relation (by the TSW
> > battling with its peers) to be a bit difficult to understand. I
> > thought most of the refactoring seemed to arise from choosing to do it
> > that way.
>
> No, the refactoring is not related to the way of assigning a new
> table. In fact, the patch did not include such refactoring a couple
> versions earlier [1] and was still assigning tables the same way. It
> was suggested here [2]. Then, I made the patch 0001 which includes
> some refactoring and only reuses the worker and nothing else. Also I
> find it more understandable this way, maybe it's a bit subjective.
>
> I feel that logical replication related files are getting more and
> more complex and hard to understand with each change. IMHO, even
> without reusing anything, those need some refactoring anyway. But for
> this patch, refactoring some places made it simpler to reuse workers
> and/or replication slots, regardless of how tables are assigned to
> TSW's.

If refactoring is wanted anyway (regardless of the chosen "reuse"
logic), then will it be better to split off a separate 0001 patch just
to get that part out of the way first?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  66. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2023-06-05T11:06:45Z

    On Thu, Jun 1, 2023 6:54 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> 
> Hi,
> 
> I rebased the patch and addressed the following reviews.
> 

Thanks for updating the patch. Here are some comments on 0001 patch.

1.
-	ereport(LOG,
-			(errmsg("logical replication table synchronization worker for subscription \"%s\", table \"%s\" has finished",
-					MySubscription->name,
-					get_rel_name(MyLogicalRepWorker->relid))));

Could we move this to somewhere else instead of removing it?

2.
+	if (!OidIsValid(originid))
+		originid = replorigin_create(originname);
+	replorigin_session_setup(originid, 0);
+	replorigin_session_origin = originid;
+	*origin_startpos = replorigin_session_get_progress(false);
+	CommitTransactionCommand();
+
+	/* Is the use of a password mandatory? */
+	must_use_password = MySubscription->passwordrequired &&
+		!superuser_arg(MySubscription->owner);
+	LogRepWorkerWalRcvConn = walrcv_connect(MySubscription->conninfo, true,
+											must_use_password,
+											MySubscription->name, &err);

It seems that there is a problem when refactoring.
See commit e7e7da2f8d5.

3.
+	/* Set this to false for safety, in case we're already reusing the worker */
+	MyLogicalRepWorker->ready_to_reuse = false;
+

I am not sure do we need to lock when setting it.

4.
+	/*
+	 * Allocate the origin name in long-lived context for error context
+	 * message.
+	 */
+	StartTransactionCommand();
+	ReplicationOriginNameForLogicalRep(MySubscription->oid,
+									   MyLogicalRepWorker->relid,
+									   originname,
+									   originname_size);
+	CommitTransactionCommand();

Do we need the call to StartTransactionCommand() and CommitTransactionCommand()
here? Besides, the comment here is the same as the comment atop
set_apply_error_context_origin(), do we need it?

5.
I saw a segmentation fault when debugging.

It happened when calling sync_worker_exit() called (see the code below in
LogicalRepSyncTableStart()). In the case that this is not the first table the
worker synchronizes, clean_sync_worker() has been called before (in
TablesyncWorkerMain()), and LogRepWorkerWalRcvConn has been set to NULL. Then, a
segmentation fault happened because LogRepWorkerWalRcvConn is a null pointer.

	switch (relstate)
	{
		case SUBREL_STATE_SYNCDONE:
		case SUBREL_STATE_READY:
		case SUBREL_STATE_UNKNOWN:
			sync_worker_exit();	/* doesn't return */
	}

Here is the backtrace.

#0  0x00007fc8a8ce4c95 in libpqrcv_disconnect (conn=0x0) at libpqwalreceiver.c:757
#1  0x000000000092b8c0 in clean_sync_worker () at tablesync.c:150
#2  0x000000000092b8ed in sync_worker_exit () at tablesync.c:164
#3  0x000000000092d8f6 in LogicalRepSyncTableStart (origin_startpos=0x7ffd50f30f08) at tablesync.c:1293
#4  0x0000000000934f76 in start_table_sync (origin_startpos=0x7ffd50f30f08, myslotname=0x7ffd50f30e80) at worker.c:4457
#5  0x000000000093513b in run_tablesync_worker (options=0x7ffd50f30ec0, slotname=0x0, originname=0x7ffd50f30f10 "pg_16394_16395",
    originname_size=64, origin_startpos=0x7ffd50f30f08) at worker.c:4532
#6  0x0000000000935a3a in TablesyncWorkerMain (main_arg=1) at worker.c:4853
#7  0x00000000008e97f6 in StartBackgroundWorker () at bgworker.c:864
#8  0x00000000008f350b in do_start_bgworker (rw=0x12fc1a0) at postmaster.c:5762
#9  0x00000000008f38b7 in maybe_start_bgworkers () at postmaster.c:5986
#10 0x00000000008f2975 in process_pm_pmsignal () at postmaster.c:5149
#11 0x00000000008ee98a in ServerLoop () at postmaster.c:1770
#12 0x00000000008ee3bb in PostmasterMain (argc=3, argv=0x12c4af0) at postmaster.c:1463
#13 0x00000000007b6d3a in main (argc=3, argv=0x12c4af0) at main.c:198


The steps to reproduce: 
Worker1, in TablesyncWorkerMain(), the relstate of new table to sync (obtained
by GetSubscriptionRelations) is SUBREL_STATE_INIT, and in the foreach loop,
before the following Check (it needs a breakpoint before locking),

			LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE);
			if (rstate->state != SUBREL_STATE_SYNCDONE &&
				!logicalrep_worker_find(MySubscription->oid, rstate->relid, false))
			{
				/* Update worker state for the next table */
				MyLogicalRepWorker->relid = rstate->relid;
				MyLogicalRepWorker->relstate = rstate->state;
				MyLogicalRepWorker->relstate_lsn = rstate->lsn;
				LWLockRelease(LogicalRepWorkerLock);
				break;
			}
			LWLockRelease(LogicalRepWorkerLock);

let this table to be synchronized by another table sync worker (Worker2), and
Worker2 has finished before logicalrep_worker_find was called(). Then Worker1
tried to sync a table whose state is SUBREL_STATE_READY and the segmentation
fault happened.

Regards,
Shi Yu

  67. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-06-13T10:05:56Z

    Dear Melih,

Thank you for making the patch!
I'm also interested in the patchset. Here are the comments for 0001.

Some codes are not suit for our coding conventions, but followings do not contain them
because patches seems in the early statge.
Moreover, 0003 needs rebase.

01. general

Why do tablesync workers have to disconnect from publisher for every iterations?
I think connection initiation overhead cannot be negligible in the postgres's basic
architecture. I have not checked yet, but could we add a new replication message
like STOP_STREAMING or CLEANUP? Or, engineerings for it is quite larger than the benefit?

02. logicalrep_worker_launch()

```
-       else
+       else if (!OidIsValid(relid))
                snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
+       else
+               snprintf(bgw.bgw_function_name, BGW_MAXLEN, "TablesyncWorkerMain");
```

You changed the entry point of tablesync workers, but bgw_type is still the same.
Do you have any decisions about it? 

03. process_syncing_tables_for_sync()

```
+               /*
+                * Sync worker is cleaned at this point. It's ready to sync next table,
+                * if needed.
+                */
+               SpinLockAcquire(&MyLogicalRepWorker->relmutex);
+               MyLogicalRepWorker->ready_to_reuse = true;
+               SpinLockRelease(&MyLogicalRepWorker->relmutex);
```

Maybe acquiring the lock for modifying ready_to_reuse is not needed because all
the sync workers check only the own attribute. Moreover, other processes do not read.

04. sync_worker_exit()

```
+/*
+ * Exit routine for synchronization worker.
+ */
+void
+pg_attribute_noreturn()
+sync_worker_exit(void)
```

I think we do not have to rename the function from finish_sync_worker().

05. LogicalRepApplyLoop()

```
+                       if (MyLogicalRepWorker->ready_to_reuse)
+                       {
+                               endofstream = true;
+                       }
```

We should add comments here to clarify the reason.

06. stream_build_options()

I think we can set twophase attribute here.

07. TablesyncWorkerMain()

```
+       ListCell   *lc;
```

This variable should be declared inside the loop.

08. TablesyncWorkerMain()

```
+               /*
+                * If a relation with INIT state is assigned, clean up the worker for
+                * the next iteration.
+                *
+                * If there is no more work left for this worker, break the loop to
+                * exit.
+                */
+               if ( MyLogicalRepWorker->relstate == SUBREL_STATE_INIT)
+                       clean_sync_worker();
```

The sync worker sends a signal to its leader per the iteration, but it may be too
often. Maybe it is added for changing the rstate to READY, however, it is OK to
change it when the next change have come because should_apply_changes_for_rel()
returns true even if rel->state == SUBREL_STATE_SYNCDONE. I think the notification
should be done only at the end of sync workers. How do you think? 

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  68. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-06-14T05:45:11Z

    Here are some review comments for the patch v2-0001.

======
Commit message

1. General
Better to use consistent terms in this message. Either "relations" or
"tables" -- not a mixture of both.

~~~

2.
Before this commit, tablesync workers were capable of syncing only one
relation. For each table, a new sync worker was launched and the worker
would exit when the worker is done with the current table.

~

SUGGESTION (2nd sentence)
For each table, a new sync worker was launched and that worker would
exit when done processing the table.

~~~

3.
Now, tablesync workers are not only limited with one relation and can
move to another relation in the same subscription. This reduces the
overhead of launching a new background worker and exiting from that
worker for each relation.

~

SUGGESTION (1st sentence)
Now, tablesync workers are not limited to processing only one
relation. When done, they can move to processing another relation in
the same subscription.

~~~

4.
A new tablesync worker gets launched only if the number of tablesync
workers for the subscription does not exceed
max_sync_workers_per_subscription. If there is a table needs to be synced,
a tablesync worker picks that up and syncs it.The worker continues to
picking new tables to sync until there is no table left for synchronization
in the subscription.

~

This seems to be missing the point that only "available" workers go
looking for more tables to process. Maybe reword something like below:

SUGGESTION
If there is a table that needs to be synced, an "available" tablesync
worker picks up that table and syncs it. Each tablesync worker
continues to pick new tables to sync until there are no tables left
requiring synchronization. If there was no "available" worker to
process the table, then a new tablesync worker will be launched,
provided the number of tablesync workers for the subscription does not
exceed max_sync_workers_per_subscription.

======
src/backend/replication/logical/launcher.c

5. logicalrep_worker_launch

@@ -460,8 +461,10 @@ retry:

  if (is_parallel_apply_worker)
  snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
- else
+ else if (!OidIsValid(relid))
  snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
+ else
+ snprintf(bgw.bgw_function_name, BGW_MAXLEN, "TablesyncWorkerMain");

  if (OidIsValid(relid))
  snprintf(bgw.bgw_name, BGW_MAXLEN,

~

5a.
I felt at least these conditions can be rearranged, so you can use
OidIsValid(relid) instead of !OidIsValid(relid).

~

5b.
Probably it can all be simplified, if you are happy to do it in one line:

snprintf(bgw.bgw_function_name, BGW_MAXLEN,
    OidIsValid(relid) ? "TablesyncWorkerMain" :
    is_parallel_apply_worker ? "ParallelApplyWorkerMain" :
"ApplyWorkerMain");

======
src/backend/replication/logical/tablesync.c

6. finish_sync_worker

This function is removed/renamed but there are still commenting in
this file referring to 'finish-sync_worker'

~~~

7. clean_sync_worker

I agree with comment from Shi-san. There should still be logging
somewhere that say this tablesync worker has completed the processing
of the current table.

~~~

8. sync_worker_exit

There is inconsistent function naming for clean_sync_worker versus
sync_worker_exit.

How about: clean_sync_worker/exit_sync_worker?
Or: sync_worker_clean/sync_worker_exit?

~~~

9. process_syncing_tables_for_sync

@@ -378,7 +387,13 @@ process_syncing_tables_for_sync(XLogRecPtr current_lsn)
  */
  replorigin_drop_by_name(originname, true, false);

- finish_sync_worker();
+ /*
+ * Sync worker is cleaned at this point. It's ready to sync next table,
+ * if needed.
+ */
+ SpinLockAcquire(&MyLogicalRepWorker->relmutex);
+ MyLogicalRepWorker->ready_to_reuse = true;
+ SpinLockRelease(&MyLogicalRepWorker->relmutex);

9a.
I did not quite follow the logic. It says "Sync worker is cleaned at
this point", but who is doing that? -- more details are needed. But,
why not just call clean_sync_worker() right here like it use to call
finish_sync_worker?

~

9b.
Shouldn't this "ready_to_use" flag be assigned within the
clean_sync_worker() function, since that is the function making is
clean for next re-use. The function comment even says so: "Prepares
the synchronization worker for reuse or exit."

======
src/backend/replication/logical/worker.c

10. General -- run_tablesync_worker, TablesyncWorkerMain

IMO these functions would be more appropriately reside in the
tablesync.c instead of the (common) worker.c. Was there some reason
why they cannot be put there?

~~~

11. LogicalRepApplyLoop

+ /*
+ * apply_dispatch() may have gone into apply_handle_commit()
+ * which can go into process_syncing_tables_for_sync early.
+ * Before we were able to reuse tablesync workers, that
+ * process_syncing_tables_for_sync call would exit the worker
+ * instead of preparing for reuse. Now that tablesync workers
+ * can be reused and process_syncing_tables_for_sync is not
+ * responsible for exiting. We need to take care of memory
+ * contexts here before moving to sync the nex table or exit.
+ */

11a.
IMO it does not seem good to explain the reason by describing how the
logic USED to work, with code that is removed (e.g. "Before we
were..."). It's better to describe why this is needed here based on
all the CURRENT code logic.

~

11b.
/nex table/next table/

~

12.
+ if (MyLogicalRepWorker->ready_to_reuse)
+ {
+ endofstream = true;
+ }

Unnecessary parentheses.

~

13.
+ /*
+ * If it's still not ready to reuse, this is probably an apply worker.
+ * End streaming before exiting.
+ */
+ if (!MyLogicalRepWorker->ready_to_reuse)
+ {
+ /* All done */
+ walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);
+ }

How can we not be 100% sure of the kind of worker we are dealing with?
E.g. "probably" ??

Should this code be using macros like am_tablesync_worker() to have
some certainty what it is dealing with here?

~~~

14. stream_build_options

+ /* stream_build_options
+  * Build logical replication streaming options.
+  *
+  * This function sets streaming options including replication slot name
+  * and origin start position. Workers need these options for logical
replication.
+  */
+static void
+stream_build_options(WalRcvStreamOptions *options, char *slotname,
XLogRecPtr *origin_startpos)

The function name seem a bit strange -- it's not really "building"
anything. How about something like SetStreamOptions, or
set_stream_options.

~~~

15. run_tablesync_worker

+static void
+run_tablesync_worker(WalRcvStreamOptions *options,
+ char *slotname,
+ char *originname,
+ int originname_size,
+ XLogRecPtr *origin_startpos)
+{
+ /* Set this to false for safety, in case we're already reusing the worker */
+ MyLogicalRepWorker->ready_to_reuse = false;

Maybe reword the comment so it does not say set 'this' to false.

~

16.
+ /* Start applying changes to catcup. */
+ start_apply(*origin_startpos);

typo: catcup

~~~

17. run_apply_worker

+static void
+run_apply_worker(WalRcvStreamOptions *options,
+ char *slotname,
+ char *originname,
+ int originname_size,
+ XLogRecPtr *origin_startpos)
+{
+ /* This is the leader apply worker */
+ RepOriginId originid;
+ TimeLineID startpointTLI;
+ char    *err;
+ bool must_use_password;


The comment above the variable declarations seems redundant/misplaced.

~~

18. InitializeLogRepWorker

  if (am_tablesync_worker())
  ereport(LOG,
- (errmsg("logical replication table synchronization worker for
subscription \"%s\", table \"%s\" has started",
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\", relation \"%s\" with relid %u has started",
  MySubscription->name,
- get_rel_name(MyLogicalRepWorker->relid))));
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));
  else


I felt this code could be using get_worker_name() function like the
"else" does instead of the hardwired: "logical replication table
synchronization worker" string

~~~

19. TablesyncWorkerMain

+TablesyncWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos = InvalidXLogRecPtr;
+ char    *myslotname = NULL;
+ WalRcvStreamOptions options;
+ List    *rstates;
+ SubscriptionRelState *rstate;
+ ListCell   *lc;

- /* Setup replication origin tracking. */
- StartTransactionCommand();
- ReplicationOriginNameForLogicalRep(MySubscription->oid, InvalidOid,
-    originname, sizeof(originname));
- originid = replorigin_by_name(originname, true);
- if (!OidIsValid(originid))
- originid = replorigin_create(originname);
- replorigin_session_setup(originid, 0);
- replorigin_session_origin = originid;
- origin_startpos = replorigin_session_get_progress(false);
-
- /* Is the use of a password mandatory? */
- must_use_password = MySubscription->passwordrequired &&
- !superuser_arg(MySubscription->owner);
-
- /* Note that the superuser_arg call can access the DB */
- CommitTransactionCommand();
+ elog(LOG, "logical replication table synchronization worker has started");

Would it be better if that elog was using the common function get_worker_name()?

~~~

20.
+ if (MyLogicalRepWorker->ready_to_reuse)
+ {
+ /* This transaction will be committed by clean_sync_worker. */
+ StartTransactionCommand();

The indentation is broken.

~~~

21.
+ * Check if any table whose relation state is still INIT. If a table
+ * in INIT state is found, the worker will not be finished, it will be
+ * reused instead.
  */

First sentence is not meaningful. Should it say: "Check if there is
any table whose relation state is still INIT." ??

~~~

22.
+ /*
+ * Pick the table for the next run if it is not already picked up
+ * by another worker.
+ *
+ * Take exclusive lock to prevent any other sync worker from picking
+ * the same table.
+ */
+ LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE);
+ if (rstate->state != SUBREL_STATE_SYNCDONE &&
+ !logicalrep_worker_find(MySubscription->oid, rstate->relid, false))
+ {
+ /* Update worker state for the next table */
+ MyLogicalRepWorker->relid = rstate->relid;
+ MyLogicalRepWorker->relstate = rstate->state;
+ MyLogicalRepWorker->relstate_lsn = rstate->lsn;
+ LWLockRelease(LogicalRepWorkerLock);
+ break;
+ }
+ LWLockRelease(LogicalRepWorkerLock);
  }
+
+ /*
+ * If a relation with INIT state is assigned, clean up the worker for
+ * the next iteration.
+ *
+ * If there is no more work left for this worker, break the loop to
+ * exit.
+ */
+ if ( MyLogicalRepWorker->relstate == SUBREL_STATE_INIT)
+ clean_sync_worker();
  else
- {
- walrcv_startstreaming(LogRepWorkerWalRcvConn, &options);
- }
+ break;

I was unsure about this logic, but shouldn't the
MyLogicalRepWorker->relstate be assigned a default value prior to all
these loops, so that there can be no chance for it to be
SUBREL_STATE_INIT by accident.

~

23.
+ /* If not exited yet, then the worker will sync another table. */
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" has moved to sync table \"%s\" with relid %u.",
+ MySubscription->name, get_rel_name(MyLogicalRepWorker->relid),
MyLogicalRepWorker->relid)));
+ CommitTransactionCommand();

23a
This code seems strangely structured. Why is the "not exited yet" part
not within the preceding "if" block where the clean_sync_worker was
done?

~~~

23b.
Wont it be better for that errmsg to use the common function
get_worker_name() instead of having the hardcoded string?


======
src/include/replication/worker_internal.h

24.
+ /*
+ * Used to indicate whether sync worker is ready for being reused
+ * to sync another relation.
+ */
+ bool ready_to_reuse;
+

IIUC this field has no meaning except for a tablesync worker, but the
fieldname give no indication of that at all.

To make this more obvious it might be better to put this with the
other tablesync fields:

/* Used for initial table synchronization. */
Oid relid;
char relstate;
XLogRecPtr relstate_lsn;
slock_t relmutex;
And maybe rename it according to that convention relXXX -- e.g.
'relworker_available' or something

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  69. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-06-23T13:32:47Z

    Hi hackers,

You can find the updated patchset attached.
I worked to address the reviews and made some additional changes.

Let me first explain the new patchset.
0001: Refactors the logical replication code, mostly worker.c and
tablesync.c. Although this patch makes it easier to reuse workers, I
believe that it's useful even by itself without other patches. It does
not improve performance or anything but aims to increase readability
and such.
0002: This is only to reuse worker processes, everything else stays
the same (replication slots/origins etc.).
0003: Adds a new command for streaming replication protocol to create
a snapshot by an existing replication slot.
0004: Reuses replication slots/origins together with workers.

Even only 0001 and 0002 are enough to improve table sync performance
at the rates previously shared on this thread. This also means that
currently 0004 (reusing replication slots/origins) does not improve as
much as I would expect, even though it does not harm either.
I just wanted to share what I did so far, while I'm continuing to
investigate it more to see what I'm missing in patch 0004.

Thanks,
-- 
Melih Mutlu
Microsoft

  70. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-06-23T13:39:57Z

    Hi,

Thanks for your reviews.

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 13 Haz 2023 Sal,
13:06 tarihinde şunu yazdı:
> 01. general
>
> Why do tablesync workers have to disconnect from publisher for every iterations?
> I think connection initiation overhead cannot be negligible in the postgres's basic
> architecture. I have not checked yet, but could we add a new replication message
> like STOP_STREAMING or CLEANUP? Or, engineerings for it is quite larger than the benefit?

This actually makes sense. I quickly try to do that without adding any
new replication message. As you would expect, it did not work.
I don't really know what's needed to make a connection to last for
more than one iteration. Need to look into this. Happy to hear any
suggestions and thoughts.

> The sync worker sends a signal to its leader per the iteration, but it may be too
> often. Maybe it is added for changing the rstate to READY, however, it is OK to
> change it when the next change have come because should_apply_changes_for_rel()
> returns true even if rel->state == SUBREL_STATE_SYNCDONE. I think the notification
> should be done only at the end of sync workers. How do you think?

I tried to move the logicalrep_worker_wakeup call from
clean_sync_worker (end of an iteration) to finish_sync_worker (end of
sync worker). I made table sync much slower for some reason, then I
reverted that change. Maybe I should look a bit more into the reason
why that happened some time.

Thanks,
-- 
Melih Mutlu
Microsoft

  71. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-06-23T13:50:24Z

    Hi Peter,

Thanks for your reviews. I tried to apply most of them. I just have
some comments below for some of them.

Peter Smith <smithpb2250@gmail.com>, 14 Haz 2023 Çar, 08:45 tarihinde
şunu yazdı:
>
> 9. process_syncing_tables_for_sync
>
> @@ -378,7 +387,13 @@ process_syncing_tables_for_sync(XLogRecPtr current_lsn)
>   */
>   replorigin_drop_by_name(originname, true, false);
>
> - finish_sync_worker();
> + /*
> + * Sync worker is cleaned at this point. It's ready to sync next table,
> + * if needed.
> + */
> + SpinLockAcquire(&MyLogicalRepWorker->relmutex);
> + MyLogicalRepWorker->ready_to_reuse = true;
> + SpinLockRelease(&MyLogicalRepWorker->relmutex);
>
> 9a.
> I did not quite follow the logic. It says "Sync worker is cleaned at
> this point", but who is doing that? -- more details are needed. But,
> why not just call clean_sync_worker() right here like it use to call
> finish_sync_worker?

I agree that these explanations at places where the worker decides to
not continue with the current table were confusing. Even the name of
ready_to_reuse was misleading. I renamed it and tried to improve
comments in such places.
Can you please check if those make more sense now?


> ======
> src/backend/replication/logical/worker.c
>
> 10. General -- run_tablesync_worker, TablesyncWorkerMain
>
> IMO these functions would be more appropriately reside in the
> tablesync.c instead of the (common) worker.c. Was there some reason
> why they cannot be put there?

I'm not really against moving those functions to tablesync.c. But
what's not clear to me is worker.c. Is it the places to put common
functions for all log. rep. workers? Then, what about apply worker?
Should we consider a separate file for apply worker too?

Thanks,
-- 
Melih Mutlu
Microsoft

  72. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-06-26T02:21:14Z

    On Fri, Jun 23, 2023 at 11:50 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> > src/backend/replication/logical/worker.c
> >
> > 10. General -- run_tablesync_worker, TablesyncWorkerMain
> >
> > IMO these functions would be more appropriately reside in the
> > tablesync.c instead of the (common) worker.c. Was there some reason
> > why they cannot be put there?
>
> I'm not really against moving those functions to tablesync.c. But
> what's not clear to me is worker.c. Is it the places to put common
> functions for all log. rep. workers? Then, what about apply worker?
> Should we consider a separate file for apply worker too?

IIUC
- tablesync.c = for tablesync only
- applyparallelworker = for parallel apply worker only
- worker.c = for both normal apply worker, plus "common" worker code

Regarding making another file (e.g. applyworker.c). It sounds
sensible, but I guess you would need to first demonstrate the end
result will be much cleaner to get support for such a big refactor.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  73. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-06-27T07:42:49Z

    Dear Melih,

Thank you for updating the patch! I have not reviewed yet, but I wanted
to reply your comments.

> This actually makes sense. I quickly try to do that without adding any
> new replication message. As you would expect, it did not work.
> I don't really know what's needed to make a connection to last for
> more than one iteration. Need to look into this. Happy to hear any
> suggestions and thoughts.

I have analyzed how we handle this. Please see attached the patch (0003) which
allows reusing connection. The patchset passed tests on my CI.
To make cfbot happy I reassigned the patch number.

In this patch, the tablesync worker does not call clean_sync_worker() at the end
of iterations, and the establishment of the connection is done only once.
The creation of memory context is also suppressed.

Regarding the walsender, streamingDone{Sending|Receiving} is now initialized
before executing StartLogicalReplication(). These flags have been used to decide
when the process exits copy mode. The default value is false, and they are set
to true when the copy mode is finished.
I think there was no use-case that the same walsender executes START_REPLICATION
replication twice so there were no codes for restoring flags. Please tell me if any other
reasons.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  74. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-06-27T10:50:56Z

    On Fri, Jun 23, 2023 at 7:03 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> You can find the updated patchset attached.
> I worked to address the reviews and made some additional changes.
>
> Let me first explain the new patchset.
> 0001: Refactors the logical replication code, mostly worker.c and
> tablesync.c. Although this patch makes it easier to reuse workers, I
> believe that it's useful even by itself without other patches. It does
> not improve performance or anything but aims to increase readability
> and such.
> 0002: This is only to reuse worker processes, everything else stays
> the same (replication slots/origins etc.).
> 0003: Adds a new command for streaming replication protocol to create
> a snapshot by an existing replication slot.
> 0004: Reuses replication slots/origins together with workers.
>
> Even only 0001 and 0002 are enough to improve table sync performance
> at the rates previously shared on this thread. This also means that
> currently 0004 (reusing replication slots/origins) does not improve as
> much as I would expect, even though it does not harm either.
> I just wanted to share what I did so far, while I'm continuing to
> investigate it more to see what I'm missing in patch 0004.
>

I think the reason why you don't see the benefit of the 0004 patches
is that it still pays the cost of disconnect/connect and we haven't
saved much on network transfer costs because of the new snapshot you
are creating in patch 0003. Is it possible to avoid disconnect/connect
each time the patch needs to reuse the same tablesync worker? Once, we
do that and save the cost of drop_slot and associated network round
trip, you may see the benefit of 0003 and 0004 patches.

-- 
With Regards,
Amit Kapila.

  75. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-06-27T15:00:39Z

    Dear Melih,

Thanks for updating the patch. Followings are my comments.
Note that some lines exceeds 80 characters and some other lines seem too short.
And comments about coding conventions were skipped.

0001

01. logicalrep_worker_launch()

```
        if (is_parallel_apply_worker)
+       {
                snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ParallelApplyWorkerMain");
-       else
-               snprintf(bgw.bgw_function_name, BGW_MAXLEN, "ApplyWorkerMain");
-
-       if (OidIsValid(relid))
                snprintf(bgw.bgw_name, BGW_MAXLEN,
-                                "logical replication worker for subscription %u sync %u", subid, relid);
-       else if (is_parallel_apply_worker)
+                                "logical replication parallel apply worker for subscription %u", subid);
                snprintf(bgw.bgw_name, BGW_MAXLEN,
                                 "logical replication parallel apply worker for subscription %u", subid);
```

Latter snprintf(bgw.bgw_name...) should be snprintf(bgw.bgw_type, BGW_MAXLEN, "logical replication worker").

02. ApplyWorkerMain

```
        /*
         * Setup callback for syscache so that we know when something changes in
-        * the subscription relation state.
+        * the subscription relation state. Do this outside the loop to avoid
+        * exceeding MAX_SYSCACHE_CALLBACKS
         */
```

I'm not sure this change is really needed. CacheRegisterSyscacheCallback() must
be outside the loop to avoid duplicated register, and it seems trivial.

0002

03. TablesyncWorkerMain()

Regarding the inner loop, the exclusive lock is acquired even if the rstate is
SUBREL_STATE_SYNCDONE. Moreover, palloc() and memcpy() for rstate seemsed not
needed. How about following?

```
        for (;;)
        {
                List       *rstates;
-               SubscriptionRelState *rstate;
                ListCell   *lc;
...
-                       rstate = (SubscriptionRelState *) palloc(sizeof(SubscriptionRelState));
 
                        foreach(lc, rstates)
                        {
-                               memcpy(rstate, lfirst(lc), sizeof(SubscriptionRelState));
+                               SubscriptionRelState *rstate =
+                                                                               (SubscriptionRelState *) lfirst(lc);
+
+                               if (rstate->state == SUBREL_STATE_SYNCDONE)
+                                       continue;

                                /*
-                               * Pick the table for the next run if it is not already picked up
-                               * by another worker.
-                               *
-                               * Take exclusive lock to prevent any other sync worker from picking
-                               * the same table.
-                               */
+                                * Take exclusive lock to prevent any other sync worker from
+                                * picking the same table.
+                                */
                                LWLockAcquire(LogicalRepWorkerLock, LW_EXCLUSIVE);
-                               if (rstate->state != SUBREL_STATE_SYNCDONE &&
-                                       !logicalrep_worker_find(MySubscription->oid, rstate->relid, false))
+
+                               /*
+                                * Pick the table for the next run if it is not already picked up
+                                * by another worker.
+                                */
+                               if (!logicalrep_worker_find(MySubscription->oid,
+                                                                                       rstate->relid, false))
```

04. TablesyncWorkerMain

I think rstates should be pfree'd at the end of the outer loop, but it's OK
if other parts do not.

05. repsponse for for post

>
I tried to move the logicalrep_worker_wakeup call from
clean_sync_worker (end of an iteration) to finish_sync_worker (end of
sync worker). I made table sync much slower for some reason, then I
reverted that change. Maybe I should look a bit more into the reason
why that happened some time.
>

I want to see the testing method to reproduce the same issue, could you please
share it to -hackers?

0003, 0004

I did not checked yet but I could say same as above:
I want to see the testing method to reproduce the same issue.
Could you please share it to -hackers?
My previous post (an approach for reuse connection) may help the performance.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  76. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-06-28T04:57:09Z

    On Tue, Jun 27, 2023 at 1:12 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> > This actually makes sense. I quickly try to do that without adding any
> > new replication message. As you would expect, it did not work.
> > I don't really know what's needed to make a connection to last for
> > more than one iteration. Need to look into this. Happy to hear any
> > suggestions and thoughts.
>

It is not clear to me what exactly you tried here which didn't work.
Can you please explain a bit more?

> I have analyzed how we handle this. Please see attached the patch (0003) which
> allows reusing connection.
>

Why did you change the application name during the connection?

-- 
With Regards,
Amit Kapila.

  77. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-06-28T06:31:54Z

    Dear Amit,

> > > This actually makes sense. I quickly try to do that without adding any
> > > new replication message. As you would expect, it did not work.
> > > I don't really know what's needed to make a connection to last for
> > > more than one iteration. Need to look into this. Happy to hear any
> > > suggestions and thoughts.
> >
> 
> It is not clear to me what exactly you tried here which didn't work.
> Can you please explain a bit more?

Just to confirm, this is not my part. Melih can answer this...

> > I have analyzed how we handle this. Please see attached the patch (0003) which
> > allows reusing connection.
> >
> 
> Why did you change the application name during the connection?

It was because the lifetime of tablesync worker is longer than slots's one and
tablesync worker creates temporary replication slots many times, per the target
relation. The name of each slots has relid, so I thought that it was not suitable.
But in the later patch the tablesync worker tries to reuse the slot during the
synchronization, so in this case the application_name should be same as slotname.

I added comment in 0003, and new file 0006 file to use slot name as application_name
again. Note again that the separation was just for specifying changes, Melih can
include them to one part of files if needed.


Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  78. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-03T04:12:42Z

    On Wed, Jun 28, 2023 at 12:02 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> > > I have analyzed how we handle this. Please see attached the patch (0003) which
> > > allows reusing connection.
> > >
> >
> > Why did you change the application name during the connection?
>
> It was because the lifetime of tablesync worker is longer than slots's one and
> tablesync worker creates temporary replication slots many times, per the target
> relation. The name of each slots has relid, so I thought that it was not suitable.
>

Okay, but let's try to give a unique application name to each
tablesync worker for the purpose of pg_stat_activity and synchronous
replication (as mentioned in existing comments as well). One idea is
to generate a name like pg_<sub_id>_sync_<worker_slot> but feel free
to suggest if you have any better ideas.

> But in the later patch the tablesync worker tries to reuse the slot during the
> synchronization, so in this case the application_name should be same as slotname.
>

Fair enough. I am slightly afraid that if we can't show the benefits
with later patches then we may need to drop them but at this stage I
feel we need to investigate why those are not helping?

-- 
With Regards,
Amit Kapila.

  79. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-03T05:58:48Z

    On Mon, Jul 3, 2023 at 9:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jun 28, 2023 at 12:02 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
>
> > But in the later patch the tablesync worker tries to reuse the slot during the
> > synchronization, so in this case the application_name should be same as slotname.
> >
>
> Fair enough. I am slightly afraid that if we can't show the benefits
> with later patches then we may need to drop them but at this stage I
> feel we need to investigate why those are not helping?
>

On thinking about this, I think the primary benefit we were expecting
by saving network round trips for slot drop/create but now that we
anyway need an extra round trip to establish a snapshot, so such a
benefit was not visible. This is just a theory so we should validate
it. The another idea as discussed before [1] could be to try copying
multiple tables in a single transaction. Now, keeping a transaction
open for a longer time could have side-effects on the publisher node.
So, we probably need to ensure that we don't perform multiple large
syncs and even for smaller tables (and later sequences) perform it
only for some threshold number of tables which we can figure out by
some tests. Also, the other safety-check could be that anytime we need
to perform streaming (sync with apply worker), we won't copy more
tables in same transaction.

Thoughts?

[1] - https://www.postgresql.org/message-id/CAGPVpCRWEVhXa7ovrhuSQofx4to7o22oU9iKtrOgAOtz_%3DY6vg%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  80. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-07-03T11:49:50Z

    On Wed, 28 Jun 2023 at 12:02, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Amit,
>
> > > > This actually makes sense. I quickly try to do that without adding any
> > > > new replication message. As you would expect, it did not work.
> > > > I don't really know what's needed to make a connection to last for
> > > > more than one iteration. Need to look into this. Happy to hear any
> > > > suggestions and thoughts.
> > >
> >
> > It is not clear to me what exactly you tried here which didn't work.
> > Can you please explain a bit more?
>
> Just to confirm, this is not my part. Melih can answer this...
>
> > > I have analyzed how we handle this. Please see attached the patch (0003) which
> > > allows reusing connection.
> > >
> >
> > Why did you change the application name during the connection?
>
> It was because the lifetime of tablesync worker is longer than slots's one and
> tablesync worker creates temporary replication slots many times, per the target
> relation. The name of each slots has relid, so I thought that it was not suitable.
> But in the later patch the tablesync worker tries to reuse the slot during the
> synchronization, so in this case the application_name should be same as slotname.
>
> I added comment in 0003, and new file 0006 file to use slot name as application_name
> again. Note again that the separation was just for specifying changes, Melih can
> include them to one part of files if needed.

Few comments:
1) Should these error messages say "Could not create a snapshot by
replication slot":
+       if (!pubnames_str)
+               ereport(ERROR,
+                               (errcode(ERRCODE_OUT_OF_MEMORY),
 /* likely guess */
+                                errmsg("could not start WAL streaming: %s",
+
pchomp(PQerrorMessage(conn->streamConn)))));
+       pubnames_literal = PQescapeLiteral(conn->streamConn, pubnames_str,
+
    strlen(pubnames_str));
+       if (!pubnames_literal)
+               ereport(ERROR,
+                               (errcode(ERRCODE_OUT_OF_MEMORY),
 /* likely guess */
+                                errmsg("could not start WAL streaming: %s",
+
pchomp(PQerrorMessage(conn->streamConn)))));
+       appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
+       PQfreemem(pubnames_literal);
+       pfree(pubnames_str);

2) These checks are present in CreateReplicationSlot too, can we have
a common function to check these for both CreateReplicationSlot and
CreateReplicationSnapshot:
+       if (!IsTransactionBlock())
+               ereport(ERROR,
+                               (errmsg("%s must be called inside a
transaction",
+
"CREATE_REPLICATION_SNAPSHOT ...")));
+
+       if (XactIsoLevel != XACT_REPEATABLE_READ)
+               ereport(ERROR,
+                               (errmsg("%s must be called in
REPEATABLE READ isolation mode transaction",
+
"CREATE_REPLICATION_SNAPSHOT ...")));
+
+       if (!XactReadOnly)
+               ereport(ERROR,
+                               (errmsg("%s must be called in a read
only transaction",
+
"CREATE_REPLICATION_SNAPSHOT ...")));
+
+       if (FirstSnapshotSet)
+               ereport(ERROR,
+                               (errmsg("%s must be called before any query",
+
"CREATE_REPLICATION_SNAPSHOT ...")));
+
+       if (IsSubTransaction())
+               ereport(ERROR,
+                               (errmsg("%s must not be called in a
subtransaction",
+
"CREATE_REPLICATION_SNAPSHOT ...")));

3) Probably we can add the function header at this point of time:
+/*
+ * TODO
+ */
+static void
+libpqrcv_slot_snapshot(WalReceiverConn *conn,
+                                          char *slotname,
+                                          const WalRcvStreamOptions *options,
+                                          XLogRecPtr *lsn)

4) Either or relation name or relid should be sufficient here, no need
to print both:
                                       StartTransactionCommand();
+                                       ereport(LOG,
+                                                       (errmsg("%s
for subscription \"%s\" has moved to sync table \"%s\" with relid
%u.",
+
 get_worker_name(),
+
 MySubscription->name,
+
 get_rel_name(MyLogicalRepWorker->relid),
+
 MyLogicalRepWorker->relid)));
+                                       CommitTransactionCommand();

5) Why is this check of logicalrep_worker_find is required required,
will it not be sufficient to pick the relations that are in
SUBREL_STATE_INIT state?
+                               /*
+                               * Pick the table for the next run if
it is not already picked up
+                               * by another worker.
+                               *
+                               * Take exclusive lock to prevent any
other sync worker from picking
+                               * the same table.
+                               */
+                               LWLockAcquire(LogicalRepWorkerLock,
LW_EXCLUSIVE);
+                               if (rstate->state != SUBREL_STATE_SYNCDONE &&
+
!logicalrep_worker_find(MySubscription->oid, rstate->relid, false))
+                               {
+                                       /* Update worker state for the
next table */

6) This comment is missed while refactoring:
-       /* Build logical replication streaming options. */
-       options.logical = true;
-       options.startpoint = origin_startpos;
-       options.slotname = myslotname;

7) We could keep twophase and origin as the same order as it was
earlier so that it is easy to review that the existing code is kept as
is in this case:
+       options->proto.logical.publication_names = MySubscription->publications;
+       options->proto.logical.binary = MySubscription->binary;
+       options->proto.logical.twophase = false;
+       options->proto.logical.origin = pstrdup(MySubscription->origin);
+
+       /*
+        * Assign the appropriate option value for streaming option according to
+        * the 'streaming' mode and the publisher's ability to support
that mode.
+        */
+       if (server_version >= 160000 &&

8)  There are few indentation issues, we could run pgindent once:
8.a)
+               /* Sync worker has completed synchronization of the
current table. */
+               MyLogicalRepWorker->is_sync_completed = true;
+
+               ereport(LOG,
+               (errmsg("logical replication table synchronization
worker for subscription \"%s\", relation \"%s\" with relid %u has
finished",
+                               MySubscription->name,
+                               get_rel_name(MyLogicalRepWorker->relid),
+                               MyLogicalRepWorker->relid)));
+               CommitTransactionCommand();

8.b)
+               ereport(DEBUG2,
+                       (errmsg("process_syncing_tables_for_sync:
updated originname: %s, slotname: %s, state: %c for relation \"%u\" in
subscription \"%u\".",
+                                       "NULL", "NULL",
MyLogicalRepWorker->relstate,
+                                       MyLogicalRepWorker->relid,
MyLogicalRepWorker->subid)));
+               CommitTransactionCommand();
+               pgstat_report_stat(false);

Regards,
Vignesh

  81. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-07-04T05:42:48Z

    Dear Amit,

> > > > I have analyzed how we handle this. Please see attached the patch (0003)
> which
> > > > allows reusing connection.
> > > >
> > >
> > > Why did you change the application name during the connection?
> >
> > It was because the lifetime of tablesync worker is longer than slots's one and
> > tablesync worker creates temporary replication slots many times, per the target
> > relation. The name of each slots has relid, so I thought that it was not suitable.
> >
> 
> Okay, but let's try to give a unique application name to each
> tablesync worker for the purpose of pg_stat_activity and synchronous
> replication (as mentioned in existing comments as well). One idea is
> to generate a name like pg_<sub_id>_sync_<worker_slot> but feel free
> to suggest if you have any better ideas.

Good point. The slot id is passed as an argument of TablesyncWorkerMain(),
so I passed it to LogicalRepSyncTableStart(). PSA new set.

> > But in the later patch the tablesync worker tries to reuse the slot during the
> > synchronization, so in this case the application_name should be same as
> slotname.
> >
> 
> Fair enough. I am slightly afraid that if we can't show the benefits
> with later patches then we may need to drop them but at this stage I
> feel we need to investigate why those are not helping?

Agreed. Now I'm planning to do performance testing independently. We can discuss
based on that or Melih's one.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  82. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-04T19:47:34Z

    Hi,

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 27 Haz 2023 Sal,
10:42 tarihinde şunu yazdı:
>
> Dear Melih,
>
> Thank you for updating the patch! I have not reviewed yet, but I wanted
> to reply your comments.
>
> > This actually makes sense. I quickly try to do that without adding any
> > new replication message. As you would expect, it did not work.
> > I don't really know what's needed to make a connection to last for
> > more than one iteration. Need to look into this. Happy to hear any
> > suggestions and thoughts.
>
> I have analyzed how we handle this. Please see attached the patch (0003) which
> allows reusing connection. The patchset passed tests on my CI.
> To make cfbot happy I reassigned the patch number.
>
> In this patch, the tablesync worker does not call clean_sync_worker() at the end
> of iterations, and the establishment of the connection is done only once.
> The creation of memory context is also suppressed.
>
> Regarding the walsender, streamingDone{Sending|Receiving} is now initialized
> before executing StartLogicalReplication(). These flags have been used to decide
> when the process exits copy mode. The default value is false, and they are set
> to true when the copy mode is finished.
> I think there was no use-case that the same walsender executes START_REPLICATION
> replication twice so there were no codes for restoring flags. Please tell me if any other
> reasons.

Thanks for the 0003 patch. But it did not work for me. Can you create
a subscription successfully with patch 0003 applied?
I get the following error: " ERROR:  table copy could not start
transaction on publisher: another command is already in progress".

I think streaming needs to be ended before moving to another table. So
I changed the patch a little bit and also addressed the reviews from
recent emails. Please see the attached patch set.

I'm still keeping the reuse connection patch separate for now to see
what is needed clearly.

Thanks,
Melih

  83. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-04T20:18:14Z

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 4 Tem 2023 Sal,
08:42 tarihinde şunu yazdı:
> > > But in the later patch the tablesync worker tries to reuse the slot during the
> > > synchronization, so in this case the application_name should be same as
> > slotname.
> > >
> >
> > Fair enough. I am slightly afraid that if we can't show the benefits
> > with later patches then we may need to drop them but at this stage I
> > feel we need to investigate why those are not helping?
>
> Agreed. Now I'm planning to do performance testing independently. We can discuss
> based on that or Melih's one.

Here I attached  what I use for performance testing of this patch.

I only benchmarked the patch set with reusing connections very roughly
so far. But seems like it improves quite significantly. For example,
it took 611 ms to sync 100 empty tables, it was 1782 ms without
reusing connections.
First 3 patches from the set actually bring a good amount of
improvement, but not sure about the later patches yet.

Amit Kapila <amit.kapila16@gmail.com>, 3 Tem 2023 Pzt, 08:59 tarihinde
şunu yazdı:
> On thinking about this, I think the primary benefit we were expecting
> by saving network round trips for slot drop/create but now that we
> anyway need an extra round trip to establish a snapshot, so such a
> benefit was not visible. This is just a theory so we should validate
> it. The another idea as discussed before [1] could be to try copying
> multiple tables in a single transaction. Now, keeping a transaction
> open for a longer time could have side-effects on the publisher node.
> So, we probably need to ensure that we don't perform multiple large
> syncs and even for smaller tables (and later sequences) perform it
> only for some threshold number of tables which we can figure out by
> some tests. Also, the other safety-check could be that anytime we need
> to perform streaming (sync with apply worker), we won't copy more
> tables in same transaction.
>
> Thoughts?

Yeah, maybe going to the publisher for creating a slot or only a
snapshot does not really make enough difference. I was hoping that
creating only snapshot by an existing replication slot would help the
performance. I guess I was either wrong or am missing something in the
implementation.

The tricky bit with keeping a long transaction to copy multiple tables
is deciding how many tables one transaction can copy.

Thanks,
-- 
Melih Mutlu
Microsoft

  84. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-06T03:56:37Z

    On Wed, Jul 5, 2023 at 1:48 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 4 Tem 2023 Sal,
> 08:42 tarihinde şunu yazdı:
> > > > But in the later patch the tablesync worker tries to reuse the slot during the
> > > > synchronization, so in this case the application_name should be same as
> > > slotname.
> > > >
> > >
> > > Fair enough. I am slightly afraid that if we can't show the benefits
> > > with later patches then we may need to drop them but at this stage I
> > > feel we need to investigate why those are not helping?
> >
> > Agreed. Now I'm planning to do performance testing independently. We can discuss
> > based on that or Melih's one.
>
> Here I attached  what I use for performance testing of this patch.
>
> I only benchmarked the patch set with reusing connections very roughly
> so far. But seems like it improves quite significantly. For example,
> it took 611 ms to sync 100 empty tables, it was 1782 ms without
> reusing connections.
> First 3 patches from the set actually bring a good amount of
> improvement, but not sure about the later patches yet.
>

I suggest then we should focus first on those 3, get them committed
and then look at the remaining.

> Amit Kapila <amit.kapila16@gmail.com>, 3 Tem 2023 Pzt, 08:59 tarihinde
> şunu yazdı:
> > On thinking about this, I think the primary benefit we were expecting
> > by saving network round trips for slot drop/create but now that we
> > anyway need an extra round trip to establish a snapshot, so such a
> > benefit was not visible. This is just a theory so we should validate
> > it. The another idea as discussed before [1] could be to try copying
> > multiple tables in a single transaction. Now, keeping a transaction
> > open for a longer time could have side-effects on the publisher node.
> > So, we probably need to ensure that we don't perform multiple large
> > syncs and even for smaller tables (and later sequences) perform it
> > only for some threshold number of tables which we can figure out by
> > some tests. Also, the other safety-check could be that anytime we need
> > to perform streaming (sync with apply worker), we won't copy more
> > tables in same transaction.
> >
> > Thoughts?
>
> Yeah, maybe going to the publisher for creating a slot or only a
> snapshot does not really make enough difference. I was hoping that
> creating only snapshot by an existing replication slot would help the
> performance. I guess I was either wrong or am missing something in the
> implementation.
>
> The tricky bit with keeping a long transaction to copy multiple tables
> is deciding how many tables one transaction can copy.
>

Yeah, I was thinking that we should not allow copying some threshold
data in one transaction. After every copy, we will check the size of
the table and add it to the previously copied table size in the same
transaction. Once the size crosses a certain threshold, we will end
the transaction. This may not be a very good scheme but I think it
this helps then it would be much simpler than creating-only-snapshot
approach.

-- 
With Regards,
Amit Kapila.

  85. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-07-06T09:47:40Z

    Dear Melih,

> Thanks for the 0003 patch. But it did not work for me. Can you create
> a subscription successfully with patch 0003 applied?
> I get the following error: " ERROR:  table copy could not start
> transaction on publisher: another command is already in progress".

You got the ERROR when all the patches (0001-0005) were applied, right?
I have focused on 0001 and 0002 only, so I missed something.
If it was not correct, please attach the logfile and test script what you did.

As you might know, the error is output when the worker executs walrcv_endstreaming()
before doing walrcv_startstreaming().

> I think streaming needs to be ended before moving to another table. So
> I changed the patch a little bit

Your modification seemed not correct. I applied only first three patches (0001-0003), and
executed attached script. Then I got following error on subscriber (attached as N2.log):

> ERROR:  could not send end-of-streaming message to primary: no COPY in progress

IIUC the tablesync worker has been already stopped streaming without your modification.
Please see process_syncing_tables_for_sync():

```
	if (MyLogicalRepWorker->relstate == SUBREL_STATE_CATCHUP &&
		current_lsn >= MyLogicalRepWorker->relstate_lsn)
	{
		TimeLineID	tli;
		char		syncslotname[NAMEDATALEN] = {0};
		char		originname[NAMEDATALEN] = {0};

		MyLogicalRepWorker->relstate = SUBREL_STATE_SYNCDONE;
...
		/*
		 * End streaming so that LogRepWorkerWalRcvConn can be used to drop
		 * the slot.
		 */
		walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);
```

This means that following changes should not be in the 0003, should be at 0005.
PSA fixed patches.

```
+	/*
+	 * If it's already connected to the publisher, end streaming before using
+	 * the same connection for another iteration
+	 */
+	if (LogRepWorkerWalRcvConn != NULL)
+	{
+		TimeLineID tli;
+		walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);
+	}
```


Besides, cfbot could not apply your patch set [1]. According to the log, the
bot tried to apply 0004 and 0005 first and got error. IIUC you should assign
same version number within the same mail, like v16-0001, v16-0002,....

[1]: http://cfbot.cputube.org/patch_43_3784.log

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  86. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-07T09:37:54Z

    Hi. Here are some review comments for the patch v16-0001

======
Commit message.

1.
Also; most of the code shared by both worker types are already combined
in LogicalRepApplyLoop(). There is no need to combine the rest in
ApplyWorkerMain() anymore.

~

/are already/is already/

/Also;/Also,/

~~~

2.
This commit introduces TablesyncWorkerMain() as a new entry point for
tablesync workers and separates both type of workers from each other.
This aims to increase code readability and help to maintain logical
replication workers separately.

2a.
/This commit/This patch/

~

2b.
"and separates both type of workers from each other"

Maybe that part can all be removed. The following sentence says the
same again anyhow.

======
src/backend/replication/logical/worker.c

3.
 static void stream_write_change(char action, StringInfo s);
 static void stream_open_and_write_change(TransactionId xid, char
action, StringInfo s);
 static void stream_close_file(void);
+static void set_stream_options(WalRcvStreamOptions *options,
+    char *slotname,
+    XLogRecPtr *origin_startpos);

~

Maybe a blank line was needed here because this static should not be
grouped with the other functions that are grouped for "Serialize and
deserialize changes for a toplevel transaction." comment.

~~~

4. set_stream_options

+ /* set_stream_options
+  * Set logical replication streaming options.
+  *
+  * This function sets streaming options including replication slot name and
+  * origin start position. Workers need these options for logical replication.
+  */
+static void
+set_stream_options(WalRcvStreamOptions *options,

The indentation is not right for this function comment.

~~~

5. set_stream_options

+ /*
+ * Even when the two_phase mode is requested by the user, it remains as
+ * the tri-state PENDING until all tablesyncs have reached READY state.
+ * Only then, can it become ENABLED.
+ *
+ * Note: If the subscription has no tables then leave the state as
+ * PENDING, which allows ALTER SUBSCRIPTION ... REFRESH PUBLICATION to
+ * work.
+ */
+ if (MySubscription->twophasestate == LOGICALREP_TWOPHASE_STATE_PENDING &&
+ AllTablesyncsReady())
+ options->proto.logical.twophase = true;
+}

This part of the refactoring seems questionable...

IIUC this new function was extracted from code in originally in
function ApplyWorkerMain()

But in that original code, this fragment above was guarded by the condition
if (!am_tablesync_worker())

But now where is that condition? e.g. What is stopping tablesync
working from getting into this code it previously would not have
executed?

~~~

6.
  AbortOutOfAnyTransaction();
- pgstat_report_subscription_error(MySubscription->oid, !am_tablesync_worker());
+ pgstat_report_subscription_error(MySubscription->oid,
+ !am_tablesync_worker());

Does this change have anything to do with this patch? Is it a quirk of
running pg_indent?

~~~
7. run_tablesync_worker

Since the stated intent of the patch is the separation of apply and
tablesync workers then shouldn't this function belong in the
tablesync.c file?

~~~
8. run_tablesync_worker

+ * Runs the tablesync worker.
+ * It starts syncing tables. After a successful sync, sets streaming options
+ * and starts streaming to catchup.
+ */
+static void
+run_tablesync_worker(WalRcvStreamOptions *options,

Nicer to have a blank line after the first sentence of that function comment?

~~~
9. run_apply_worker

+/*
+ * Runs the leader apply worker.
+ * It sets up replication origin, streaming options and then starts streaming.
+ */
+static void
+run_apply_worker(WalRcvStreamOptions *options,

Nicer to have a blank line after the first sentence of that function comment?

~~~
10. InitializeLogRepWorker

+/*
+ * Common initialization for logical replication workers; leader apply worker,
+ * parallel apply worker and tablesync worker.
  *
  * Initialize the database connection, in-memory subscription and necessary
  * config options.
  */
 void
-InitializeApplyWorker(void)
+InitializeLogRepWorker(void)

typo:

/workers;/workers:/

~~~
11. TablesyncWorkerMain

Since the stated intent of the patch is the separation of apply and
tablesync workers then shouldn't this function belong in the
tablesync.c file?

======
src/include/replication/worker_internal.h

12.
 #define isParallelApplyWorker(worker) ((worker)->leader_pid != InvalidPid)

+extern void finish_sync_worker(void);

~

I think the macro isParallelApplyWorker is associated with the am_XXX
inline functions that follow it, so it doesn’t seem the best place to
jam this extern in the middle of that.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  87. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-07-10T02:37:30Z

    Dear hackers,

Hi, I did a performance testing for v16 patch set.
Results show that patches significantly improves the performance in most cases.

# Method

Following tests were done 10 times per condition, and compared by median.
do_one_test.sh was used for the testing.

1.	Create tables on publisher
2.	Insert initial data on publisher
3.	Create tables on subscriber
4.	Create a replication slot (mysub_slot) on publisher
5.	Create a publication on publisher
6.	Create tables on subscriber
--- timer on ---
7.	Create subscription with pre-existing replication slot (mysub_slot)
8.	Wait until all srsubstate in pg_subscription_rel becomes 'r'
--- timer off ---

# Tested sources

I used three types of sources

* HEAD (f863d82)
* HEAD + 0001 + 0002
* HEAD + 0001 + 0002 + 0003

# Tested conditions

Following parameters were changed during the measurement.

### table size

* empty
* around 10kB

### number of tables

* 10
* 100
* 1000
* 2000

### max_sync_workers_per_subscription

* 2
* 4
* 8
* 16

## Results

Please see the attached image file. Each cell shows the improvement percentage of
measurement comapred with HEAD, HEAD + 0001 + 0002, and HEAD + 0001 + 0002 + 0003.

According to the measurement, we can say following things:

* In any cases the performance was improved from the HEAD.
* The improvement became more significantly if number of synced tables were increased.
* 0003 basically improved performance from first two patches
* Increasing workers could sometimes lead to lesser performance due to contention.
  This was occurred when the number of tables were small. Moreover, this was not only happen by patchset - it happened even if we used HEAD.
  Detailed analysis will be done later.

Mored deital, please see the excel file. It contains all the results of measurement.

## Detailed configuration

* Powerful machine was used:
 - Number of CPU: 120
 - Memory: 755 GB

* Both publisher and subscriber were on the same machine.
* Following GUC settings were used for both pub/sub:

```
wal_level = logical
shared_buffers = 40GB
max_worker_processes = 32
max_parallel_maintenance_workers = 24
max_parallel_workers = 32
synchronous_commit = off
checkpoint_timeout = 1d
max_wal_size = 24GB
min_wal_size = 15GB
autovacuum = off
max_wal_senders = 200
max_replication_slots = 200
```

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  88. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-10T07:08:31Z

    Hi, here are some review comments for patch v16-0002.

======
Commit message

1.
This commit allows reusing tablesync workers for syncing more than one
table sequentially during their lifetime, instead of exiting after
only syncing one table.

Before this commit, tablesync workers were capable of syncing only one
table. For each table, a new sync worker was launched and that worker would
exit when done processing the table.

Now, tablesync workers are not limited to processing only one
table. When done, they can move to processing another table in
the same subscription.

~

IMO that first paragraph can be removed because AFAIK the other
paragraphs are saying exactly the same thing but worded differently.

======
src/backend/replication/logical/tablesync.c

2. General -- for clean_sync_worker and finish_sync_worker

TBH, I found the separation of clean_sync_worker() and
finish_sync_worker() to be confusing. Can't it be rearranged to keep
the same function but just pass a boolean to tell it to exit or not
exit?

e.g.

finish_sync_worker(bool reuse_worker) { ... }

~~~

3. clean_sync_worker

  /*
- * Commit any outstanding transaction. This is the usual case, unless
- * there was nothing to do for the table.
+ * Commit any outstanding transaction. This is the usual case, unless there
+ * was nothing to do for the table.
  */

The word wrap seems OK, except the change seemed unrelated to this patch (??)

~~~

4.
+ /*
+ * Disconnect from publisher. Otherwise reused sync workers causes
+ * exceeding max_wal_senders
+ */

Missing period, and not an English sentence.

SUGGESTION (??)
Disconnect from the publisher otherwise reusing the sync worker can
error due to exceeding max_wal_senders.

~~~

5. finish_sync_worker

+/*
+ * Exit routine for synchronization worker.
+ */
+void
+pg_attribute_noreturn()
+finish_sync_worker(void)
+{
+ clean_sync_worker();
+
  /* And flush all writes. */
  XLogFlush(GetXLogWriteRecPtr());

  StartTransactionCommand();
  ereport(LOG,
- (errmsg("logical replication table synchronization worker for
subscription \"%s\", table \"%s\" has finished",
- MySubscription->name,
- get_rel_name(MyLogicalRepWorker->relid))));
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" has finished",
+ MySubscription->name)));
  CommitTransactionCommand();

In the original code, the XLogFlush was in a slightly different order
than in this refactored code. E.g. it came before signalling the apply
worker. Is it OK to be changed?

Keeping one function (suggested in #2) can maybe remove this potential issue.

======
src/backend/replication/logical/worker.c

6. LogicalRepApplyLoop

+ /*
+ * apply_dispatch() may have gone into apply_handle_commit()
+ * which can call process_syncing_tables_for_sync.
+ *
+ * process_syncing_tables_for_sync decides whether the sync of
+ * the current table is completed. If it is completed,
+ * streaming must be already ended. So, we can break the loop.
+ */
+ if (MyLogicalRepWorker->is_sync_completed)
+ {
+ endofstream = true;
+ break;
+ }
+

and

+ /*
+ * If is_sync_completed is true, this means that the tablesync
+ * worker is done with synchronization. Streaming has already been
+ * ended by process_syncing_tables_for_sync. We should move to the
+ * next table if needed, or exit.
+ */
+ if (MyLogicalRepWorker->is_sync_completed)
+ endofstream = true;

~

Instead of those code fragments above assigning 'endofstream' as a
side-effect, would it be the same (but tidier) to just modify the
other "breaking" condition below:

BEFORE:
/* Check if we need to exit the streaming loop. */
if (endofstream)
break;

AFTER:
/* Check if we need to exit the streaming loop. */
if (endofstream || MyLogicalRepWorker->is_sync_completed)
break;

~~~

7. LogicalRepApplyLoop

+ /*
+ * Tablesync workers should end streaming before exiting the main loop to
+ * drop replication slot. Only end streaming here for apply workers.
+ */
+ if (!am_tablesync_worker())
+ walrcv_endstreaming(LogRepWorkerWalRcvConn, &tli);

This comment does not seem very clear. Maybe it can be reworded:

SUGGESTION
End streaming here only for apply workers. Ending streaming for
tablesync workers is deferred until ... because ...

~~~

8. TablesyncWorkerMain

+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("%s for subscription \"%s\" has moved to sync table \"%s\"
with relid %u.",
+ get_worker_name(),
+ MySubscription->name,
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));
+ CommitTransactionCommand();

The "has moved to..." terminology is unusual. If you say something
"will be reused to..." then it matches better the commit message etc.

~~~

9.

+ if (!is_table_found)
+ break;

Instead of an infinite loop that is exited by this 'break' it might be
better to rearrange the logic slightly so the 'for' loop can exit
normally:

BEFORE:
for (;;)

AFTER
for (; !done;)

======
src/include/replication/worker_internal.h

10.
  XLogRecPtr relstate_lsn;
  slock_t relmutex;

+ /*
+ * Indicates whether tablesync worker has completed sycning its assigned
+ * table. If true, no need to continue with that table.
+ */
+ bool is_sync_completed;
+

10a.
Typo /sycning/syncing/

~

10b.
All the other tablesync-related fields of this struct are named as
relXXX, so I wonder if is better for this to follow the same pattern.
e.g. 'relsync_completed'

~

10c.
"If true, no need to continue with that table.".

I am not sure if this sentence is adding anything useful.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  89. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-10T14:22:58Z

    Hi,

Amit Kapila <amit.kapila16@gmail.com>, 6 Tem 2023 Per, 06:56 tarihinde
şunu yazdı:
>
> On Wed, Jul 5, 2023 at 1:48 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> >
> > Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 4 Tem 2023 Sal,
> > 08:42 tarihinde şunu yazdı:
> > > > > But in the later patch the tablesync worker tries to reuse the slot during the
> > > > > synchronization, so in this case the application_name should be same as
> > > > slotname.
> > > > >
> > > >
> > > > Fair enough. I am slightly afraid that if we can't show the benefits
> > > > with later patches then we may need to drop them but at this stage I
> > > > feel we need to investigate why those are not helping?
> > >
> > > Agreed. Now I'm planning to do performance testing independently. We can discuss
> > > based on that or Melih's one.
> >
> > Here I attached  what I use for performance testing of this patch.
> >
> > I only benchmarked the patch set with reusing connections very roughly
> > so far. But seems like it improves quite significantly. For example,
> > it took 611 ms to sync 100 empty tables, it was 1782 ms without
> > reusing connections.
> > First 3 patches from the set actually bring a good amount of
> > improvement, but not sure about the later patches yet.
> >
>
> I suggest then we should focus first on those 3, get them committed
> and then look at the remaining.
>

That sounds good. I'll do my best to address any review/concern from
reviewers now for the first 3 patches and hopefully those can get
committed first. I'll continue working on the remaining patches later.

-- 
Melih Mutlu
Microsoft

  90. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-10T14:31:24Z

    Hi,

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 6 Tem 2023 Per,
12:47 tarihinde şunu yazdı:
>
> Dear Melih,
>
> > Thanks for the 0003 patch. But it did not work for me. Can you create
> > a subscription successfully with patch 0003 applied?
> > I get the following error: " ERROR:  table copy could not start
> > transaction on publisher: another command is already in progress".
>
> You got the ERROR when all the patches (0001-0005) were applied, right?
> I have focused on 0001 and 0002 only, so I missed something.
> If it was not correct, please attach the logfile and test script what you did.

Yes, I did get an error with all patches applied. But with only 0001
and 0002, your version seems like working and mine does not.
What do you think about combining 0002 and 0003? Or should those stay separate?

> Hi, I did a performance testing for v16 patch set.
> Results show that patches significantly improves the performance in most cases.
>
> # Method
>
> Following tests were done 10 times per condition, and compared by median.
> do_one_test.sh was used for the testing.
>
> 1.      Create tables on publisher
> 2.      Insert initial data on publisher
> 3.      Create tables on subscriber
> 4.      Create a replication slot (mysub_slot) on publisher
> 5.      Create a publication on publisher
> 6.      Create tables on subscriber
> --- timer on ---
> 7.      Create subscription with pre-existing replication slot (mysub_slot)
> 8.      Wait until all srsubstate in pg_subscription_rel becomes 'r'
> --- timer off ---
>

Thanks for taking the time to do testing and sharing the results. This
is also how I've been doing the testing since, but the process was
half scripted, half manual work.

> According to the measurement, we can say following things:
>
> * In any cases the performance was improved from the HEAD.
> * The improvement became more significantly if number of synced tables were increased.

Yes, I believe it becomes more significant when workers spend less
time with actually copying data but more with other stuff like
launching workers, opening connections etc.

> * 0003 basically improved performance from first two patches

Agree, 0003 is definitely a good addition which was missing earlier.


Thanks,
-- 
Melih Mutlu
Microsoft

  91. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-11T02:36:09Z

    Here are some review comments for patch v16-00003

======
1. Commit Message.

The patch description is missing.

======
2. General.

+LogicalRepSyncTableStart(XLogRecPtr *origin_startpos, int worker_slot)

and

+start_table_sync(XLogRecPtr *origin_startpos,
+ char **myslotname,
+ int worker_slot)

and

@@ -4548,12 +4552,13 @@ run_tablesync_worker(WalRcvStreamOptions *options,
  char *slotname,
  char *originname,
  int originname_size,
- XLogRecPtr *origin_startpos)
+ XLogRecPtr *origin_startpos,
+ int worker_slot)


It seems the worker_slot is being passed all over the place as an
additional function argument so that it can be used to construct an
application_name. Is it possible/better to introduce a new
'MyLogicalRepWorker' field for the 'worker_slot' so it does not have
to be passed like this?

======
src/backend/replication/logical/tablesync.c

3.
+ /*
+ * Disconnect from publisher. Otherwise reused sync workers causes
+ * exceeding max_wal_senders.
+ */
+ if (LogRepWorkerWalRcvConn != NULL)
+ {
+ walrcv_disconnect(LogRepWorkerWalRcvConn);
+ LogRepWorkerWalRcvConn = NULL;
+ }
+

Why is this comment mentioning anything about "reused workers" at all?
The worker process exits in this function, right?

~~~

4. LogicalRepSyncTableStart

  /*
- * Here we use the slot name instead of the subscription name as the
- * application_name, so that it is different from the leader apply worker,
- * so that synchronous replication can distinguish them.
+ * Connect to publisher if not yet. The application_name must be also
+ * different from the leader apply worker because synchronous replication
+ * must distinguish them.
  */

I felt all the details in the 2nd part of this comment belong inside
the condition, not outside.

SUGGESTION
/* Connect to the publisher if haven't done so already. */

~~~

5.
+ if (LogRepWorkerWalRcvConn == NULL)
+ {
+ char application_name[NAMEDATALEN];
+
+ /*
+ * FIXME: set appropriate application_name. Previously, the slot name
+ * was used becasue the lifetime of the tablesync worker was same as
+ * that, but now the tablesync worker handles many slots during the
+ * synchronization so that it is not suitable. So what should be?
+ * Note that if the tablesync worker starts to reuse the replication
+ * slot during synchronization, we should use the slot name as
+ * application_name again.
+ */
+ snprintf(application_name, NAMEDATALEN, "pg_%u_sync_%i",
+ MySubscription->oid, worker_slot);
+ LogRepWorkerWalRcvConn =
+ walrcv_connect(MySubscription->conninfo, true,
+    must_use_password,
+    application_name, &err);
+ }

5a.
/becasue/because/

~

5b.
I am not sure about what name this should ideally use, but anyway for
uniqueness doesn't it still need to include the GetSystemIdentifier()
same as function ReplicationSlotNameForTablesync() was doing?

Maybe this can use the same function ReplicationSlotNameForTablesync()
can be used but just pass the worker_slot instead of the relid?

======
src/backend/replication/logical/worker.c

6. LogicalRepApplyLoop

  /*
  * Init the ApplyMessageContext which we clean up after each replication
- * protocol message.
+ * protocol message, if needed.
  */
- ApplyMessageContext = AllocSetContextCreate(ApplyContext,
- "ApplyMessageContext",
- ALLOCSET_DEFAULT_SIZES);
+ if (!ApplyMessageContext)
+ ApplyMessageContext = AllocSetContextCreate(ApplyContext,
+ "ApplyMessageContext",
+

Maybe slightly reword the comment.

BEFORE:
Init the ApplyMessageContext which we clean up after each replication
protocol message, if needed.

AFTER:
Init the ApplyMessageContext if needed. This context is cleaned up
after each replication protocol message.

======
src/backend/replication/walsender.c

7.
+ /*
+ * Initialize the flag again because this streaming may be
+ * second time.
+ */
+ streamingDoneSending = streamingDoneReceiving = false;

Isn't this only possible to be 2nd time because the "reuse tablesync
worker" might re-issue a START_REPLICATION again to the same
WALSender? So, should this flag reset ONLY be done for the logical
replication ('else' part), otherwise it should be asserted false?

e.g. Would it be better to be like this?

if (cmd->kind == REPLICATION_KIND_PHYSICAL)
{
Assert(!streamingDoneSending && !streamingDoneReceiving)
StartReplication(cmd);
}
else
{
/* Reset flags because reusing tablesync workers can mean this is the
second time here. */
streamingDoneSending = streamingDoneReceiving = false;
StartLogicalReplication(cmd);
}

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  92. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-11T02:59:24Z

    On Tue, Jul 11, 2023 at 12:31 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 6 Tem 2023 Per,
> 12:47 tarihinde şunu yazdı:
> >
> > Dear Melih,
> >
> > > Thanks for the 0003 patch. But it did not work for me. Can you create
> > > a subscription successfully with patch 0003 applied?
> > > I get the following error: " ERROR:  table copy could not start
> > > transaction on publisher: another command is already in progress".
> >
> > You got the ERROR when all the patches (0001-0005) were applied, right?
> > I have focused on 0001 and 0002 only, so I missed something.
> > If it was not correct, please attach the logfile and test script what you did.
>
> Yes, I did get an error with all patches applied. But with only 0001
> and 0002, your version seems like working and mine does not.
> What do you think about combining 0002 and 0003? Or should those stay separate?
>

Even if patches 0003 and 0002 are to be combined, I think that should
not happen until after the "reuse" design is confirmed which way is
best.

e.g. IMO it might be easier to compare the different PoC designs for
patch 0002 if there is no extra logic involved.

PoC design#1 -- each tablesync decides for itself what to do next
after it finishes
PoC design#2 -- reuse tablesync using a "pool" of available workers

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  93. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-11T03:44:56Z

    On Mon, Jul 10, 2023 at 8:01 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 6 Tem 2023 Per,
> 12:47 tarihinde şunu yazdı:
> >
> > Dear Melih,
> >
> > > Thanks for the 0003 patch. But it did not work for me. Can you create
> > > a subscription successfully with patch 0003 applied?
> > > I get the following error: " ERROR:  table copy could not start
> > > transaction on publisher: another command is already in progress".
> >
> > You got the ERROR when all the patches (0001-0005) were applied, right?
> > I have focused on 0001 and 0002 only, so I missed something.
> > If it was not correct, please attach the logfile and test script what you did.
>
> Yes, I did get an error with all patches applied. But with only 0001
> and 0002, your version seems like working and mine does not.
> What do you think about combining 0002 and 0003? Or should those stay separate?
>

I am fine either way but I think one minor advantage of keeping 0003
separate is that we can focus on some of the problems specific to that
patch. For example, the following comment in the 0003 patch: "FIXME:
set appropriate application_name...". I have given a suggestion to
address it in [1] and Kuroda-San seems to have addressed the same but
I am not sure if all of us agree with that or if there is any better
way to address it. What do you think?

>
> > * 0003 basically improved performance from first two patches
>
> Agree, 0003 is definitely a good addition which was missing earlier.
>

+1.

[1] - https://www.postgresql.org/message-id/CAA4eK1JOZHmy2o2F2wTCPKsjpwDiKZPOeTa_jt%3Dwm2JLbf-jsg%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  94. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-07-13T04:09:12Z

    Dear Melih,

> > > Thanks for the 0003 patch. But it did not work for me. Can you create
> > > a subscription successfully with patch 0003 applied?
> > > I get the following error: " ERROR:  table copy could not start
> > > transaction on publisher: another command is already in progress".
> >
> > You got the ERROR when all the patches (0001-0005) were applied, right?
> > I have focused on 0001 and 0002 only, so I missed something.
> > If it was not correct, please attach the logfile and test script what you did.
> 
> Yes, I did get an error with all patches applied. But with only 0001
> and 0002, your version seems like working and mine does not.

Hmm, really? IIUC I did not modify 0001 and 0002 patches, I just re-assigned the
version number. I compared between yours and mine, but no meaningful differences
were found. E.g., following command compared v4-0002 and v16-0002:

```
diff --git a/../reuse_workers/v4-0002-Reuse-Tablesync-Workers.patch b/../reuse_workers/hayato/v16-0002-Reuse-Tablesync-Workers.patch
index 5350216e98..7785a573e4 100644
--- a/../reuse_workers/v4-0002-Reuse-Tablesync-Workers.patch
+++ b/../reuse_workers/hayato/v16-0002-Reuse-Tablesync-Workers.patch
@@ -1,7 +1,7 @@
-From d482022b40e0a5ce1b74fd0e320cb5b45da2f671 Mon Sep 17 00:00:00 2001
+From db3e8e2d7aadea79126c5816bce8b06dc82f33c2 Mon Sep 17 00:00:00 2001
 From: Melih Mutlu <m.melihmutlu@gmail.com>
 Date: Tue, 4 Jul 2023 22:04:46 +0300
-Subject: [PATCH 2/5] Reuse Tablesync Workers
+Subject: [PATCH v16 2/5] Reuse Tablesync Workers
 
 This commit allows reusing tablesync workers for syncing more than one
 table sequentially during their lifetime, instead of exiting after
@@ -324,5 +324,5 @@ index 7aba034774..1e9f8e6e72 100644
  static inline bool
  am_tablesync_worker(void)
 -- 
-2.25.1
+2.27.0
```

For confirmation, please attach the logfile and test script what you did
if you could reproduce?

> What do you think about combining 0002 and 0003? Or should those stay
> separate?

I have no strong opinion, but it may be useful to keep them pluggable.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  95. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-07-13T04:12:49Z

    Dear Peter,

Thanks for reviewing! I'm not sure what should be, but I modified only my part - 0003.
PSA new patchset. Other patches were not changed.
(I attached till 0005 just in case, but I did not consider about 0004 and 0005)

> ======
> 1. Commit Message.
> 
> The patch description is missing.

Briefly added.

> 2. General.
> 
> +LogicalRepSyncTableStart(XLogRecPtr *origin_startpos, int worker_slot)
> 
> and
> 
> +start_table_sync(XLogRecPtr *origin_startpos,
> + char **myslotname,
> + int worker_slot)
> 
> and
> 
> @@ -4548,12 +4552,13 @@ run_tablesync_worker(WalRcvStreamOptions
> *options,
>   char *slotname,
>   char *originname,
>   int originname_size,
> - XLogRecPtr *origin_startpos)
> + XLogRecPtr *origin_startpos,
> + int worker_slot)
> 
> 
> It seems the worker_slot is being passed all over the place as an
> additional function argument so that it can be used to construct an
> application_name. Is it possible/better to introduce a new
> 'MyLogicalRepWorker' field for the 'worker_slot' so it does not have
> to be passed like this?

I'm not sure it should be, but I did. How do you think?

> src/backend/replication/logical/tablesync.c
> 
> 3.
> + /*
> + * Disconnect from publisher. Otherwise reused sync workers causes
> + * exceeding max_wal_senders.
> + */
> + if (LogRepWorkerWalRcvConn != NULL)
> + {
> + walrcv_disconnect(LogRepWorkerWalRcvConn);
> + LogRepWorkerWalRcvConn = NULL;
> + }
> +
> 
> Why is this comment mentioning anything about "reused workers" at all?
> The worker process exits in this function, right?

I considered that code again, and I found this part is not needed anymore.

Initially it was added in 0002, this is because workers established new connections
without exiting and walsenders on publisher might be remained. So This was correct
for 0002 patch.
But now, in 0003 patch, workers reuse connections, which means that no need to call
walrcv_disconnect() explicitly. It is done when processes are exit.

> 4. LogicalRepSyncTableStart
> 
>   /*
> - * Here we use the slot name instead of the subscription name as the
> - * application_name, so that it is different from the leader apply worker,
> - * so that synchronous replication can distinguish them.
> + * Connect to publisher if not yet. The application_name must be also
> + * different from the leader apply worker because synchronous replication
> + * must distinguish them.
>   */
> 
> I felt all the details in the 2nd part of this comment belong inside
> the condition, not outside.
> 
> SUGGESTION
> /* Connect to the publisher if haven't done so already. */

Changed.

> 5.
> + if (LogRepWorkerWalRcvConn == NULL)
> + {
> + char application_name[NAMEDATALEN];
> +
> + /*
> + * FIXME: set appropriate application_name. Previously, the slot name
> + * was used becasue the lifetime of the tablesync worker was same as
> + * that, but now the tablesync worker handles many slots during the
> + * synchronization so that it is not suitable. So what should be?
> + * Note that if the tablesync worker starts to reuse the replication
> + * slot during synchronization, we should use the slot name as
> + * application_name again.
> + */
> + snprintf(application_name, NAMEDATALEN, "pg_%u_sync_%i",
> + MySubscription->oid, worker_slot);
> + LogRepWorkerWalRcvConn =
> + walrcv_connect(MySubscription->conninfo, true,
> +    must_use_password,
> +    application_name, &err);
> + }
> 
> 5a.
> /becasue/because/

Modified. Also, comments were moved atop ApplicationNameForTablesync.
I was not sure when it is removed, but I kept it.

> 
> 5b.
> I am not sure about what name this should ideally use, but anyway for
> uniqueness doesn't it still need to include the GetSystemIdentifier()
> same as function ReplicationSlotNameForTablesync() was doing?
> 
> Maybe this can use the same function ReplicationSlotNameForTablesync()
> can be used but just pass the worker_slot instead of the relid?

Good point. ApplicationNameForTablesync() was defined and used.

> src/backend/replication/logical/worker.c
> 
> 6. LogicalRepApplyLoop
> 
>   /*
>   * Init the ApplyMessageContext which we clean up after each replication
> - * protocol message.
> + * protocol message, if needed.
>   */
> - ApplyMessageContext = AllocSetContextCreate(ApplyContext,
> - "ApplyMessageContext",
> - ALLOCSET_DEFAULT_SIZES);
> + if (!ApplyMessageContext)
> + ApplyMessageContext = AllocSetContextCreate(ApplyContext,
> + "ApplyMessageContext",
> +
> 
> Maybe slightly reword the comment.
> 
> BEFORE:
> Init the ApplyMessageContext which we clean up after each replication
> protocol message, if needed.
> 
> AFTER:
> Init the ApplyMessageContext if needed. This context is cleaned up
> after each replication protocol message.

Changed.

> src/backend/replication/walsender.c
> 
> 7.
> + /*
> + * Initialize the flag again because this streaming may be
> + * second time.
> + */
> + streamingDoneSending = streamingDoneReceiving = false;
> 
> Isn't this only possible to be 2nd time because the "reuse tablesync
> worker" might re-issue a START_REPLICATION again to the same
> WALSender? So, should this flag reset ONLY be done for the logical
> replication ('else' part), otherwise it should be asserted false?
> 
> e.g. Would it be better to be like this?
> 
> if (cmd->kind == REPLICATION_KIND_PHYSICAL)
> {
> Assert(!streamingDoneSending && !streamingDoneReceiving)
> StartReplication(cmd);
> }
> else
> {
> /* Reset flags because reusing tablesync workers can mean this is the
> second time here. */
> streamingDoneSending = streamingDoneReceiving = false;
> StartLogicalReplication(cmd);
> }
>

It's OK to modify the comment. But after considering more, I started to think that
any specification for physical replication should not be changed.
So I accepted comments only for the logical rep.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  96. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-13T20:27:41Z

    Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 11 Tem 2023 Sal, 05:59 tarihinde şunu
yazdı:
> Even if patches 0003 and 0002 are to be combined, I think that should
> not happen until after the "reuse" design is confirmed which way is
> best.
>
> e.g. IMO it might be easier to compare the different PoC designs for
> patch 0002 if there is no extra logic involved.
>
> PoC design#1 -- each tablesync decides for itself what to do next
> after it finishes
> PoC design#2 -- reuse tablesync using a "pool" of available workers

Right. I made a patch 0003 to change 0002 so that tables will be assigned
to sync workers by apply worker.
It's a rough POC and ignores some edge cases. But this is what I think how
apply worker would take the responsibility of table assignments. Hope the
implementation makes sense and I'm not missing anything that may cause
degraded perforrmance.

PoC design#1 --> apply only patch 0001 and 0002
PoC design#2 --> apply all patches, 0001, 0002 and 0003

Here are some quick numbers with 100 empty tables.

+--------------+----------------+----------------+----------------+
|              | 2 sync workers | 4 sync workers | 8 sync workers |
+--------------+----------------+----------------+----------------+
| POC design#1 | 1909.873 ms    | 986.261 ms     | 552.404 ms     |
+--------------+----------------+----------------+----------------+
| POC design#2 | 4962.208 ms    | 1240.503 ms    | 1165.405 ms    |
+--------------+----------------+----------------+----------------+
| master       | 2666.008 ms    | 1462.012 ms    | 986.848 ms     |
+--------------+----------------+----------------+----------------+

Seems like design#1 is better than both design#2 and master overall. It's
surprising to see that even master beats design#2 in some cases though. Not
sure if that is expected or there are some places to improve design#2 even
more.

What do you think?

PS: I only attached the related patches and not the whole patch set. 0001
and 0002 may contain some of your earlier reviews, but I'll send a proper
updated set soon.

Thanks,
-- 
Melih Mutlu
Microsoft

  97. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-14T08:11:35Z

    On Fri, Jul 14, 2023 at 1:58 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Here are some quick numbers with 100 empty tables.
>
> +--------------+----------------+----------------+----------------+
> |              | 2 sync workers | 4 sync workers | 8 sync workers |
> +--------------+----------------+----------------+----------------+
> | POC design#1 | 1909.873 ms    | 986.261 ms     | 552.404 ms     |
> +--------------+----------------+----------------+----------------+
> | POC design#2 | 4962.208 ms    | 1240.503 ms    | 1165.405 ms    |
> +--------------+----------------+----------------+----------------+
> | master       | 2666.008 ms    | 1462.012 ms    | 986.848 ms     |
> +--------------+----------------+----------------+----------------+
>
> Seems like design#1 is better than both design#2 and master overall. It's surprising to see that even master beats design#2 in some cases though. Not sure if that is expected or there are some places to improve design#2 even more.
>

Yeah, it is quite surprising that Design#2 is worse than master. I
suspect there is something wrong going on with your Design#2 patch.
One area to check is whether apply worker is able to quickly assign
the new relations to tablesync workers. Note that currently after the
first time assigning the tables to workers, the apply worker may wait
before processing the next set of tables in the main loop of
LogicalRepApplyLoop(). The other minor point about design#2
implementation is that you may want to first assign the allocated
tablesync workers before trying to launch a new worker.

>
> PS: I only attached the related patches and not the whole patch set. 0001 and 0002 may contain some of your earlier reviews, but I'll send a proper updated set soon.
>

Yeah, that would be helpful.

-- 
With Regards,
Amit Kapila.

  98. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-14T08:23:50Z

    Hi Kuroda-san.

Here are some review comments for the v17-0003 patch. They are all minor.

======
Commit message

1.
Previously tablesync workers establish new connections when it changes
the syncing
table, but this might have additional overhead. This patch allows to
reuse connections
instead.

~

/This patch allows to reuse connections instead./This patch allows the
existing connection to be reused./

~~~

2.
As for the publisher node, this patch allows to reuse logical
walsender processes
after the streaming is done once.

~

Is this paragraph even needed? Since the connection is reused then it
already implies the other end (the Wlasender) is being reused, right?

======
src/backend/replication/logical/tablesync.c

3.
+ * FIXME: set appropriate application_name. Previously, the slot name was used
+ * because the lifetime of the tablesync worker was same as that, but now the
+ * tablesync worker handles many slots during the synchronization so that it is
+ * not suitable. So what should be? Note that if the tablesync worker starts to
+ * reuse the replication slot during synchronization, we should use the slot
+ * name as application_name again.
+ */
+static void
+ApplicationNameForTablesync(Oid suboid, int worker_slot,
+ char *application_name, Size szapp)

3a.
I felt that most of this FIXME comment belongs with the calling code,
not here.

3b.
Also, maybe it needs some rewording -- I didn't understand exactly
what it is trying to say.


~~~

4.
- /*
- * Here we use the slot name instead of the subscription name as the
- * application_name, so that it is different from the leader apply worker,
- * so that synchronous replication can distinguish them.
- */
- LogRepWorkerWalRcvConn =
- walrcv_connect(MySubscription->conninfo, true,
-    must_use_password,
-    slotname, &err);
+ /* Connect to the publisher if haven't done so already. */
+ if (LogRepWorkerWalRcvConn == NULL)
+ {
+ char application_name[NAMEDATALEN];
+
+ /*
+ * The application_name must be also different from the leader apply
+ * worker because synchronous replication must distinguish them.
+ */
+ ApplicationNameForTablesync(MySubscription->oid,
+ MyLogicalRepWorker->worker_slot,
+ application_name,
+ NAMEDATALEN);
+ LogRepWorkerWalRcvConn =
+ walrcv_connect(MySubscription->conninfo, true,
+    must_use_password,
+    application_name, &err);
+ }
+

Should the comment mention the "subscription name" as it did before?

SUGGESTION
The application_name must differ from the subscription name (used by
the leader apply worker) because synchronous replication has to be
able to distinguish this worker from the leader apply worker.

======
src/backend/replication/logical/worker.c

5.
-start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
+start_table_sync(XLogRecPtr *origin_startpos,
+ char **myslotname)

This is a wrapping change only. It looks like an unnecessary hangover
from a previous version of 0003.

======
src/backend/replication/walsender.c

6. exec_replication_command

+
  if (cmd->kind == REPLICATION_KIND_PHYSICAL)
  StartReplication(cmd);
~

The extra blank line does not belong in this patch.

======
src/include/replication/worker_internal.h

+ /* Indicates the slot number which corresponds to this LogicalRepWorker. */
+ int worker_slot;
+

6a
I think this field is very fundamental, so IMO it should be defined at
the top of the struct, maybe nearby the other 'in_use' and
'generation' fields.

~

6b.
Also, since this is already a "worker" struct so there is no need to
have "worker" in the field name again -- just "slot_number" or
"slotnum" might be a better name.

And then the comment can also be simplified.

SUGGESTION
/* Slot number of this worker. */
int slotnum;

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  99. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-14T09:36:49Z

    Hi,

Amit Kapila <amit.kapila16@gmail.com>, 14 Tem 2023 Cum, 11:11 tarihinde
şunu yazdı:

> Yeah, it is quite surprising that Design#2 is worse than master. I
> suspect there is something wrong going on with your Design#2 patch.
> One area to check is whether apply worker is able to quickly assign
> the new relations to tablesync workers. Note that currently after the
> first time assigning the tables to workers, the apply worker may wait
> before processing the next set of tables in the main loop of
> LogicalRepApplyLoop(). The other minor point about design#2
> implementation is that you may want to first assign the allocated
> tablesync workers before trying to launch a new worker.
>

It's not actually worse than master all the time. It seems like it's just
unreliable.
Here are some consecutive runs for both designs and master.

design#1 = 1621,527 ms, 1788,533 ms, 1645,618 ms, 1702,068 ms, 1745,753 ms
design#2 = 2089,077 ms, 1864,571 ms, 4574,799 ms, 5422,217 ms, 1905,944 ms
master = 2815,138 ms, 2481,954 ms , 2594,413 ms, 2620,690 ms, 2489,323 ms

And apply worker was not busy with applying anything during these
experiments since there were not any writes to the publisher. I'm not sure
how that would also affect the performance if there were any writes.

Thanks,
-- 
Melih Mutlu
Microsoft

  100. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-15T11:18:14Z

    On Fri, Jul 14, 2023 at 3:07 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com>, 14 Tem 2023 Cum, 11:11 tarihinde şunu yazdı:
>>
>> Yeah, it is quite surprising that Design#2 is worse than master. I
>> suspect there is something wrong going on with your Design#2 patch.
>> One area to check is whether apply worker is able to quickly assign
>> the new relations to tablesync workers. Note that currently after the
>> first time assigning the tables to workers, the apply worker may wait
>> before processing the next set of tables in the main loop of
>> LogicalRepApplyLoop(). The other minor point about design#2
>> implementation is that you may want to first assign the allocated
>> tablesync workers before trying to launch a new worker.
>
>
> It's not actually worse than master all the time. It seems like it's just unreliable.
> Here are some consecutive runs for both designs and master.
>
> design#1 = 1621,527 ms, 1788,533 ms, 1645,618 ms, 1702,068 ms, 1745,753 ms
> design#2 = 2089,077 ms, 1864,571 ms, 4574,799 ms, 5422,217 ms, 1905,944 ms
> master = 2815,138 ms, 2481,954 ms , 2594,413 ms, 2620,690 ms, 2489,323 ms
>
> And apply worker was not busy with applying anything during these experiments since there were not any writes to the publisher. I'm not sure how that would also affect the performance if there were any writes.
>

Yeah, this is a valid point. I think this is in favor of the Design#1
approach we are discussing here. One thing I was thinking whether we
can do anything to alleviate the contention at the higher worker
count. One possibility is to have some kind of available worker list
which can be used to pick up the next worker instead of checking all
the workers while assigning the next table. We can probably explore it
separately once the first three patches are ready because anyway, this
will be an optimization atop the Design#1 approach.

-- 
With Regards,
Amit Kapila.

  101. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-17T15:54:30Z

    Hi,

PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's
reviews for 0001 and 0002 with some small comments below.

Peter Smith <smithpb2250@gmail.com>, 10 Tem 2023 Pzt, 10:09 tarihinde şunu
yazdı:

> 6. LogicalRepApplyLoop
>
> + /*
> + * apply_dispatch() may have gone into apply_handle_commit()
> + * which can call process_syncing_tables_for_sync.
> + *
> + * process_syncing_tables_for_sync decides whether the sync of
> + * the current table is completed. If it is completed,
> + * streaming must be already ended. So, we can break the loop.
> + */
> + if (MyLogicalRepWorker->is_sync_completed)
> + {
> + endofstream = true;
> + break;
> + }
> +
>
> and
>
> + /*
> + * If is_sync_completed is true, this means that the tablesync
> + * worker is done with synchronization. Streaming has already been
> + * ended by process_syncing_tables_for_sync. We should move to the
> + * next table if needed, or exit.
> + */
> + if (MyLogicalRepWorker->is_sync_completed)
> + endofstream = true;
>
> ~
>
> Instead of those code fragments above assigning 'endofstream' as a
> side-effect, would it be the same (but tidier) to just modify the
> other "breaking" condition below:
>
> BEFORE:
> /* Check if we need to exit the streaming loop. */
> if (endofstream)
> break;
>
> AFTER:
> /* Check if we need to exit the streaming loop. */
> if (endofstream || MyLogicalRepWorker->is_sync_completed)
> break;
>

First place you mentioned also breaks the infinite loop. Such an if
statement is needed there with or without endofstream assignment.

I think if there is a flag to break a loop, using that flag to indicate
that we should exit the loop seems more appropriate to me. I see that it
would be a bit tidier without endofstream = true lines, but I feel like it
would also be less readable.

I don't have a strong opinion though. I'm just keeping them as they are for
now, but I can change them if you disagree.


>
> 10b.
> All the other tablesync-related fields of this struct are named as
> relXXX, so I wonder if is better for this to follow the same pattern.
> e.g. 'relsync_completed'
>

Aren't those start with rel because they're related to the relation that
the tablesync worker is syncing? is_sync_completed is not a relation
specific field. I'm okay with changing the name but feel like
relsync_completed would be misleading.

Thanks,
-- 
Melih Mutlu
Microsoft

  102. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-18T01:25:40Z

    On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's reviews for 0001 and 0002 with some small comments below.
>

Thanks, I will take another look at these soon. FYI, the 0001 patch
does not apply cleanly. It needs to be rebased again because
get_worker_name() function was recently removed from HEAD.

replication/logical/worker.o: In function `InitializeLogRepWorker':
/home/postgres/oss_postgres_misc/src/backend/replication/logical/worker.c:4605:
undefined reference to `get_worker_name'

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  103. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-18T01:33:27Z

    On Tue, Jul 18, 2023 at 11:25 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> >
> > Hi,
> >
> > PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's reviews for 0001 and 0002 with some small comments below.
> >
>
> Thanks, I will take another look at these soon. FYI, the 0001 patch
> does not apply cleanly. It needs to be rebased again because
> get_worker_name() function was recently removed from HEAD.
>

Sorry, to be more correct -- it applied OK, but failed to build.

> replication/logical/worker.o: In function `InitializeLogRepWorker':
> /home/postgres/oss_postgres_misc/src/backend/replication/logical/worker.c:4605:
> undefined reference to `get_worker_name'
>
> ------
> Kind Regards,
> Peter Smith.
> Fujitsu Australia

  104. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-18T09:03:38Z

    Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 18 Tem 2023 Sal, 04:33 tarihinde şunu
yazdı:

> On Tue, Jul 18, 2023 at 11:25 AM Peter Smith <smithpb2250@gmail.com>
> wrote:
> >
> > On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu <m.melihmutlu@gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > PFA updated patches. Rebased 0003 with minor changes. Addressed
> Peter's reviews for 0001 and 0002 with some small comments below.
> > >
> >
> > Thanks, I will take another look at these soon. FYI, the 0001 patch
> > does not apply cleanly. It needs to be rebased again because
> > get_worker_name() function was recently removed from HEAD.
> >
>
> Sorry, to be more correct -- it applied OK, but failed to build.
>

Attached the fixed patchset.

Thanks,
-- 
Melih Mutlu
Microsoft

  105. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-18T12:47:19Z

    On Tue, Jul 18, 2023 at 2:33 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Attached the fixed patchset.
>

Few comments on 0001
====================
1.
+ logicalrep_worker_attach(worker_slot);
+
+ /* Setup signal handling */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * We don't currently need any ResourceOwner in a walreceiver process, but
+ * if we did, we could call CreateAuxProcessResourceOwner here.
+ */
+
+ /* Initialise stats to a sanish value */
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
+
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
+
+ InitializeLogRepWorker();
+
+ /* Connect to the origin and start the replication. */
+ elog(DEBUG1, "connecting to publisher using connection string \"%s\"",
+ MySubscription->conninfo);
+
+ /*
+ * Setup callback for syscache so that we know when something changes in
+ * the subscription relation state.
+ */
+ CacheRegisterSyscacheCallback(SUBSCRIPTIONRELMAP,
+   invalidate_syncing_table_states,
+   (Datum) 0);

It seems this part of the code is the same for ApplyWorkerMain() and
TablesyncWorkerMain(). So, won't it be better to move it into a common
function?

2. Can LogicalRepSyncTableStart() be static function?

3. I think you don't need to send 0004, 0005 each time till we are
able to finish patches till 0003.

4. In 0001's commit message, you can say that it will help the
upcoming reuse tablesync worker patch.

-- 
With Regards,
Amit Kapila.

  106. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-07-18T14:11:52Z

    On Tue, 11 Jul 2023 at 08:30, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jul 11, 2023 at 12:31 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> >
> > Hi,
> >
> > Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 6 Tem 2023 Per,
> > 12:47 tarihinde şunu yazdı:
> > >
> > > Dear Melih,
> > >
> > > > Thanks for the 0003 patch. But it did not work for me. Can you create
> > > > a subscription successfully with patch 0003 applied?
> > > > I get the following error: " ERROR:  table copy could not start
> > > > transaction on publisher: another command is already in progress".
> > >
> > > You got the ERROR when all the patches (0001-0005) were applied, right?
> > > I have focused on 0001 and 0002 only, so I missed something.
> > > If it was not correct, please attach the logfile and test script what you did.
> >
> > Yes, I did get an error with all patches applied. But with only 0001
> > and 0002, your version seems like working and mine does not.
> > What do you think about combining 0002 and 0003? Or should those stay separate?
> >
>
> Even if patches 0003 and 0002 are to be combined, I think that should
> not happen until after the "reuse" design is confirmed which way is
> best.
>
> e.g. IMO it might be easier to compare the different PoC designs for
> patch 0002 if there is no extra logic involved.
>
> PoC design#1 -- each tablesync decides for itself what to do next
> after it finishes
> PoC design#2 -- reuse tablesync using a "pool" of available workers

I did a POC for design#2 for implementing a worker pool to synchronize
the tables for a subscriber. The core design is the same as what Melih
had implemented at [1]. I had already started the implementation of
POC based on one of the earlier e-mail [2] Peter had shared.
The POC has been implemented like:
a) Apply worker will check the tablesync pool and see if any tablesync
worker is free:
 i) If there are no free workers in the pool, start a table sync
worker and add it to the table sync pool.
 ii) If there are free workers in the pool, re-use the tablesync
worker for synchronizing another table.
b) Apply worker will check if the tables are synchronized, if all the
tables are synchronized apply worker will release all the workers from
the tablesync pool
c) Apply worker and tablesync worker has shared memory to share the
following relation data and execution state between the apply worker
and the tablesync worker
d) The apply worker and tablesync worker's pid are also stored in the
shared memory so that we need not take a lock on LogicalRepWorkerLock
and loop on max_logical_replication_workers every time. We use the pid
stored in shared memory to wake up the apply worker and tablesync
worker whenever needed.

While I was implementing the POC I found one issue in the POC
patch(there is no problem with the HEAD code, issue was only with the
POC):
1) Apply worker was waiting for the table to be set to SYNCDONE.
2) Mean time tablesync worker sets the table to SYNCDONE and sets
apply worker's latch.
3) Apply worker will reset the latch set by tablesync and go to main
loop and wait in main loop latch(since tablesync worker's latch was
already reset, apply worker will wait for 1 second)
To fix this I had to set apply worker's latch once in 1ms in this case
alone which is not a good solution as it will consume a lot of cpu
cycles. A better fix for this would be to introduce a new subscription
relation state.

Attached patch has the changes for the same. 001, 0002 and 0003 are
the patches shared by Melih and Kuroda-san earlier. 0004 patch has the
changes for the POC of Tablesync worker pool implementation.
POC design 1: Tablesync worker identifies the tables that should be
synced and reuses the connection.
POC design 2: Tablesync worker pool with apply worker scheduling the
work to tablesync workers in the tablesync pool and reusing the
connection.
Performance results for 10 empty tables:
+-------------------+--------------------+--------------------+----------------------+----------------+
|                        | 2 sync workers | 4 sync workers | 8 sync
workers  | 16 sync workers|
+-------------------+--------------------+--------------------+----------------------+----------------+
| HEAD              | 128.4685 ms    | 121.271 ms     | 136.5455 ms
 |   N/A          |
+-------------------+--------------------+--------------------+----------------------+----------------+
| POC design#1|  70.7095 ms    |  80.9805 ms    | 102.773  ms       |
 N/A          |
+-------------------+--------------------+--------------------+----------------------+----------------+
| POC design#2|  70.858 ms      |  83.0845 ms    | 112.505 ms        |
  N/A          |
+-------------------+--------------------+--------------------+----------------------+----------------+

Performance results for 100 empty tables:
+-------------------+--------------------+--------------------+----------------------+----------------+
|                         | 2 sync workers | 4 sync workers | 8 sync
workers | 16 sync workers|
+-------------------+--------------------+--------------------+----------------------+----------------+
| HEAD              | 1039.89 ms      | 860.88 ms       | 1112.312 ms
    |  1122.52 ms    |
+-------------------+--------------------+--------------------+----------------------+----------------+
| POC design#1|  310.920 ms    | 293.14 ms       |  385.698  ms      |
  456.64 ms    |
+-------------------+--------------------+--------------------+----------------------+----------------+
| POC design#2 |  318.464 ms    | 313.98 ms      |  352.316 ms       |
  441.53 ms    |
+-------------------+--------------------+--------------------+----------------------+----------------+

Performance results for 1000 empty tables:
+-------------------+--------------------+--------------------+----------------------+----------------+
|                         | 2 sync workers | 4 sync workers | 8 sync
workers | 16 sync workers|
+------------------+---------------------+---------------------+---------------------+----------------+
| HEAD              | 16327.96 ms    | 10253.65 ms    | 9741.986 ms
|  10278.98 ms   |
+-------------------+--------------------+---------------------+---------------------+----------------+
| POC design#1|  3598.21 ms    |  3099.54 ms     | 2944.386 ms     |
2588.20 ms   |
+-------------------+--------------------+---------------------+---------------------+----------------+
| POC design#2|  4131.72 ms    |  2840.36 ms     | 3001.159 ms    |
5461.82 ms   |
+-------------------+--------------------+---------------------+--------------------+----------------+

Performance results for 2000 empty tables:
+-------------------+--------------------+--------------------+----------------------+----------------+
|                         | 2 sync workers | 4 sync workers | 8 sync
workers | 16 sync workers|
+-------------------+--------------------+--------------------+----------------------+----------------+
| HEAD              | 47210.92 ms    | 25239.90 ms   | 19171.48 ms
|   19556.46 ms  |
+-------------------+--------------------+--------------------+---------------------+----------------+
| POC design#1| 10598.32 ms   |  6995.61 ms    |  6507.53 ms      |
5295.72 ms  |
+-------------------+--------------------+--------------------+-------------------------------------+
| POC design#2| 11121.00 ms   |  6659.74 ms    |  6253.66 ms      |
15433.81 ms  |
+-------------------+--------------------+--------------------+-------------------------------------+

The performance result execution for the same is attached in
Perftest_Results.xlsx.
Also testing with a) table having data and b) apply worker applying
changes while table sync in progress is not done. One of us will do
and try to share the results for these too.
It is noticed that performance of POC design #1 and POC design #2 are
good but POC design #2's performance degrades when there are a greater
number of workers and more tables. In POC design #2, when there are a
greater number of workers and tables, apply worker is becoming a
bottleneck as it must allocate work for all the workers.
Based on the test results, POC design #1 is better.

Thanks to Kuroda-san for helping me in running the performance tests.

[1] - https://www.postgresql.org/message-id/CAGPVpCSk4v-V1WbFDy8a5dL7Es5z8da6hoQbuVyrqP5s3Yh6Cg%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/CAHut%2BPs8gWP9tCPK9gdMnxyshRKgVP3pJnAnaJto_T07uR9xUA%40mail.gmail.com

Regards,
Vignesh

  107. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-19T03:07:58Z

    Some review comments for v19-0001

======
src/backend/replication/logical/tablesync.c

1. run_tablesync_worker
+run_tablesync_worker(WalRcvStreamOptions *options,
+ char *slotname,
+ char *originname,
+ int originname_size,
+ XLogRecPtr *origin_startpos)
+{
+ /* Start table synchronization. */
+ start_table_sync(origin_startpos, &slotname);

There was no such comment ("/* Start table synchronization. */") in
the original HEAD code, so I didn't see that it adds much value by
adding it in the refactored code.

~~~

2. LogicalRepSyncTableStart

/*
* Finally, wait until the leader apply worker tells us to catch up and
* then return to let LogicalRepApplyLoop do it.
*/
wait_for_worker_state_change(SUBREL_STATE_CATCHUP);

~

Should LogicalRepApplyLoop still be mentioned here, since that is
static in worker.c? Maybe it is better to refer instead to the common
'start_apply' wrapper? (see also #5a below)

======
src/backend/replication/logical/worker.c

3. set_stream_options

+/*
+ * Sets streaming options including replication slot name and origin start
+ * position. Workers need these options for logical replication.
+ */
+void
+set_stream_options(WalRcvStreamOptions *options,

I'm not sure if the last sentence of the comment is adding anything useful.

~~~

4. start_apply
/*
 * Run the apply loop with error handling. Disable the subscription,
 * if necessary.
 *
 * Note that we don't handle FATAL errors which are probably because
 * of system resource error and are not repeatable.
 */
void
start_apply(XLogRecPtr origin_startpos)

~

4a.
Somehow I found the function names to be confusing. Intuitively (IMO)
'start_apply' is for apply worker and 'start_tablesync' is for
tablesync worker. But actually, the start_apply() function is the
*common* function for both kinds of worker. Might be easier to
understand if start_apply function name can be changed to indicate it
is really common -- e.g. common_apply_loop(), or similar.

~

4b.
If adverse to changing the function name, it might be helpful anyway
if the function comment can emphasize this function is shared by
different worker types. e.g. "Common function to run the apply
loop..."

~~~

5. run_apply_worker

+ ReplicationOriginNameForLogicalRep(MySubscription->oid, InvalidOid,
+    originname, originname_size);
+
+ /* Setup replication origin tracking. */
+ StartTransactionCommand();

Even if you wish ReplicationOriginNameForLogicalRep() to be outside of
the transaction I thought it should still come *after* the comment,
same as it does in the HEAD code.

~~~

6. ApplyWorkerMain

- /* Run the main loop. */
- start_apply(origin_startpos);
+ /* This is leader apply worker */
+ run_apply_worker(&options, myslotname, originname,
sizeof(originname), &origin_startpos);

  proc_exit(0);
 }

~

6a.
The comment "/* This is leader apply worker */" is redundant now. This
function is the entry point for leader apply workers so it can't be
anything else.

~

6b.

Caller parameter wrapping differs from the similar code in
TablesyncWorkerMain. Shouldn't they be similar?

e.g.
+ run_apply_worker(&options, myslotname, originname,
sizeof(originname), &origin_startpos);

versus
+ run_tablesync_worker(&options,
+ myslotname,
+ originname,
+ sizeof(originname),
+ &origin_startpos);

======
src/include/replication/worker_internal.h

7.
+
+extern void set_stream_options(WalRcvStreamOptions *options,
+    char *slotname,
+    XLogRecPtr *origin_startpos);
+extern void start_apply(XLogRecPtr origin_startpos);
+extern void DisableSubscriptionAndExit(void);
+

Maybe all the externs belong together? It doesn't seem right for just
these 3 externs to be separated from all the others, with those static
inline functions in-between.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  108. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-19T03:54:49Z

    On Wed, Jul 19, 2023 at 8:38 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Some review comments for v19-0001
>
...
> ======
> src/backend/replication/logical/worker.c
>
> 3. set_stream_options
>
> +/*
> + * Sets streaming options including replication slot name and origin start
> + * position. Workers need these options for logical replication.
> + */
> +void
> +set_stream_options(WalRcvStreamOptions *options,
>
> I'm not sure if the last sentence of the comment is adding anything useful.
>

Personally, I find it useful as at a high-level it tells the purpose
of setting these options.

> ~~~
>
> 4. start_apply
> /*
>  * Run the apply loop with error handling. Disable the subscription,
>  * if necessary.
>  *
>  * Note that we don't handle FATAL errors which are probably because
>  * of system resource error and are not repeatable.
>  */
> void
> start_apply(XLogRecPtr origin_startpos)
>
> ~
>
> 4a.
> Somehow I found the function names to be confusing. Intuitively (IMO)
> 'start_apply' is for apply worker and 'start_tablesync' is for
> tablesync worker. But actually, the start_apply() function is the
> *common* function for both kinds of worker. Might be easier to
> understand if start_apply function name can be changed to indicate it
> is really common -- e.g. common_apply_loop(), or similar.
>
> ~
>
> 4b.
> If adverse to changing the function name, it might be helpful anyway
> if the function comment can emphasize this function is shared by
> different worker types. e.g. "Common function to run the apply
> loop..."
>

I would prefer to change the comments as suggested by you in 4b
because both the workers (apply and tablesync) need to perform apply,
so it seems logical for both of them to invoke start_apply.

-- 
With Regards,
Amit Kapila.

  109. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-20T02:32:06Z

    On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's reviews for 0001 and 0002 with some small comments below.
>
> Peter Smith <smithpb2250@gmail.com>, 10 Tem 2023 Pzt, 10:09 tarihinde şunu yazdı:
>>
>> 6. LogicalRepApplyLoop
>>
>> + /*
>> + * apply_dispatch() may have gone into apply_handle_commit()
>> + * which can call process_syncing_tables_for_sync.
>> + *
>> + * process_syncing_tables_for_sync decides whether the sync of
>> + * the current table is completed. If it is completed,
>> + * streaming must be already ended. So, we can break the loop.
>> + */
>> + if (MyLogicalRepWorker->is_sync_completed)
>> + {
>> + endofstream = true;
>> + break;
>> + }
>> +
>>
>> and
>>
>> + /*
>> + * If is_sync_completed is true, this means that the tablesync
>> + * worker is done with synchronization. Streaming has already been
>> + * ended by process_syncing_tables_for_sync. We should move to the
>> + * next table if needed, or exit.
>> + */
>> + if (MyLogicalRepWorker->is_sync_completed)
>> + endofstream = true;
>>
>> ~
>>
>> Instead of those code fragments above assigning 'endofstream' as a
>> side-effect, would it be the same (but tidier) to just modify the
>> other "breaking" condition below:
>>
>> BEFORE:
>> /* Check if we need to exit the streaming loop. */
>> if (endofstream)
>> break;
>>
>> AFTER:
>> /* Check if we need to exit the streaming loop. */
>> if (endofstream || MyLogicalRepWorker->is_sync_completed)
>> break;
>
>
> First place you mentioned also breaks the infinite loop. Such an if statement is needed there with or without endofstream assignment.
>
> I think if there is a flag to break a loop, using that flag to indicate that we should exit the loop seems more appropriate to me. I see that it would be a bit tidier without endofstream = true lines, but I feel like it would also be less readable.
>
> I don't have a strong opinion though. I'm just keeping them as they are for now, but I can change them if you disagree.
>

I felt it was slightly sneaky to re-use the existing variable as a
convenient way to do what you want. But, I don’t feel strongly enough
on this point to debate it -- maybe see later if others have an
opinion about this.

>>
>>
>> 10b.
>> All the other tablesync-related fields of this struct are named as
>> relXXX, so I wonder if is better for this to follow the same pattern.
>> e.g. 'relsync_completed'
>
>
> Aren't those start with rel because they're related to the relation that the tablesync worker is syncing? is_sync_completed is not a relation specific field. I'm okay with changing the name but feel like relsync_completed would be misleading.

My reading of the code is slightly different: Only these fields have
the prefix ‘rel’ and they are all grouped under the comment “/* Used
for initial table synchronization. */” because AFAIK only these fields
are TWS specific (not used for other kinds of workers).

Since this new flag field is also TWS-specific, therefore IMO it
should follow the same consistent name pattern. But, if you are
unconvinced, maybe see later if others have an opinion about it.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  110. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-20T02:41:17Z

    Some review comments for patch v20-0002

======
src/backend/replication/logical/tablesync.c

1. finish_sync_worker
/*
 * Exit routine for synchronization worker.
 *
 * If reuse_worker is false, the worker will not be reused and exit.
 */

~

IMO the "will not be reused" part doesn't need saying -- it is
self-evident from the fact "reuse_worker is false".

SUGGESTION
If reuse_worker is false, at the conclusion of this function the
worker process will exit.

~~~

2. finish_sync_worker

- StartTransactionCommand();
- ereport(LOG,
- (errmsg("logical replication table synchronization worker for
subscription \"%s\", table \"%s\" has finished",
- MySubscription->name,
- get_rel_name(MyLogicalRepWorker->relid))));
- CommitTransactionCommand();
-
  /* Find the leader apply worker and signal it. */
  logicalrep_worker_wakeup(MyLogicalRepWorker->subid, InvalidOid);

- /* Stop gracefully */
- proc_exit(0);
+ if (!reuse_worker)
+ {
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" has finished",
+ MySubscription->name)));
+ CommitTransactionCommand();
+
+ /* Stop gracefully */
+ proc_exit(0);
+ }

In the HEAD code the log message came *before* it signalled to the
apply leader. Won't it be better to keep the logic in that same order?

~~~

3. process_syncing_tables_for_sync

- finish_sync_worker();
+ /* Sync worker has completed synchronization of the current table. */
+ MyLogicalRepWorker->is_sync_completed = true;
+
+ ereport(LOG,
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\", relation \"%s\" with relid %u has finished",
+ MySubscription->name,
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));
+ CommitTransactionCommand();

IIUC it is only the " table synchronization" part that is finished
here; not the whole "table synchronization worker" (compared to
finish_sync_worker function), so maybe the word "worker"  should not
be in this message.

~~~

4. TablesyncWorkerMain

+ if (MyLogicalRepWorker->is_sync_completed)
+ {
+ /* tablesync is done unless a table that needs syncning is found */
+ done = true;

SUGGESTION (Typo "syncning" and minor rewording.)
This tablesync worker is 'done' unless another table that needs
syncing is found.

~

5.
+ /* Found a table for next iteration */
+ finish_sync_worker(true);
+
+ StartTransactionCommand();
+ ereport(LOG,
+ (errmsg("logical replication worker for subscription \"%s\" will be
reused to sync table \"%s\" with relid %u.",
+ MySubscription->name,
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));
+ CommitTransactionCommand();
+
+ done = false;
+ break;
+ }
+ LWLockRelease(LogicalRepWorkerLock);

5a.
IMO it seems better to put this ereport *inside* the
finish_sync_worker() function alongside the similar log for when the
worker is not reused.

~

5b.
Isn't there a missing call to that LWLockRelease, if the 'break' happens?

======
src/backend/replication/logical/worker.c

6. LogicalRepApplyLoop

Refer to [1] for my reply to a previous review comment

~~~

7. InitializeLogRepWorker

  if (am_tablesync_worker())
  ereport(LOG,
- (errmsg("logical replication worker for subscription \"%s\", table
\"%s\" has started",
+ (errmsg("logical replication worker for subscription \"%s\", table
\"%s\" with relid %u has started",
  MySubscription->name,
- get_rel_name(MyLogicalRepWorker->relid))));
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));

But this is certainly a tablesync worker so the message here should
say "logical replication table synchronization worker" like the HEAD
code used to do.

It seems this mistake was introduced in patch v20-0001.

======
src/include/replication/worker_internal.h

8.
Refer to [1] for my reply to a previous review comment

------
[1] Replies to previous 0002 comments --
https://www.postgresql.org/message-id/CAHut%2BPtiAtGJC52SGNdobOah5ctYDDhWWKd%3DuP%3DrkRgXzg5rdg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  111. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-20T03:59:41Z

    On Thu, Jul 20, 2023 at 8:02 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jul 18, 2023 at 1:54 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> >
> > Hi,
> >
> > PFA updated patches. Rebased 0003 with minor changes. Addressed Peter's reviews for 0001 and 0002 with some small comments below.
> >
> > Peter Smith <smithpb2250@gmail.com>, 10 Tem 2023 Pzt, 10:09 tarihinde şunu yazdı:
> >>
> >> 6. LogicalRepApplyLoop
> >>
> >> + /*
> >> + * apply_dispatch() may have gone into apply_handle_commit()
> >> + * which can call process_syncing_tables_for_sync.
> >> + *
> >> + * process_syncing_tables_for_sync decides whether the sync of
> >> + * the current table is completed. If it is completed,
> >> + * streaming must be already ended. So, we can break the loop.
> >> + */
> >> + if (MyLogicalRepWorker->is_sync_completed)
> >> + {
> >> + endofstream = true;
> >> + break;
> >> + }
> >> +
> >>
> >> and
> >>
> >> + /*
> >> + * If is_sync_completed is true, this means that the tablesync
> >> + * worker is done with synchronization. Streaming has already been
> >> + * ended by process_syncing_tables_for_sync. We should move to the
> >> + * next table if needed, or exit.
> >> + */
> >> + if (MyLogicalRepWorker->is_sync_completed)
> >> + endofstream = true;
> >>
> >> ~
> >>
> >> Instead of those code fragments above assigning 'endofstream' as a
> >> side-effect, would it be the same (but tidier) to just modify the
> >> other "breaking" condition below:
> >>
> >> BEFORE:
> >> /* Check if we need to exit the streaming loop. */
> >> if (endofstream)
> >> break;
> >>
> >> AFTER:
> >> /* Check if we need to exit the streaming loop. */
> >> if (endofstream || MyLogicalRepWorker->is_sync_completed)
> >> break;
> >
> >
> > First place you mentioned also breaks the infinite loop. Such an if statement is needed there with or without endofstream assignment.
> >
> > I think if there is a flag to break a loop, using that flag to indicate that we should exit the loop seems more appropriate to me. I see that it would be a bit tidier without endofstream = true lines, but I feel like it would also be less readable.
> >
> > I don't have a strong opinion though. I'm just keeping them as they are for now, but I can change them if you disagree.
> >
>
> I felt it was slightly sneaky to re-use the existing variable as a
> convenient way to do what you want. But, I don’t feel strongly enough
> on this point to debate it -- maybe see later if others have an
> opinion about this.
>

I feel it is okay to use the existing variable 'endofstream' here but
shall we have an assertion that it is a tablesync worker?

> >>
> >>
> >> 10b.
> >> All the other tablesync-related fields of this struct are named as
> >> relXXX, so I wonder if is better for this to follow the same pattern.
> >> e.g. 'relsync_completed'
> >
> >
> > Aren't those start with rel because they're related to the relation that the tablesync worker is syncing? is_sync_completed is not a relation specific field. I'm okay with changing the name but feel like relsync_completed would be misleading.
>
> My reading of the code is slightly different: Only these fields have
> the prefix ‘rel’ and they are all grouped under the comment “/* Used
> for initial table synchronization. */” because AFAIK only these fields
> are TWS specific (not used for other kinds of workers).
>
> Since this new flag field is also TWS-specific, therefore IMO it
> should follow the same consistent name pattern. But, if you are
> unconvinced, maybe see later if others have an opinion about it.
>

+1 to use the prefix 'rel' here as the sync is specific to the
relation. Even during apply phase, we will apply the relation-specific
changes. See should_apply_changes_for_rel().

-- 
With Regards,
Amit Kapila.

  112. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-20T04:10:24Z

    Hi, I had a look at the latest 00003 patch (v20-0003).

Although this patch was recently modified, the updates are mostly only
to make it compatible with the updated v20-0002 patch. Specifically,
the v20-0003 updates did not yet address my review comments from
v17-0003 [1].

Anyway, this post is just a reminder so the earlier review doesn't get
forgotten.

------
[1] v17-0003 review -
https://www.postgresql.org/message-id/CAHut%2BPuMAiO_X_Kw6ud-jr5WOm%2Brpkdu7CppDU6mu%3DgY7UVMzQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  113. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-20T08:38:02Z

    Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 20 Tem 2023 Per, 07:10 tarihinde şunu
yazdı:

> Hi, I had a look at the latest 00003 patch (v20-0003).
>
> Although this patch was recently modified, the updates are mostly only
> to make it compatible with the updated v20-0002 patch. Specifically,
> the v20-0003 updates did not yet address my review comments from
> v17-0003 [1].
>

Yes, I only addressed your reviews for 0001 and 0002, and rebased 0003 in
latest patches as stated here [1].

I'll update the patch soon according to recent reviews, including yours for
0003.


[1]
https://www.postgresql.org/message-id/CAGPVpCTvALKEXe0%3DN-%2BiMmVxVQ-%2BP8KZ_1qQ1KsSSZ-V9wJ5hw%40mail.gmail.com

Thanks for the reminder.
-- 
Melih Mutlu
Microsoft

  114. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-20T11:42:29Z

    Hi,

Peter Smith <smithpb2250@gmail.com>, 20 Tem 2023 Per, 05:41 tarihinde şunu
yazdı:

> 7. InitializeLogRepWorker
>
>   if (am_tablesync_worker())
>   ereport(LOG,
> - (errmsg("logical replication worker for subscription \"%s\", table
> \"%s\" has started",
> + (errmsg("logical replication worker for subscription \"%s\", table
> \"%s\" with relid %u has started",
>   MySubscription->name,
> - get_rel_name(MyLogicalRepWorker->relid))));
> + get_rel_name(MyLogicalRepWorker->relid),
> + MyLogicalRepWorker->relid)));
>
> But this is certainly a tablesync worker so the message here should
> say "logical replication table synchronization worker" like the HEAD
> code used to do.
>
> It seems this mistake was introduced in patch v20-0001.
>

I'm a bit confused here. Isn't it decided to use "logical replication
worker" regardless of the worker's type [1]. That's why I made this change.
If that's not the case here, I'll put it back.

[1]
https://www.postgresql.org/message-id/flat/CAHut%2BPt1xwATviPGjjtJy5L631SGf3qjV9XUCmxLu16cHamfgg%40mail.gmail.com

Thanks,
-- 
Melih Mutlu
Microsoft

  115. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-20T12:37:59Z

    On Thu, Jul 20, 2023 at 5:12 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Peter Smith <smithpb2250@gmail.com>, 20 Tem 2023 Per, 05:41 tarihinde şunu yazdı:
>>
>> 7. InitializeLogRepWorker
>>
>>   if (am_tablesync_worker())
>>   ereport(LOG,
>> - (errmsg("logical replication worker for subscription \"%s\", table
>> \"%s\" has started",
>> + (errmsg("logical replication worker for subscription \"%s\", table
>> \"%s\" with relid %u has started",
>>   MySubscription->name,
>> - get_rel_name(MyLogicalRepWorker->relid))));
>> + get_rel_name(MyLogicalRepWorker->relid),
>> + MyLogicalRepWorker->relid)));
>>
>> But this is certainly a tablesync worker so the message here should
>> say "logical replication table synchronization worker" like the HEAD
>> code used to do.
>>
>> It seems this mistake was introduced in patch v20-0001.
>
>
> I'm a bit confused here. Isn't it decided to use "logical replication worker" regardless of the worker's type [1]. That's why I made this change. If that's not the case here, I'll put it back.
>

I feel where the worker type is clear, it is better to use it unless
the same can lead to translation issues.

-- 
With Regards,
Amit Kapila.

  116. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-20T13:40:47Z

    Hi,

Attached the updated patches with recent reviews addressed.

See below for my comments:

Peter Smith <smithpb2250@gmail.com>, 19 Tem 2023 Çar, 06:08 tarihinde şunu
yazdı:

> Some review comments for v19-0001
>
> 2. LogicalRepSyncTableStart
>
> /*
> * Finally, wait until the leader apply worker tells us to catch up and
> * then return to let LogicalRepApplyLoop do it.
> */
> wait_for_worker_state_change(SUBREL_STATE_CATCHUP);
>
> ~
>
> Should LogicalRepApplyLoop still be mentioned here, since that is
> static in worker.c? Maybe it is better to refer instead to the common
> 'start_apply' wrapper? (see also #5a below)


Isn't' LogicalRepApplyLoop static on HEAD and also mentioned in the exact
comment in tablesync.c while the common "start_apply" function also exists?
I'm not sure how such a change would be related to this patch.

---

5.
> + /* Found a table for next iteration */
> + finish_sync_worker(true);
> +
> + StartTransactionCommand();
> + ereport(LOG,
> + (errmsg("logical replication worker for subscription \"%s\" will be
> reused to sync table \"%s\" with relid %u.",
> + MySubscription->name,
> + get_rel_name(MyLogicalRepWorker->relid),
> + MyLogicalRepWorker->relid)));
> + CommitTransactionCommand();
> +
> + done = false;
> + break;
> + }
> + LWLockRelease(LogicalRepWorkerLock);


> 5b.
> Isn't there a missing call to that LWLockRelease, if the 'break' happens?


Lock is already released before break, if that's the lock you meant:

/* Update worker state for the next table */
> MyLogicalRepWorker->relid = rstate->relid;
> MyLogicalRepWorker->relstate = rstate->state;
> MyLogicalRepWorker->relstate_lsn = rstate->lsn;
> LWLockRelease(LogicalRepWorkerLock);


> /* Found a table for next iteration */
> finish_sync_worker(true);
> done = false;
> break;


---

2.
> As for the publisher node, this patch allows to reuse logical
> walsender processes
> after the streaming is done once.


> ~


> Is this paragraph even needed? Since the connection is reused then it
> already implies the other end (the Wlasender) is being reused, right?


I actually see no harm in explaining this explicitly.


Thanks,
-- 
Melih Mutlu
Microsoft

  117. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-21T01:48:48Z

    On Thu, Jul 20, 2023 at 11:41 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> Attached the updated patches with recent reviews addressed.
>
> See below for my comments:
>
> Peter Smith <smithpb2250@gmail.com>, 19 Tem 2023 Çar, 06:08 tarihinde şunu yazdı:
>>
>> Some review comments for v19-0001
>>
>> 2. LogicalRepSyncTableStart
>>
>> /*
>> * Finally, wait until the leader apply worker tells us to catch up and
>> * then return to let LogicalRepApplyLoop do it.
>> */
>> wait_for_worker_state_change(SUBREL_STATE_CATCHUP);
>>
>> ~
>>
>> Should LogicalRepApplyLoop still be mentioned here, since that is
>> static in worker.c? Maybe it is better to refer instead to the common
>> 'start_apply' wrapper? (see also #5a below)
>
>
> Isn't' LogicalRepApplyLoop static on HEAD and also mentioned in the exact comment in tablesync.c while the common "start_apply" function also exists? I'm not sure how such a change would be related to this patch.
>

Fair enough. I thought it was questionable for one module to refer to
another module's static functions, but you are correct - it is not
really related to your patch. Sorry for the noise.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  118. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-21T02:00:01Z

    Some review comments for v21-0001

======
src/backend/replication/logical/worker.c

1. InitializeLogRepWorker

  if (am_tablesync_worker())
  ereport(LOG,
- (errmsg("logical replication table synchronization worker for
subscription \"%s\", table \"%s\" has started",
+ (errmsg("logical replication worker for subscription \"%s\", table
\"%s\" has started",
  MySubscription->name,
  get_rel_name(MyLogicalRepWorker->relid))));

I think this should not be changed. IIUC that decision for using the
generic worker name for translations was only when the errmsg was in
shared code where the worker type was not clear from existing
conditions. See also previous review comments [1].

~~~

2. StartLogRepWorker

/* Common function to start the leader apply or tablesync worker. */
void
StartLogRepWorker(int worker_slot)
{
/* Attach to slot */
logicalrep_worker_attach(worker_slot);

/* Setup signal handling */
pqsignal(SIGHUP, SignalHandlerForConfigReload);
pqsignal(SIGTERM, die);
BackgroundWorkerUnblockSignals();

/*
* We don't currently need any ResourceOwner in a walreceiver process, but
* if we did, we could call CreateAuxProcessResourceOwner here.
*/

/* Initialise stats to a sanish value */
MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
MyLogicalRepWorker->reply_time = GetCurrentTimestamp();

/* Load the libpq-specific functions */
load_file("libpqwalreceiver", false);

InitializeLogRepWorker();

/* Connect to the origin and start the replication. */
elog(DEBUG1, "connecting to publisher using connection string \"%s\"",
MySubscription->conninfo);

/*
* Setup callback for syscache so that we know when something changes in
* the subscription relation state.
*/
CacheRegisterSyscacheCallback(SUBSCRIPTIONRELMAP,
  invalidate_syncing_table_states,
  (Datum) 0);
}

~

2a.
The function name seems a bit misleading because it is not really
"starting" anything here - it is just more "initialization" code,
right? Nor is it common to all kinds of LogRepWorker. Maybe the
function could be named something else like 'InitApplyOrSyncWorker()'.
-- see also #2c

~

2b.
Should this have Assert to ensure this is only called from leader
apply or tablesync? -- see also #2c

~

2c.
IMO maybe the best/tidiest way to do this is not to introduce a new
function at all. Instead, just put all this "common init" code into
the existing "common init" function ('InitializeLogRepWorker') and
execute it only if (am_tablesync_worker() || am_leader_apply_worker())
{ }.

======
src/include/replication/worker_internal.h

3.
 extern void pa_xact_finish(ParallelApplyWorkerInfo *winfo,
     XLogRecPtr remote_lsn);
+extern void set_stream_options(WalRcvStreamOptions *options,
+    char *slotname,
+    XLogRecPtr *origin_startpos);
+
+extern void start_apply(XLogRecPtr origin_startpos);
+extern void DisableSubscriptionAndExit(void);
+extern void StartLogRepWorker(int worker_slot);

This placement (esp. with the missing whitespace) seems to be grouping
the set_stream_options with the other 'pa' externs, which are all
under the comment "/* Parallel apply worker setup and interactions
*/".

Putting all these up near the other "extern void
InitializeLogRepWorker(void)" might be less ambiguous.

------
[1] worker name in errmsg -
https://www.postgresql.org/message-id/CAA4eK1%2B%2BwkxxMjsPh-z2aKa9ZjNhKsjv0Tnw%2BTVX-hCBkDHusw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  119. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-21T04:09:15Z

    Some review comments for v21-0002.

On Thu, Jul 20, 2023 at 11:41 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi,
>
> Attached the updated patches with recent reviews addressed.
>
> See below for my comments:
>
> Peter Smith <smithpb2250@gmail.com>, 19 Tem 2023 Çar, 06:08 tarihinde şunu yazdı:
>>
>> 5.
>> + /* Found a table for next iteration */
>> + finish_sync_worker(true);
>> +
>> + StartTransactionCommand();
>> + ereport(LOG,
>> + (errmsg("logical replication worker for subscription \"%s\" will be
>> reused to sync table \"%s\" with relid %u.",
>> + MySubscription->name,
>> + get_rel_name(MyLogicalRepWorker->relid),
>> + MyLogicalRepWorker->relid)));
>> + CommitTransactionCommand();
>> +
>> + done = false;
>> + break;
>> + }
>> + LWLockRelease(LogicalRepWorkerLock);
>>
>>
>> 5b.
>> Isn't there a missing call to that LWLockRelease, if the 'break' happens?
>
>
> Lock is already released before break, if that's the lock you meant:
>
>> /* Update worker state for the next table */
>> MyLogicalRepWorker->relid = rstate->relid;
>> MyLogicalRepWorker->relstate = rstate->state;
>> MyLogicalRepWorker->relstate_lsn = rstate->lsn;
>> LWLockRelease(LogicalRepWorkerLock);
>>
>>
>> /* Found a table for next iteration */
>> finish_sync_worker(true);
>> done = false;
>> break;
>
>

Sorry, I misread the code. You are right.

======
src/backend/replication/logical/tablesync.c

1.
+ if (!reuse_worker)
+ {
+ ereport(LOG,
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" has finished",
+ MySubscription->name)));
+ }
+ else
+ {
+ ereport(LOG,
+ (errmsg("logical replication worker for subscription \"%s\" will be
reused to sync table \"%s\" with relid %u.",
+ MySubscription->name,
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));
+ }

1a.
We know this must be a tablesync worker, so I think that second errmsg
should also be saying "logical replication table synchronization
worker".

~

1b.
Since this is if/else anyway, is it simpler to be positive and say "if
(reuse_worker)" instead of the negative "if (!reuse_worker)"

~~~

2. run_tablesync_worker
 {
+ MyLogicalRepWorker->relsync_completed = false;
+
+ /* Start table synchronization. */
  start_table_sync(origin_startpos, &slotname);
This still contains the added comment that I'd previously posted I
thought was adding anything useful. Also, I didn't think this comment
exists in the HEAD code.
======
src/backend/replication/logical/worker.c

3. LogicalRepApplyLoop

+ /*
+ * apply_dispatch() may have gone into apply_handle_commit()
+ * which can call process_syncing_tables_for_sync.
+ *
+ * process_syncing_tables_for_sync decides whether the sync of
+ * the current table is completed. If it is completed,
+ * streaming must be already ended. So, we can break the loop.
+ */
+ if (am_tablesync_worker() &&
+ MyLogicalRepWorker->relsync_completed)
+ {
+ endofstream = true;
+ break;
+ }
+

Maybe just personal taste, but IMO it is better to rearrange like
below because then there is no reason to read the long comment except
for tablesync workers.

if (am_tablesync_worker())
{
    /*
     * apply_dispatch() may have gone into apply_handle_commit()
     * which can call process_syncing_tables_for_sync.
     *
     * process_syncing_tables_for_sync decides whether the sync of
     * the current table is completed. If it is completed,
     * streaming must be already ended. So, we can break the loop.
    */
    if (MyLogicalRepWorker->relsync_completed)
    {
        endofstream = true;
        break;
    }
}

~~~

4. LogicalRepApplyLoop

+
+ /*
+ * If relsync_completed is true, this means that the tablesync
+ * worker is done with synchronization. Streaming has already been
+ * ended by process_syncing_tables_for_sync. We should move to the
+ * next table if needed, or exit.
+ */
+ if (am_tablesync_worker() &&
+ MyLogicalRepWorker->relsync_completed)
+ endofstream = true;

Ditto the same comment about rearranging the condition, as #3 above.

======
src/include/replication/worker_internal.h

5.
+ /*
+ * Indicates whether tablesync worker has completed syncing its assigned
+ * table.
+ */
+ bool relsync_completed;
+

Isn't it better to arrange this to be adjacent to other relXXX fields,
so they all clearly belong to that "Used for initial table
synchronization." group?

For example, something like:

/* Used for initial table synchronization. */
Oid              relid;
char             relstate;
XLogRecPtr relstate_lsn;
slock_t         relmutex;
bool             relsync_completed; /* has tablesync finished syncing
the assigned table? */

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  120. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-21T05:39:22Z

    On Fri, Jul 21, 2023 at 7:30 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> ~~~
>
> 2. StartLogRepWorker
>
> /* Common function to start the leader apply or tablesync worker. */
> void
> StartLogRepWorker(int worker_slot)
> {
> /* Attach to slot */
> logicalrep_worker_attach(worker_slot);
>
> /* Setup signal handling */
> pqsignal(SIGHUP, SignalHandlerForConfigReload);
> pqsignal(SIGTERM, die);
> BackgroundWorkerUnblockSignals();
>
> /*
> * We don't currently need any ResourceOwner in a walreceiver process, but
> * if we did, we could call CreateAuxProcessResourceOwner here.
> */
>
> /* Initialise stats to a sanish value */
> MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
> MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
>
> /* Load the libpq-specific functions */
> load_file("libpqwalreceiver", false);
>
> InitializeLogRepWorker();
>
> /* Connect to the origin and start the replication. */
> elog(DEBUG1, "connecting to publisher using connection string \"%s\"",
> MySubscription->conninfo);
>
> /*
> * Setup callback for syscache so that we know when something changes in
> * the subscription relation state.
> */
> CacheRegisterSyscacheCallback(SUBSCRIPTIONRELMAP,
>   invalidate_syncing_table_states,
>   (Datum) 0);
> }
>
> ~
>
> 2a.
> The function name seems a bit misleading because it is not really
> "starting" anything here - it is just more "initialization" code,
> right? Nor is it common to all kinds of LogRepWorker. Maybe the
> function could be named something else like 'InitApplyOrSyncWorker()'.
> -- see also #2c
>

How about SetupLogRepWorker? The other thing I noticed is that we
don't seem to be consistent in naming functions in these files. For
example, shall we make all exposed functions follow camel case (like
InitializeLogRepWorker) and static functions follow _ style (like
run_apply_worker) or the other possibility is to use _ style for all
functions except may be the entry functions like ApplyWorkerMain()? I
don't know if there is already a pattern but if not then let's form it
now, so that code looks consistent.

> ~
>
> 2b.
> Should this have Assert to ensure this is only called from leader
> apply or tablesync? -- see also #2c
>
> ~
>
> 2c.
> IMO maybe the best/tidiest way to do this is not to introduce a new
> function at all. Instead, just put all this "common init" code into
> the existing "common init" function ('InitializeLogRepWorker') and
> execute it only if (am_tablesync_worker() || am_leader_apply_worker())
> { }.
>

I don't like 2c much because it will make InitializeLogRepWorker()
have two kinds of initializations.

> ======
> src/include/replication/worker_internal.h
>
> 3.
>  extern void pa_xact_finish(ParallelApplyWorkerInfo *winfo,
>      XLogRecPtr remote_lsn);
> +extern void set_stream_options(WalRcvStreamOptions *options,
> +    char *slotname,
> +    XLogRecPtr *origin_startpos);
> +
> +extern void start_apply(XLogRecPtr origin_startpos);
> +extern void DisableSubscriptionAndExit(void);
> +extern void StartLogRepWorker(int worker_slot);
>
> This placement (esp. with the missing whitespace) seems to be grouping
> the set_stream_options with the other 'pa' externs, which are all
> under the comment "/* Parallel apply worker setup and interactions
> */".
>
> Putting all these up near the other "extern void
> InitializeLogRepWorker(void)" might be less ambiguous.
>

+1. Also, note that they should be in the same order as they are in .c files.

-- 
With Regards,
Amit Kapila.

  121. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-21T06:34:40Z

    On Fri, Jul 21, 2023 at 3:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jul 21, 2023 at 7:30 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > ~~~
> >
> > 2. StartLogRepWorker
> >
> > /* Common function to start the leader apply or tablesync worker. */
> > void
> > StartLogRepWorker(int worker_slot)
> > {
> > /* Attach to slot */
> > logicalrep_worker_attach(worker_slot);
> >
> > /* Setup signal handling */
> > pqsignal(SIGHUP, SignalHandlerForConfigReload);
> > pqsignal(SIGTERM, die);
> > BackgroundWorkerUnblockSignals();
> >
> > /*
> > * We don't currently need any ResourceOwner in a walreceiver process, but
> > * if we did, we could call CreateAuxProcessResourceOwner here.
> > */
> >
> > /* Initialise stats to a sanish value */
> > MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
> > MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
> >
> > /* Load the libpq-specific functions */
> > load_file("libpqwalreceiver", false);
> >
> > InitializeLogRepWorker();
> >
> > /* Connect to the origin and start the replication. */
> > elog(DEBUG1, "connecting to publisher using connection string \"%s\"",
> > MySubscription->conninfo);
> >
> > /*
> > * Setup callback for syscache so that we know when something changes in
> > * the subscription relation state.
> > */
> > CacheRegisterSyscacheCallback(SUBSCRIPTIONRELMAP,
> >   invalidate_syncing_table_states,
> >   (Datum) 0);
> > }
> >
> > ~
> >
> > 2a.
> > The function name seems a bit misleading because it is not really
> > "starting" anything here - it is just more "initialization" code,
> > right? Nor is it common to all kinds of LogRepWorker. Maybe the
> > function could be named something else like 'InitApplyOrSyncWorker()'.
> > -- see also #2c
> >
>
> How about SetupLogRepWorker?

The name is better than StartXXX, but still, SetupXXX seems a synonym
of InitXXX. That is why I thought it is a bit awkward having 2
functions with effectively the same name and the same
initialization/setup purpose (the only difference is one function
excludes parallel workers, and the other function is common to all
workers).

> The other thing I noticed is that we
> don't seem to be consistent in naming functions in these files. For
> example, shall we make all exposed functions follow camel case (like
> InitializeLogRepWorker) and static functions follow _ style (like
> run_apply_worker) or the other possibility is to use _ style for all
> functions except may be the entry functions like ApplyWorkerMain()? I
> don't know if there is already a pattern but if not then let's form it
> now, so that code looks consistent.
>

+1 for using some consistent rule, but I think this may result in
*many* changes, so it would be safer to itemize all the changes first,
just to make sure everybody is OK with it first before updating
everything.

------
Kind Regards,
Peter Smith

  122. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-21T07:24:21Z

    On Fri, Jul 21, 2023 at 12:05 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Jul 21, 2023 at 3:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Jul 21, 2023 at 7:30 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > ~~~
> > >
> > > 2. StartLogRepWorker
> > >
> > > /* Common function to start the leader apply or tablesync worker. */
> > > void
> > > StartLogRepWorker(int worker_slot)
> > > {
> > > /* Attach to slot */
> > > logicalrep_worker_attach(worker_slot);
> > >
> > > /* Setup signal handling */
> > > pqsignal(SIGHUP, SignalHandlerForConfigReload);
> > > pqsignal(SIGTERM, die);
> > > BackgroundWorkerUnblockSignals();
> > >
> > > /*
> > > * We don't currently need any ResourceOwner in a walreceiver process, but
> > > * if we did, we could call CreateAuxProcessResourceOwner here.
> > > */
> > >
> > > /* Initialise stats to a sanish value */
> > > MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
> > > MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
> > >
> > > /* Load the libpq-specific functions */
> > > load_file("libpqwalreceiver", false);
> > >
> > > InitializeLogRepWorker();
> > >
> > > /* Connect to the origin and start the replication. */
> > > elog(DEBUG1, "connecting to publisher using connection string \"%s\"",
> > > MySubscription->conninfo);
> > >
> > > /*
> > > * Setup callback for syscache so that we know when something changes in
> > > * the subscription relation state.
> > > */
> > > CacheRegisterSyscacheCallback(SUBSCRIPTIONRELMAP,
> > >   invalidate_syncing_table_states,
> > >   (Datum) 0);
> > > }
> > >
> > > ~
> > >
> > > 2a.
> > > The function name seems a bit misleading because it is not really
> > > "starting" anything here - it is just more "initialization" code,
> > > right? Nor is it common to all kinds of LogRepWorker. Maybe the
> > > function could be named something else like 'InitApplyOrSyncWorker()'.
> > > -- see also #2c
> > >
> >
> > How about SetupLogRepWorker?
>
> The name is better than StartXXX, but still, SetupXXX seems a synonym
> of InitXXX. That is why I thought it is a bit awkward having 2
> functions with effectively the same name and the same
> initialization/setup purpose (the only difference is one function
> excludes parallel workers, and the other function is common to all
> workers).
>

I can't know of a better way. We can probably name it as
SetupApplyOrSyncWorker or something like that if you find that better.

> > The other thing I noticed is that we
> > don't seem to be consistent in naming functions in these files. For
> > example, shall we make all exposed functions follow camel case (like
> > InitializeLogRepWorker) and static functions follow _ style (like
> > run_apply_worker) or the other possibility is to use _ style for all
> > functions except may be the entry functions like ApplyWorkerMain()? I
> > don't know if there is already a pattern but if not then let's form it
> > now, so that code looks consistent.
> >
>
> +1 for using some consistent rule, but I think this may result in
> *many* changes, so it would be safer to itemize all the changes first,
> just to make sure everybody is OK with it first before updating
> everything.
>

Fair enough. We can do that as a first patch and then work on the
refactoring patch to avoid introducing more inconsistencies or we can
do the refactoring patch first but keep all the new function names to
follow _ style.

Apart from this, few more comments on 0001:
1.
+run_apply_worker(WalRcvStreamOptions *options,
+ char *slotname,
+ char *originname,
+ int originname_size,
+ XLogRecPtr *origin_startpos)

The caller neither uses nor passes the value of origin_startpos. So,
isn't it better to make origin_startpos local to run_apply_worker()?
It seems the same is true for some of the other parameters slotname,
originname, originname_size. Is there a reason to keep these as
arguments in this function?

2.
+static void
+run_tablesync_worker(WalRcvStreamOptions *options,
+ char *slotname,
+ char *originname,
+ int originname_size,
+ XLogRecPtr *origin_startpos)

The comments in the previous point seem to apply to this as well.

3.
+ set_stream_options(options, slotname, origin_startpos);
+
+ walrcv_startstreaming(LogRepWorkerWalRcvConn, options);
+
+ if (MySubscription->twophasestate == LOGICALREP_TWOPHASE_STATE_PENDING &&
+ AllTablesyncsReady())

This last check is done in set_stream_options() and here as well. I
don't see any reason to give different answers at both places but
before the patch, we were not relying on any such assumption that this
check will always give the same answer considering the answer could be
different due to AllTablesyncsReady(). Can we move this check outside
set_stream_options()?

-- 
With Regards,
Amit Kapila.

  123. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-21T09:47:46Z

    Amit Kapila <amit.kapila16@gmail.com>, 21 Tem 2023 Cum, 08:39 tarihinde
şunu yazdı:

> On Fri, Jul 21, 2023 at 7:30 AM Peter Smith <smithpb2250@gmail.com> wrote:
> How about SetupLogRepWorker? The other thing I noticed is that we
> don't seem to be consistent in naming functions in these files. For
> example, shall we make all exposed functions follow camel case (like
> InitializeLogRepWorker) and static functions follow _ style (like
> run_apply_worker) or the other possibility is to use _ style for all
> functions except may be the entry functions like ApplyWorkerMain()? I
> don't know if there is already a pattern but if not then let's form it
> now, so that code looks consistent.
>

I agree that these files have inconsistencies in naming things.
Most of the time I can't really figure out which naming convention I should
use. I try to name things by looking at other functions with similar
responsibilities.


> 3.
> >  extern void pa_xact_finish(ParallelApplyWorkerInfo *winfo,
> >      XLogRecPtr remote_lsn);
> > +extern void set_stream_options(WalRcvStreamOptions *options,
> > +    char *slotname,
> > +    XLogRecPtr *origin_startpos);
> > +
> > +extern void start_apply(XLogRecPtr origin_startpos);
> > +extern void DisableSubscriptionAndExit(void);
> > +extern void StartLogRepWorker(int worker_slot);
> >
> > This placement (esp. with the missing whitespace) seems to be grouping
> > the set_stream_options with the other 'pa' externs, which are all
> > under the comment "/* Parallel apply worker setup and interactions
> > */".
> >
> > Putting all these up near the other "extern void
> > InitializeLogRepWorker(void)" might be less ambiguous.
> >
>
> +1. Also, note that they should be in the same order as they are in .c
> files.
>

I did not realize the order is the same with .c files. Good to know. I'll
fix it along with other comments.

Thanks,
-- 
Melih Mutlu
Microsoft

  124. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-21T09:48:07Z

    On Fri, Jul 21, 2023 at 5:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jul 21, 2023 at 12:05 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Fri, Jul 21, 2023 at 3:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >

> > > The other thing I noticed is that we
> > > don't seem to be consistent in naming functions in these files. For
> > > example, shall we make all exposed functions follow camel case (like
> > > InitializeLogRepWorker) and static functions follow _ style (like
> > > run_apply_worker) or the other possibility is to use _ style for all
> > > functions except may be the entry functions like ApplyWorkerMain()? I
> > > don't know if there is already a pattern but if not then let's form it
> > > now, so that code looks consistent.
> > >
> >
> > +1 for using some consistent rule, but I think this may result in
> > *many* changes, so it would be safer to itemize all the changes first,
> > just to make sure everybody is OK with it first before updating
> > everything.
> >
>
> Fair enough. We can do that as a first patch and then work on the
> refactoring patch to avoid introducing more inconsistencies or we can
> do the refactoring patch first but keep all the new function names to
> follow _ style.
>

Fixing the naming inconsistency will be more far-reaching than just a
few functions affected by these "reuse" patches. There are plenty of
existing functions already inconsistently named in the HEAD code. So
perhaps this topic should be moved to a separate thread?

For example, here are some existing/proposed names:

===

worker.c (HEAD)

static functions
  DisableSubscriptionAndExit -> disable_subscription_and_exit
  FindReplTupleInLocalRel -> find_repl_tuple_in_local_rel
  TwoPhaseTransactionGid -> two_phase_transaction_gid
  TargetPrivilegesCheck -> target_privileges_check
  UpdateWorkerStats -> update_worker_stats
  LogicalRepApplyLoop -> logical_rep_apply_loop

non-static functions
  stream_stop_internal -> StreamStopInternal
  apply_spooled_messages -> ApplySpooledMessages
  apply_dispatch -> ApplyDispatch
  store_flush_position -> StoreFlushPosition
  set_apply_error_context_origin -> SetApplyErrorContextOrigin

===

tablesync.c (HEAD)

static functions
  FetchTableStates -> fetch_table_states

non-static functions
  invalidate_syncing_table_states -> InvalidateSyncingTableStates

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  125. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-21T09:51:56Z

    Peter Smith <smithpb2250@gmail.com>, 21 Tem 2023 Cum, 12:48 tarihinde şunu
yazdı:

> On Fri, Jul 21, 2023 at 5:24 PM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> >
> > On Fri, Jul 21, 2023 at 12:05 PM Peter Smith <smithpb2250@gmail.com>
> wrote:
> > >
> > > On Fri, Jul 21, 2023 at 3:39 PM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > > >
>
> > > > The other thing I noticed is that we
> > > > don't seem to be consistent in naming functions in these files. For
> > > > example, shall we make all exposed functions follow camel case (like
> > > > InitializeLogRepWorker) and static functions follow _ style (like
> > > > run_apply_worker) or the other possibility is to use _ style for all
> > > > functions except may be the entry functions like ApplyWorkerMain()? I
> > > > don't know if there is already a pattern but if not then let's form
> it
> > > > now, so that code looks consistent.
> > > >
> > >
> > > +1 for using some consistent rule, but I think this may result in
> > > *many* changes, so it would be safer to itemize all the changes first,
> > > just to make sure everybody is OK with it first before updating
> > > everything.
> > >
> >
> > Fair enough. We can do that as a first patch and then work on the
> > refactoring patch to avoid introducing more inconsistencies or we can
> > do the refactoring patch first but keep all the new function names to
> > follow _ style.
> >
>
> Fixing the naming inconsistency will be more far-reaching than just a
> few functions affected by these "reuse" patches. There are plenty of
> existing functions already inconsistently named in the HEAD code. So
> perhaps this topic should be moved to a separate thread?
>

+1 for moving it to a separate thread. This is not something particularly
introduced by this patch.

Thanks,
-- 
Melih Mutlu
Microsoft

  126. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-25T14:57:52Z

    Hi,

Melih Mutlu <m.melihmutlu@gmail.com>, 21 Tem 2023 Cum, 12:47 tarihinde şunu
yazdı:

> I did not realize the order is the same with .c files. Good to know. I'll
> fix it along with other comments.
>

Addressed the recent reviews and attached the updated patches.

Thanks,
-- 
Melih Mutlu
Microsoft

  127. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-26T04:40:25Z

    Here are some comments for patch v22-0001.

======
1. General -- naming conventions

There is quite a lot of inconsistency with variable/parameter naming
styles in this patch. I understand in most cases the names are copied
unchanged from the original functions. Still, since this is a big
refactor anyway, it can also be a good opportunity to clean up those
inconsistencies instead of just propagating them to different places.
IIUC, the usual reluctance to rename things because it would cause
backpatch difficulties doesn't apply here (since everything is being
refactored anyway).

E.g. Consider using use snake_case names more consistently in the
following places:

~

1a. start_table_sync

+static void
+start_table_sync(XLogRecPtr *origin_startpos, char **myslotname)
+{
+ char    *syncslotname = NULL;

origin_startpos -> (no change)
myslotname -> my_slot_name (But, is there a better name for this than
calling it "my" slot name)
syncslotname -> sync_slot_name

~

1b. run_tablesync_worker

+static void
+run_tablesync_worker()
+{
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos = InvalidXLogRecPtr;
+ char    *slotname = NULL;
+ WalRcvStreamOptions options;

originname -> origin_name
origin_startpos -> (no change)
slotname -> slot_name

~

1c. set_stream_options

+void
+set_stream_options(WalRcvStreamOptions *options,
+    char *slotname,
+    XLogRecPtr *origin_startpos)
+{
+ int server_version;

options -> (no change)
slotname -> slot_name
origin_startpos -> (no change)
server_version -> (no change)

~

1d. run_apply_worker

 static void
-start_apply(XLogRecPtr origin_startpos)
+run_apply_worker()
 {
- PG_TRY();
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos = InvalidXLogRecPtr;
+ char    *slotname = NULL;
+ WalRcvStreamOptions options;
+ RepOriginId originid;
+ TimeLineID startpointTLI;
+ char    *err;
+ bool must_use_password;

originname -> origin_name
origin_startpos => (no change)
slotname -> slot_name
originid -> origin_id

======
src/backend/replication/logical/worker.c

2. SetupApplyOrSyncWorker

-ApplyWorkerMain(Datum main_arg)
+SetupApplyOrSyncWorker(int worker_slot)
 {
- int worker_slot = DatumGetInt32(main_arg);
- char originname[NAMEDATALEN];
- XLogRecPtr origin_startpos = InvalidXLogRecPtr;
- char    *myslotname = NULL;
- WalRcvStreamOptions options;
- int server_version;
-
- InitializingApplyWorker = true;
-
  /* Attach to slot */
  logicalrep_worker_attach(worker_slot);

+ Assert(am_tablesync_worker() || am_leader_apply_worker());
+

Why is the Assert not the very first statement of this function?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  128. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-26T05:42:41Z

    On Wed, Jul 26, 2023 at 10:10 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some comments for patch v22-0001.
>
> ======
> 1. General -- naming conventions
>
> There is quite a lot of inconsistency with variable/parameter naming
> styles in this patch. I understand in most cases the names are copied
> unchanged from the original functions. Still, since this is a big
> refactor anyway, it can also be a good opportunity to clean up those
> inconsistencies instead of just propagating them to different places.
>

I am not against improving consistency in the naming of existing
variables but I feel it would be better to do as a separate patch
along with improving the consistency function names. For new
functions/variables, it would be good to follow a consistent style.

-- 
With Regards,
Amit Kapila.

  129. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-27T01:13:39Z

    Here are some review comments for v22-0002

======
1. General - errmsg

AFAIK, the errmsg part does not need to be enclosed by extra parentheses.

e.g.
BEFORE
ereport(LOG,
(errmsg("logical replication table synchronization worker for
subscription \"%s\" has finished",
MySubscription->name)));
AFTER
ereport(LOG,
errmsg("logical replication table synchronization worker for
subscription \"%s\" has finished",
MySubscription->name));

~

The patch has multiple cases similar to that example.

======
src/backend/replication/logical/tablesync.c

2.
+ if (reuse_worker)
+ {
+ ereport(LOG,
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" will be reused to sync table \"%s\" with relid
%u.",
+ MySubscription->name,
+ get_rel_name(MyLogicalRepWorker->relid),
+ MyLogicalRepWorker->relid)));
+ }
+ else
+ {
+ ereport(LOG,
+ (errmsg("logical replication table synchronization worker for
subscription \"%s\" has finished",
+ MySubscription->name)));
+ }

These brackets { } are not really necessary.

~~~

3. TablesyncWorkerMain
+ for (;!done;)
+ {
+ List    *rstates;
+ ListCell   *lc;
+
+ run_tablesync_worker();
+
+ if (IsTransactionState())
+ CommitTransactionCommand();
+
+ if (MyLogicalRepWorker->relsync_completed)
+ {
+ /*
+ * This tablesync worker is 'done' unless another table that needs
+ * syncing is found.
+ */
+ done = true;

Those variables 'rstates' and 'lc' do not need to be declared at this
scope -- they can be declared further down, closer to where they are
needed.

=====
src/backend/replication/logical/worker.c

4. LogicalRepApplyLoop
+
+ if (am_tablesync_worker())
+ /*
+ * If relsync_completed is true, this means that the tablesync
+ * worker is done with synchronization. Streaming has already been
+ * ended by process_syncing_tables_for_sync. We should move to the
+ * next table if needed, or exit.
+ */
+ if (MyLogicalRepWorker->relsync_completed)
+ endofstream = true;

Here I think it is better to use bracketing { } for the outer "if",
instead of only relying on the indentation for readability. YMMV.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  130. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-27T01:16:26Z

    Here are some review comments for v22-0003

======

1. ApplicationNameForTablesync
+/*
+ * Determine the application_name for tablesync workers.
+ *
+ * Previously, the replication slot name was used as application_name. Since
+ * it's possible to reuse tablesync workers now, a tablesync worker can handle
+ * several different replication slots during its lifetime. Therefore, we
+ * cannot use the slot name as application_name anymore. Instead, the slot
+ * number of the tablesync worker is used as a part of the application_name.
+ *
+ * FIXME: if the tablesync worker starts to reuse the replication slot during
+ * synchronization, we should again use the replication slot name as
+ * application_name.
+ */
+static void
+ApplicationNameForTablesync(Oid suboid, int worker_slot,
+ char *application_name, Size szapp)
+{
+ snprintf(application_name, szapp, "pg_%u_sync_%i_" UINT64_FORMAT, suboid,
+ worker_slot, GetSystemIdentifier());
+}

1a.
The intent of the "FIXME" comment was not clear. Is this some existing
problem that needs addressing, or is this really more like just an
"XXX" warning/note for the future, in case the tablesync logic
changes?

~

1b.
Since this is a new function, should it be named according to the
convention for static functions?

e.g.
ApplicationNameForTablesync -> app_name_for_tablesync

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  131. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-07-27T05:02:46Z

    On Thu, Jul 27, 2023 at 6:46 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for v22-0003
>
> ======
>
> 1. ApplicationNameForTablesync
> +/*
> + * Determine the application_name for tablesync workers.
> + *
> + * Previously, the replication slot name was used as application_name. Since
> + * it's possible to reuse tablesync workers now, a tablesync worker can handle
> + * several different replication slots during its lifetime. Therefore, we
> + * cannot use the slot name as application_name anymore. Instead, the slot
> + * number of the tablesync worker is used as a part of the application_name.
> + *
> + * FIXME: if the tablesync worker starts to reuse the replication slot during
> + * synchronization, we should again use the replication slot name as
> + * application_name.
> + */
> +static void
> +ApplicationNameForTablesync(Oid suboid, int worker_slot,
> + char *application_name, Size szapp)
> +{
> + snprintf(application_name, szapp, "pg_%u_sync_%i_" UINT64_FORMAT, suboid,
> + worker_slot, GetSystemIdentifier());
> +}
>
> 1a.
> The intent of the "FIXME" comment was not clear. Is this some existing
> problem that needs addressing, or is this really more like just an
> "XXX" warning/note for the future, in case the tablesync logic
> changes?
>

This seems to be a Note for the future, so better to use XXX notation here.

> ~
>
> 1b.
> Since this is a new function, should it be named according to the
> convention for static functions?
>
> e.g.
> ApplicationNameForTablesync -> app_name_for_tablesync
>

I think for now let's follow the style for similar functions like
ReplicationOriginNameForLogicalRep() and
ReplicationSlotNameForTablesync().

-- 
With Regards,
Amit Kapila.

  132. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-07-27T13:29:51Z

    Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 26 Tem 2023 Çar, 07:40 tarihinde şunu
yazdı:

> Here are some comments for patch v22-0001.
>
> ======
> 1. General -- naming conventions
>
> There is quite a lot of inconsistency with variable/parameter naming
> styles in this patch. I understand in most cases the names are copied
> unchanged from the original functions. Still, since this is a big
> refactor anyway, it can also be a good opportunity to clean up those
> inconsistencies instead of just propagating them to different places.
> IIUC, the usual reluctance to rename things because it would cause
> backpatch difficulties doesn't apply here (since everything is being
> refactored anyway).
>
> E.g. Consider using use snake_case names more consistently in the
> following places:
>

I can simply change the places you mentioned, that seems okay to me.
The reason why I did not change the namings in existing variables/functions
is because I did (and still do) not get what's the naming conventions in
those files. Is snake_case the convention for variables in those files (or
in general)?

2. SetupApplyOrSyncWorker
>
> -ApplyWorkerMain(Datum main_arg)
> +SetupApplyOrSyncWorker(int worker_slot)
>  {
> - int worker_slot = DatumGetInt32(main_arg);
> - char originname[NAMEDATALEN];
> - XLogRecPtr origin_startpos = InvalidXLogRecPtr;
> - char    *myslotname = NULL;
> - WalRcvStreamOptions options;
> - int server_version;
> -
> - InitializingApplyWorker = true;
> -
>   /* Attach to slot */
>   logicalrep_worker_attach(worker_slot);
>
> + Assert(am_tablesync_worker() || am_leader_apply_worker());
> +
>
> Why is the Assert not the very first statement of this function?
>

I would also prefer to assert in the very beginning but am_tablesync_worker
and am_leader_apply_worker require MyLogicalRepWorker to be not NULL.
And MyLogicalRepWorker is assigned in logicalrep_worker_attach. I can
change this if you think there is a better way to check the worker type.

Thanks,
-- 
Melih Mutlu
Microsoft

  133. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-27T23:57:06Z

    On Thu, Jul 27, 2023 at 11:30 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Peter,
>
> Peter Smith <smithpb2250@gmail.com>, 26 Tem 2023 Çar, 07:40 tarihinde şunu yazdı:
>>
>> Here are some comments for patch v22-0001.
>>
>> ======
>> 1. General -- naming conventions
>>
>> There is quite a lot of inconsistency with variable/parameter naming
>> styles in this patch. I understand in most cases the names are copied
>> unchanged from the original functions. Still, since this is a big
>> refactor anyway, it can also be a good opportunity to clean up those
>> inconsistencies instead of just propagating them to different places.
>> IIUC, the usual reluctance to rename things because it would cause
>> backpatch difficulties doesn't apply here (since everything is being
>> refactored anyway).
>>
>> E.g. Consider using use snake_case names more consistently in the
>> following places:
>
>
> I can simply change the places you mentioned, that seems okay to me.
> The reason why I did not change the namings in existing variables/functions is because I did (and still do) not get what's the naming conventions in those files. Is snake_case the convention for variables in those files (or in general)?
>

TBH, I also don't know if there is a specific Postgres coding
guideline to use snake_case or not (and Chat-GPT did not know either
when I asked about it). I only assumed snake_case in my previous
review comment because the mentioned vars were already all lowercase.
Anyway, the point was that whatever style is chosen, it ought to be
used *consistently* because having a random mixture of styles in the
same function (e.g. worker_slot, originname, origin_startpos,
myslotname, options, server_version) seems messy. Meanwhile, I think
Amit suggested [1] that for now, we only need to worry about the name
consistency in new code.


>> 2. SetupApplyOrSyncWorker
>>
>> -ApplyWorkerMain(Datum main_arg)
>> +SetupApplyOrSyncWorker(int worker_slot)
>>  {
>> - int worker_slot = DatumGetInt32(main_arg);
>> - char originname[NAMEDATALEN];
>> - XLogRecPtr origin_startpos = InvalidXLogRecPtr;
>> - char    *myslotname = NULL;
>> - WalRcvStreamOptions options;
>> - int server_version;
>> -
>> - InitializingApplyWorker = true;
>> -
>>   /* Attach to slot */
>>   logicalrep_worker_attach(worker_slot);
>>
>> + Assert(am_tablesync_worker() || am_leader_apply_worker());
>> +
>>
>> Why is the Assert not the very first statement of this function?
>
>
> I would also prefer to assert in the very beginning but am_tablesync_worker and am_leader_apply_worker require MyLogicalRepWorker to be not NULL. And MyLogicalRepWorker is assigned in logicalrep_worker_attach. I can change this if you think there is a better way to check the worker type.
>

I see. In that case your Assert LGTM.

------
[1] https://www.postgresql.org/message-id/CAA4eK1%2Bh9hWDAKupsoiw556xqh7uvj_F1pjFJc4jQhL89HdGww%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  134. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-07-28T07:22:11Z

    Hi Melih,

BACKGROUND
----------

We wanted to compare performance for the 2 different reuse-worker
designs, when the apply worker is already busy handling other
replications, and then simultaneously the test table tablesyncs are
occurring.

To test this scenario, some test scripts were written (described
below). For comparisons, the scripts were then run using a build of
HEAD; design #1 (v21); design #2 (0718).

HOW THE TEST WORKS
------------------

Overview:
1. The apply worker is made to subscribe to a 'busy_tbl'.
2. After the SUBSCRIPTION is created, the publisher-side then loops
(forever) doing INSERTS into that busy_tbl.
3. While the apply worker is now busy, the subscriber does an ALTER
SUBSCRIPTION REFRESH PUBLICATION to subscribe to all the other test
tables.
4. We time how long it takes for all tablsyncs to complete
5. Repeat above for different numbers of empty tables (10, 100, 1000,
2000) and different numbers of sync workers (2, 4, 8, 16)

Scripts
-------

(PSA 4 scripts to implement this logic)

testrun script
- this does common setup (do_one_test_setup) and then the pub/sub
scripts (do_one_test_PUB and do_one_test_SUB -- see below) are run in
parallel
- repeat 10 times

do_one_test_setup script
- init and start instances
- ipc setup tables and procedures

do_one_test_PUB script
- ipc setup pub/sub
- table setup
- publishes the "busy_tbl", but then waits for the subscriber to
subscribe to only this one
- alters the publication to include all other tables (so subscriber
will see these only after the ALTER SUBSCRIPTION PUBLICATION REFRESH)
- enter a busy INSERT loop until it informed by the subscriber that
the test is finished

do_one_test_SUB script
- ipc setup pub/sub
- table setup
- subscribes only to "busy_tbl", then informs the publisher when that
is done (this will cause the publisher to commence the stay_busy loop)
- after it knows the publishing busy loop has started it does
- ALTER SUBSCRIPTION REFRESH PUBLICATION
- wait until all the tablesyncs are ready <=== This is the part that
is timed for the test RESULT

PROBLEM
-------

Looking at the output files (e.g. *.dat_PUB and *.dat_SUB)  they seem
to confirm the tests are working how we wanted.

Unfortunately, there is some slot problem for the patched builds (both
designs #1 and #2). e.g. Search "ERROR" in the *.log files and see
many slot-related errors.

Please note - running these same scripts with HEAD build gave no such
errors. So it appears to be a patch problem.

------
Kind Regards
Peter Smith.
Fujitsu Australia

  135. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-01T04:14:02Z

    On Fri, Jul 28, 2023 at 5:22 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Melih,
>
> BACKGROUND
> ----------
>
> We wanted to compare performance for the 2 different reuse-worker
> designs, when the apply worker is already busy handling other
> replications, and then simultaneously the test table tablesyncs are
> occurring.
>
> To test this scenario, some test scripts were written (described
> below). For comparisons, the scripts were then run using a build of
> HEAD; design #1 (v21); design #2 (0718).
>
> HOW THE TEST WORKS
> ------------------
>
> Overview:
> 1. The apply worker is made to subscribe to a 'busy_tbl'.
> 2. After the SUBSCRIPTION is created, the publisher-side then loops
> (forever) doing INSERTS into that busy_tbl.
> 3. While the apply worker is now busy, the subscriber does an ALTER
> SUBSCRIPTION REFRESH PUBLICATION to subscribe to all the other test
> tables.
> 4. We time how long it takes for all tablsyncs to complete
> 5. Repeat above for different numbers of empty tables (10, 100, 1000,
> 2000) and different numbers of sync workers (2, 4, 8, 16)
>
> Scripts
> -------
>
> (PSA 4 scripts to implement this logic)
>
> testrun script
> - this does common setup (do_one_test_setup) and then the pub/sub
> scripts (do_one_test_PUB and do_one_test_SUB -- see below) are run in
> parallel
> - repeat 10 times
>
> do_one_test_setup script
> - init and start instances
> - ipc setup tables and procedures
>
> do_one_test_PUB script
> - ipc setup pub/sub
> - table setup
> - publishes the "busy_tbl", but then waits for the subscriber to
> subscribe to only this one
> - alters the publication to include all other tables (so subscriber
> will see these only after the ALTER SUBSCRIPTION PUBLICATION REFRESH)
> - enter a busy INSERT loop until it informed by the subscriber that
> the test is finished
>
> do_one_test_SUB script
> - ipc setup pub/sub
> - table setup
> - subscribes only to "busy_tbl", then informs the publisher when that
> is done (this will cause the publisher to commence the stay_busy loop)
> - after it knows the publishing busy loop has started it does
> - ALTER SUBSCRIPTION REFRESH PUBLICATION
> - wait until all the tablesyncs are ready <=== This is the part that
> is timed for the test RESULT
>
> PROBLEM
> -------
>
> Looking at the output files (e.g. *.dat_PUB and *.dat_SUB)  they seem
> to confirm the tests are working how we wanted.
>
> Unfortunately, there is some slot problem for the patched builds (both
> designs #1 and #2). e.g. Search "ERROR" in the *.log files and see
> many slot-related errors.
>
> Please note - running these same scripts with HEAD build gave no such
> errors. So it appears to be a patch problem.
>

Hi

FYI, here is some more information about ERRORs seen.

The patches were re-tested -- applied in stages (and also against the
different scripts) to identify where the problem was introduced. Below
are the observations:

~~~

Using original test scripts

1. Using only patch v21-0001
- no errors

2. Using only patch v21-0001+0002
- no errors

3. Using patch v21-0001+0002+0003
- no errors

~~~

Using the "busy loop" test scripts for long transactions

1. Using only patch v21-0001
- no errors

2. Using only patch v21-0001+0002
- gives errors for "no copy in progress issue"
e.g. ERROR:  could not send data to WAL stream: no COPY in progress

3. Using patch v21-0001+0002+0003
- gives the same "no copy in progress issue" errors as above
e.g. ERROR:  could not send data to WAL stream: no COPY in progress
- and also gives slot consistency point errors
e.g. ERROR:  could not create replication slot
"pg_16700_sync_16514_7261998170966054867": ERROR:  could not find
logical decoding starting point
e.g. LOG:  could not drop replication slot
"pg_16700_sync_16454_7261998170966054867" on publisher: ERROR:
replication slot "pg_16700_sync_16454_7261998170966054867" does not
exist

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  136. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-08-01T06:32:29Z

    On Tue, 1 Aug 2023 at 09:44, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Jul 28, 2023 at 5:22 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Melih,
> >
> > BACKGROUND
> > ----------
> >
> > We wanted to compare performance for the 2 different reuse-worker
> > designs, when the apply worker is already busy handling other
> > replications, and then simultaneously the test table tablesyncs are
> > occurring.
> >
> > To test this scenario, some test scripts were written (described
> > below). For comparisons, the scripts were then run using a build of
> > HEAD; design #1 (v21); design #2 (0718).
> >
> > HOW THE TEST WORKS
> > ------------------
> >
> > Overview:
> > 1. The apply worker is made to subscribe to a 'busy_tbl'.
> > 2. After the SUBSCRIPTION is created, the publisher-side then loops
> > (forever) doing INSERTS into that busy_tbl.
> > 3. While the apply worker is now busy, the subscriber does an ALTER
> > SUBSCRIPTION REFRESH PUBLICATION to subscribe to all the other test
> > tables.
> > 4. We time how long it takes for all tablsyncs to complete
> > 5. Repeat above for different numbers of empty tables (10, 100, 1000,
> > 2000) and different numbers of sync workers (2, 4, 8, 16)
> >
> > Scripts
> > -------
> >
> > (PSA 4 scripts to implement this logic)
> >
> > testrun script
> > - this does common setup (do_one_test_setup) and then the pub/sub
> > scripts (do_one_test_PUB and do_one_test_SUB -- see below) are run in
> > parallel
> > - repeat 10 times
> >
> > do_one_test_setup script
> > - init and start instances
> > - ipc setup tables and procedures
> >
> > do_one_test_PUB script
> > - ipc setup pub/sub
> > - table setup
> > - publishes the "busy_tbl", but then waits for the subscriber to
> > subscribe to only this one
> > - alters the publication to include all other tables (so subscriber
> > will see these only after the ALTER SUBSCRIPTION PUBLICATION REFRESH)
> > - enter a busy INSERT loop until it informed by the subscriber that
> > the test is finished
> >
> > do_one_test_SUB script
> > - ipc setup pub/sub
> > - table setup
> > - subscribes only to "busy_tbl", then informs the publisher when that
> > is done (this will cause the publisher to commence the stay_busy loop)
> > - after it knows the publishing busy loop has started it does
> > - ALTER SUBSCRIPTION REFRESH PUBLICATION
> > - wait until all the tablesyncs are ready <=== This is the part that
> > is timed for the test RESULT
> >
> > PROBLEM
> > -------
> >
> > Looking at the output files (e.g. *.dat_PUB and *.dat_SUB)  they seem
> > to confirm the tests are working how we wanted.
> >
> > Unfortunately, there is some slot problem for the patched builds (both
> > designs #1 and #2). e.g. Search "ERROR" in the *.log files and see
> > many slot-related errors.
> >
> > Please note - running these same scripts with HEAD build gave no such
> > errors. So it appears to be a patch problem.
> >
>
> Hi
>
> FYI, here is some more information about ERRORs seen.
>
> The patches were re-tested -- applied in stages (and also against the
> different scripts) to identify where the problem was introduced. Below
> are the observations:
>
> ~~~
>
> Using original test scripts
>
> 1. Using only patch v21-0001
> - no errors
>
> 2. Using only patch v21-0001+0002
> - no errors
>
> 3. Using patch v21-0001+0002+0003
> - no errors
>
> ~~~
>
> Using the "busy loop" test scripts for long transactions
>
> 1. Using only patch v21-0001
> - no errors
>
> 2. Using only patch v21-0001+0002
> - gives errors for "no copy in progress issue"
> e.g. ERROR:  could not send data to WAL stream: no COPY in progress
>
> 3. Using patch v21-0001+0002+0003
> - gives the same "no copy in progress issue" errors as above
> e.g. ERROR:  could not send data to WAL stream: no COPY in progress
> - and also gives slot consistency point errors
> e.g. ERROR:  could not create replication slot
> "pg_16700_sync_16514_7261998170966054867": ERROR:  could not find
> logical decoding starting point
> e.g. LOG:  could not drop replication slot
> "pg_16700_sync_16454_7261998170966054867" on publisher: ERROR:
> replication slot "pg_16700_sync_16454_7261998170966054867" does not
> exist

I agree that  "no copy in progress issue" issue has nothing to do with
0001 patch. This issue is present with the 0002 patch.
In the case when the tablesync worker has to apply the transactions
after the table is synced, the tablesync worker sends the feedback of
writepos, applypos and flushpos which results in "No copy in progress"
error as the stream has ended already. Fixed it by exiting the
streaming loop if the tablesync worker is done with the
synchronization. The attached 0004 patch has the changes for the same.
The rest of v22 patches are the same patch that were posted by Melih
in the earlier mail.

Regards,
Vignesh

  137. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-02T09:01:39Z

    On Tue, Aug 1, 2023 at 9:44 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> FYI, here is some more information about ERRORs seen.
>
> The patches were re-tested -- applied in stages (and also against the
> different scripts) to identify where the problem was introduced. Below
> are the observations:
>
> ~~~
>
> Using original test scripts
>
> 1. Using only patch v21-0001
> - no errors
>
> 2. Using only patch v21-0001+0002
> - no errors
>
> 3. Using patch v21-0001+0002+0003
> - no errors
>
> ~~~
>
> Using the "busy loop" test scripts for long transactions
>
> 1. Using only patch v21-0001
> - no errors
>
> 2. Using only patch v21-0001+0002
> - gives errors for "no copy in progress issue"
> e.g. ERROR:  could not send data to WAL stream: no COPY in progress
>
> 3. Using patch v21-0001+0002+0003
> - gives the same "no copy in progress issue" errors as above
> e.g. ERROR:  could not send data to WAL stream: no COPY in progress
> - and also gives slot consistency point errors
> e.g. ERROR:  could not create replication slot
> "pg_16700_sync_16514_7261998170966054867": ERROR:  could not find
> logical decoding starting point
> e.g. LOG:  could not drop replication slot
> "pg_16700_sync_16454_7261998170966054867" on publisher: ERROR:
> replication slot "pg_16700_sync_16454_7261998170966054867" does not
> exist
>

I think we are getting the error (ERROR:  could not find logical
decoding starting point) because we wouldn't have waited for WAL to
become available before reading it. It could happen due to the
following code:
WalSndWaitForWal()
{
...
if (streamingDoneReceiving && streamingDoneSending &&
!pq_is_send_pending())
break;
..
}

Now, it seems that in 0003 patch, instead of resetting flags
streamingDoneSending, and streamingDoneReceiving before start
replication, we should reset before create logical slots because we
need to read the WAL during that time as well to find the consistent
point.

-- 
With Regards,
Amit Kapila.

  138. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-08-02T09:42:07Z

    Hi,

Amit Kapila <amit.kapila16@gmail.com>, 2 Ağu 2023 Çar, 12:01 tarihinde şunu
yazdı:

> I think we are getting the error (ERROR:  could not find logical
> decoding starting point) because we wouldn't have waited for WAL to
> become available before reading it. It could happen due to the
> following code:
> WalSndWaitForWal()
> {
> ...
> if (streamingDoneReceiving && streamingDoneSending &&
> !pq_is_send_pending())
> break;
> ..
> }
>
> Now, it seems that in 0003 patch, instead of resetting flags
> streamingDoneSending, and streamingDoneReceiving before start
> replication, we should reset before create logical slots because we
> need to read the WAL during that time as well to find the consistent
> point.
>

Thanks for the suggestion Amit. I've been looking into this recently and
couldn't figure out the cause until now.
I quickly made the fix in 0003. Seems like it resolved the "could not find
logical decoding starting point" errors.

vignesh C <vignesh21@gmail.com>, 1 Ağu 2023 Sal, 09:32 tarihinde şunu yazdı:

> I agree that  "no copy in progress issue" issue has nothing to do with
> 0001 patch. This issue is present with the 0002 patch.
> In the case when the tablesync worker has to apply the transactions
> after the table is synced, the tablesync worker sends the feedback of
> writepos, applypos and flushpos which results in "No copy in progress"
> error as the stream has ended already. Fixed it by exiting the
> streaming loop if the tablesync worker is done with the
> synchronization. The attached 0004 patch has the changes for the same.
> The rest of v22 patches are the same patch that were posted by Melih
> in the earlier mail.


Thanks for the fix. I placed it into 0002 with a slight change as follows:

- send_feedback(last_received, false, false);
> + if (!MyLogicalRepWorker->relsync_completed)
> + send_feedback(last_received, false, false);


IMHO relsync_completed means simply the same with streaming_done, that's
why I wanted to check that flag instead of an additional goto statement.
Does it make sense to you as well?

Thanks,
-- 
Melih Mutlu
Microsoft

  139. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-08-02T10:39:07Z

    Hi,

>
PFA an updated version with some of the earlier reviews addressed.
Forgot to include them in the previous email.

Thanks,
-- 
Melih Mutlu
Microsoft

  140. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-02T13:19:08Z

    On Wed, Aug 2, 2023 at 4:09 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> PFA an updated version with some of the earlier reviews addressed.
> Forgot to include them in the previous email.
>

It is always better to explicitly tell which reviews are addressed but
anyway, I have done some minor cleanup in the 0001 patch including
removing includes which didn't seem necessary, modified a few
comments, and ran pgindent. I also thought of modifying some variable
names based on suggestions by Peter Smith in an email [1] but didn't
find many of them any better than the current ones so modified just a
few of those. If you guys are okay with this then let's commit it and
then we can focus more on the remaining patches.

[1] - https://www.postgresql.org/message-id/CAHut%2BPs3Du9JFmhecWY8%2BVFD11VLOkSmB36t_xWHHQJNMpdA-A%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  141. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-03T04:05:11Z

    On Wed, Aug 2, 2023 at 11:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 2, 2023 at 4:09 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> >
> > PFA an updated version with some of the earlier reviews addressed.
> > Forgot to include them in the previous email.
> >
>
> It is always better to explicitly tell which reviews are addressed but
> anyway, I have done some minor cleanup in the 0001 patch including
> removing includes which didn't seem necessary, modified a few
> comments, and ran pgindent. I also thought of modifying some variable
> names based on suggestions by Peter Smith in an email [1] but didn't
> find many of them any better than the current ones so modified just a
> few of those. If you guys are okay with this then let's commit it and
> then we can focus more on the remaining patches.
>

I checked the latest patch v25-0001.

LGTM.

~~

BTW, I have re-tested many cases of HEAD versus HEAD+v25-0001 (using
current test scripts previously mentioned in this thread). Because
v25-0001 is only a refactoring patch we expect that the results should
be the same as for HEAD, and that is what I observed.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  142. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-03T06:21:48Z

    On Thu, Aug 3, 2023 at 9:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Wed, Aug 2, 2023 at 11:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Aug 2, 2023 at 4:09 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> > >
> > > PFA an updated version with some of the earlier reviews addressed.
> > > Forgot to include them in the previous email.
> > >
> >
> > It is always better to explicitly tell which reviews are addressed but
> > anyway, I have done some minor cleanup in the 0001 patch including
> > removing includes which didn't seem necessary, modified a few
> > comments, and ran pgindent. I also thought of modifying some variable
> > names based on suggestions by Peter Smith in an email [1] but didn't
> > find many of them any better than the current ones so modified just a
> > few of those. If you guys are okay with this then let's commit it and
> > then we can focus more on the remaining patches.
> >
>
> I checked the latest patch v25-0001.
>
> LGTM.
>

Thanks, I have pushed 0001. Let's focus on the remaining patches.


-- 
With Regards,
Amit Kapila.

  143. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-03T08:19:18Z

    Hi Melih,

Now that v25-0001 has been pushed, can you please rebase the remaining patches?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  144. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-03T09:06:16Z

    Just to clarify my previous post, I meant we will need new v26* patches

v24-0001 -> not needed because v25-0001 pushed
v24-0002 -> v26-0001
v24-0003 -> v26-0002

On Thu, Aug 3, 2023 at 6:19 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Melih,
>
> Now that v25-0001 has been pushed, can you please rebase the remaining patches?
>
> ------
> Kind Regards,
> Peter Smith.
> Fujitsu Australia

  145. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-08-03T11:29:59Z

    Hi,

Amit Kapila <amit.kapila16@gmail.com>, 3 Ağu 2023 Per, 09:22 tarihinde şunu
yazdı:

> On Thu, Aug 3, 2023 at 9:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > I checked the latest patch v25-0001.
> >
> > LGTM.
> >
>
> Thanks, I have pushed 0001. Let's focus on the remaining patches.
>

Thanks!

Peter Smith <smithpb2250@gmail.com>, 3 Ağu 2023 Per, 12:06 tarihinde şunu
yazdı:

> Just to clarify my previous post, I meant we will need new v26* patches
>

Right. I attached the v26 as you asked.

Thanks,
-- 
Melih Mutlu
Microsoft

  146. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-04T02:56:00Z

    FWIW, I confirmed that my review comments for v22* have all been
addressed in the latest v26* patches.

Thanks!

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  147. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-07T06:25:28Z

    Hi Melih.

Now that the design#1 ERRORs have been fixed, we returned to doing
performance measuring of the design#1 patch versus HEAD.

Unfortunately, we observed that under some particular conditions
(large transactions of 1000 inserts/tx for a busy apply worker, 100
empty tables to be synced) the performance was worse with the design#1
patch applied.

~~

RESULTS

Below are some recent measurements (for 100 empty tables to be synced
when apply worker is already busy). We vary the size of the published
transaction for the "busy" table, and you can see that for certain
large transaction sizes (1000 and 2000 inserts/tx) the design#1
performance was worse than HEAD:

~

The publisher "busy" table does commit every 10 inserts:
2w 4w 8w 16w
HEAD 3945 1138 1166 1205
HEAD+v24-0002 3559 886 355 490
%improvement 10% 22% 70% 59%

~

The publisher "busy" table does commit every 100 inserts:
2w 4w 8w 16w
HEAD 2363 1357 1354 1355
HEAD+v24-0002 2077 1358 762 756
%improvement 12% 0% 44% 44%

~

Publisher "busy" table does commit every 1000 inserts:
2w 4w 8w 16w
HEAD 11898 5855 1868 1631
HEAD+v24-0002 21905 8254 3531 1626
%improvement -84% -41% -89% 0%

^ Note - design#1 was slower than HEAD here

~

Publisher "busy" table does commit every 2000 inserts:
2w 4w 8w 16w
HEAD 21740 7109 3454 1703
HEAD+v24-0002 21585 10877 4779 2293
%improvement 1% -53% -38% -35%

^ Note - design#1 was slower than HEAD here

~

The publisher "busy" table does commit every 5000 inserts:
2w 4w 8w 16w
HEAD 36094 18105 8595 3567
HEAD+v24-0002 36305 18199 8151 3710
%improvement -1% -1% 5% -4%

~

The publisher "busy" table does commit every 10000 inserts:
2w 4w 8w 16w
HEAD 38077 18406 9426 5559
HEAD+v24-0002 36763 18027 8896 4166
%improvement 3% 2% 6% 25%

------

TEST SCRIPTS

The "busy apply" test scripts are basically the same as already posted
[1], but I have reattached the latest ones again anyway.

------
[1] https://www.postgresql.org/message-id/CAHut%2BPuNVNK2%2BA%2BR6eV8rKPNBHemCFE4NDtEYfpXbYr6SsvvBg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  148. RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2023-08-09T02:58:03Z

    On Thursday, August 3, 2023 7:30 PM Melih Mutlu <m.melihmutlu@gmail.com>  wrote:

> Right. I attached the v26 as you asked. 

Thanks for posting the patches.
 
While reviewing the patch, I noticed one rare case that it's possible that there
are two table sync worker for the same table in the same time.

The patch relies on LogicalRepWorkerLock to prevent concurrent access, but the
apply worker will start a new worker after releasing the lock. So, at the point[1]
where the lock is released and the new table sync worker has not been started,
it seems possible that another old table sync worker will be reused for the
same table.

				/* Now safe to release the LWLock */
				LWLockRelease(LogicalRepWorkerLock);
*[1]
				/*
				 * If there are free sync worker slot(s), start a new sync
				 * worker for the table.
				 */
				if (nsyncworkers < max_sync_workers_per_subscription)
				...
						logicalrep_worker_launch(MyLogicalRepWorker->dbid,

I can reproduce it by using gdb.

Steps:
1. set max_sync_workers_per_subscription to 1 and setup pub/sub which publishes
   two tables(table A and B).
2. when the table sync worker for the table A started, use gdb to block it
   before being reused for another table.
3. set max_sync_workers_per_subscription to 2 and use gdb to block the apply
   worker at the point after releasing the LogicalRepWorkerLock and before
   starting another table sync worker for table B.
4. release the blocked table sync worker, then we can see the table sync worker
   is also reused for table B.
5. release the apply worker, then we can see the apply worker will start
   another table sync worker for the same table(B).

I think it would be better to prevent this case from happening as this case
will give some unexpected ERROR or LOG. Note that I haven't checked if it would
cause worse problems like duplicate copy or others.

Best Regards,
Hou zj

  149. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-08-09T04:21:03Z

    Hi Melih,

Here is a patch to help in getting the execution at various phases
like: a) replication slot creation time, b) Wal reading c) Number of
WAL records read d) subscription relation state change etc
Couple of observation while we tested with this patch:
1) We noticed that the patch takes more time for finding the decoding
start point.
2) Another observation was that the number of XLOG records read for
identify the consistent point was significantly high with the v26_0001
patch.

HEAD
postgres=# select avg(counttime)/1000 "avgtime(ms)",
median(counttime)/1000 "median(ms)", min(counttime)/1000
"mintime(ms)", max(counttime)/1000 "maxtime(ms)", logtype from test
group by logtype;
      avgtime(ms)       |       median(ms)       | mintime(ms) |
maxtime(ms) |         logtype
------------------------+------------------------+-------------+-------------+--------------------------
 0.00579245283018867920 | 0.00200000000000000000 |           0 |
    1 | SNAPSHOT_BUILD
     1.2246811320754717 | 0.98550000000000000000 |           0 |
   37 | LOGICAL_SLOT_CREATION
   171.0863283018867920 |   183.9120000000000000 |           0 |
  408 | FIND_DECODING_STARTPOINT
     2.0699433962264151 |     1.4380000000000000 |           1 |
   49 | INIT_DECODING_CONTEXT
(4 rows)

HEAD + v26-0001 patch
postgres=# select avg(counttime)/1000 "avgtime(ms)",
median(counttime)/1000 "median(ms)", min(counttime)/1000
"mintime(ms)", max(counttime)/1000 "maxtime(ms)", logtype from test
group by logtype;
      avgtime(ms)       |       median(ms)       | mintime(ms) |
maxtime(ms) |         logtype
------------------------+------------------------+-------------+-------------+--------------------------
 0.00588113207547169810 | 0.00500000000000000000 |           0 |
    0 | SNAPSHOT_BUILD
     1.1270962264150943 |     1.1000000000000000 |           0 |
    2 | LOGICAL_SLOT_CREATION
   301.1745528301886790 |   410.4870000000000000 |           0 |
  427 | FIND_DECODING_STARTPOINT
     1.4814660377358491 |     1.4530000000000000 |           1 |
    9 | INIT_DECODING_CONTEXT
(4 rows)

In the above FIND_DECODING_STARTPOINT is very much higher with V26-0001 patch.

HEAD
FIND_DECODING_XLOG_RECORD_COUNT
- average =  2762
- median = 3362

HEAD + reuse worker patch(v26_0001 patch)
Where FIND_DECODING_XLOG_RECORD_COUNT
- average =  4105
- median = 5345

Similarly Number of xlog records read is higher with v26_0001 patch.

Steps to calculate the timing:
-- first collect the necessary LOG from subscriber's log.
cat *.log | grep -E
'(LOGICAL_SLOT_CREATION|INIT_DECODING_CONTEXT|FIND_DECODING_STARTPOINT|SNAPSHOT_BUILD|FIND_DECODING_XLOG_RECORD_COUNT|LOGICAL_XLOG_READ|LOGICAL_DECODE_PROCESS_RECORD|LOGICAL_WAIT_TRANSACTION)'
> grep.dat

create table testv26(logtime varchar, pid varchar, level varchar,
space varchar, logtype varchar, counttime int);
-- then copy these datas into db table to count the avg number.
COPY testv26 FROM '/home/logs/grep.dat' DELIMITER ' ';

-- Finally, use the SQL to analyze the data:
select avg(counttime)/1000 "avgtime(ms)", logtype from testv26 group by logtype;

--- To get the number of xlog records read:
select avg(counttime) from testv26 where logtype
='FIND_DECODING_XLOG_RECORD_COUNT' and counttime != 1;

Thanks to Peter and Hou-san who helped in finding these out. We are
parallely analysing this, @Melih Mutlu  posting this information so
that it might help you too in analysing this issue.

Regards,
Vignesh

  150. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-10T04:45:50Z

    On Wed, Aug 9, 2023 at 8:28 AM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Thursday, August 3, 2023 7:30 PM Melih Mutlu <m.melihmutlu@gmail.com>  wrote:
>
> > Right. I attached the v26 as you asked.
>
> Thanks for posting the patches.
>
> While reviewing the patch, I noticed one rare case that it's possible that there
> are two table sync worker for the same table in the same time.
>
> The patch relies on LogicalRepWorkerLock to prevent concurrent access, but the
> apply worker will start a new worker after releasing the lock. So, at the point[1]
> where the lock is released and the new table sync worker has not been started,
> it seems possible that another old table sync worker will be reused for the
> same table.
>
>                                 /* Now safe to release the LWLock */
>                                 LWLockRelease(LogicalRepWorkerLock);
> *[1]
>                                 /*
>                                  * If there are free sync worker slot(s), start a new sync
>                                  * worker for the table.
>                                  */
>                                 if (nsyncworkers < max_sync_workers_per_subscription)
>                                 ...
>                                                 logicalrep_worker_launch(MyLogicalRepWorker->dbid,
>

Yeah, this is a problem. I think one idea to solve this is by
extending the lock duration till we launch the tablesync worker but we
should also consider changing this locking scheme such that there is a
better way to indicate that for a particular rel, tablesync is in
progress. Currently, the code in TablesyncWorkerMain() also acquires
the lock in exclusive mode even though the tablesync for a rel is in
progress which I guess could easily heart us for larger values of
max_logical_replication_workers. So, that could be another motivation
to think for a different locking scheme.

-- 
With Regards,
Amit Kapila.

  151. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-10T05:35:18Z

    Hi Melih,

FYI -- The same testing was repeated but this time PG was configured
to say synchronous_commit=on. Other factors and scripts were the same
as before --- busy apply, 5 runs, 4 workers, 1000 inserts/tx, 100
empty tables, etc.

There are still more xlog records seen for the v26 patch, but now the
v26 performance was better than HEAD.

RESULTS (synchronous_commit=on)
---------------------------------------------------

Xlog Counts

HEAD
postgres=# select avg(counttime) "avg", median(counttime) "median",
min(counttime) "min", max(counttime) "max", logtype from test_head
group by logtype;
          avg          |        median         | min | max  |
   logtype
-----------------------+-----------------------+-----+------+-----------
-----------------------+-----------------------+-----+------+-----------
-----------------------+-----------------------+-----+------+-----------
1253.7509433962264151 | 1393.0000000000000000 |   1 | 2012 |
FIND_DECODING_XLOG_RECORD_COUNT
(1 row)


HEAD+v26-0001
postgres=# select avg(counttime) "avg", median(counttime) "median",
min(counttime) "min", max(counttime) "max", logtype from test_v26
group by logtype;
          avg          |        median         | min | max  |
   logtype
-----------------------+-----------------------+-----+------+-----------
-----------------------+-----------------------+-----+------+-----------
-----------------------+-----------------------+-----+------+-----------
1278.4075471698113208 | 1423.5000000000000000 |   1 | 2015 |
FIND_DECODING_XLOG_RECORD_COUNT
(1 row)

~~~~~~

Performance

HEAD
[peter@localhost res_0809_vignesh_timing_sync_head]$ cat *.dat_SUB |
grep RESULT | grep -v duration | awk '{print $3}'
4014.266
3892.089
4195.318
3571.862
4312.183


HEAD+v26-0001
[peter@localhost res_0809_vignesh_timing_sync_v260001]$ cat *.dat_SUB
| grep RESULT | grep -v duration | awk '{print $3}'
3326.627
3213.028
3433.611
3299.803
3258.821

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  152. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-08-10T14:54:02Z

    Hi Peter and Vignesh,

Peter Smith <smithpb2250@gmail.com>, 7 Ağu 2023 Pzt, 09:25 tarihinde şunu
yazdı:

> Hi Melih.
>
> Now that the design#1 ERRORs have been fixed, we returned to doing
> performance measuring of the design#1 patch versus HEAD.


Thanks a lot for taking the time to benchmark the patch. It's really
helpful.

Publisher "busy" table does commit every 1000 inserts:
> 2w 4w 8w 16w
> HEAD 11898 5855 1868 1631
> HEAD+v24-0002 21905 8254 3531 1626
> %improvement -84% -41% -89% 0%


> ^ Note - design#1 was slower than HEAD here


> ~


> Publisher "busy" table does commit every 2000 inserts:
> 2w 4w 8w 16w
> HEAD 21740 7109 3454 1703
> HEAD+v24-0002 21585 10877 4779 2293
> %improvement 1% -53% -38% -35%


I assume you meant HEAD+v26-0002 and not v24. I wanted to quickly reproduce
these two cases where the patch was significantly worse. Interestingly my
results are a bit different than yours.

Publisher "busy" table does commit every 1000 inserts:
2w 4w 8w 16w
HEAD 22405 10335 5008 3304
HEAD+v26  19954  8037 4068 2761
%improvement 1% 2% 2% 1%

Publisher "busy" table does commit every 2000 inserts:
2w 4w 8w 16w
HEAD 33122 14220 7251 4279
HEAD+v26 34248 16213 7356 3914
%improvement 0% -1% 0% 1%

If I'm not doing something wrong in testing (or maybe the patch doesn't
perform reliable yet for some reason), I don't see a drastic change in
performance. But I guess the patch is supposed to perform better than HEAD
in these both cases anyway. right?. I would expect the performance of the
patch to converge to HEAD's performance with large tables. But I'm not sure
what to expect when apply worker is busy with large transactions.

However, I need to investigate a bit more what Vignesh shared earlier [1].
It makes sense that those issues can cause this problem here.

It just takes a bit of time for me to figure out these things, but I'm
working on it.

[1]
https://www.postgresql.org/message-id/CALDaNm1TA068E2niJFUR9ig%2BYz3-ank%3Dj5%3Dj-2UocbzaDnQPrA%40mail.gmail.com



Thanks,
-- 
Melih Mutlu
Microsoft

  153. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-10T22:25:57Z

    On Fri, Aug 11, 2023 at 12:54 AM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Hi Peter and Vignesh,
>
> Peter Smith <smithpb2250@gmail.com>, 7 Ağu 2023 Pzt, 09:25 tarihinde şunu yazdı:
>>
>> Hi Melih.
>>
>> Now that the design#1 ERRORs have been fixed, we returned to doing
>> performance measuring of the design#1 patch versus HEAD.
>
>
> Thanks a lot for taking the time to benchmark the patch. It's really helpful.
>
>> Publisher "busy" table does commit every 1000 inserts:
>> 2w 4w 8w 16w
>> HEAD 11898 5855 1868 1631
>> HEAD+v24-0002 21905 8254 3531 1626
>> %improvement -84% -41% -89% 0%
>>
>>
>> ^ Note - design#1 was slower than HEAD here
>>
>>
>> ~
>>
>>
>> Publisher "busy" table does commit every 2000 inserts:
>> 2w 4w 8w 16w
>> HEAD 21740 7109 3454 1703
>> HEAD+v24-0002 21585 10877 4779 2293
>> %improvement 1% -53% -38% -35%
>
>
> I assume you meant HEAD+v26-0002 and not v24. I wanted to quickly reproduce these two cases where the patch was significantly worse. Interestingly my results are a bit different than yours.
>

No, I meant what I wrote there. When I ran the tests the HEAD included
the v25-0001 refactoring patch, but v26 did not yet exist.

For now, we are only performance testing the first
"Reuse-Tablesyc-Workers" patch, but not yet including the second patch
("Reuse connection when...").

Note that those "Reuse-Tablesyc-Workers" patches v24-0002 and v26-0001
are equivalent because there are only cosmetic log message differences
between them.
So, my testing was with HEAD+v24-0002 (but not including v24-0003).
Your same testing should be with HEAD+v26-0001 (but not including v26-0002).

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  154. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-08-11T10:56:26Z

    On Wed, 9 Aug 2023 at 09:51, vignesh C <vignesh21@gmail.com> wrote:
>
> Hi Melih,
>
> Here is a patch to help in getting the execution at various phases
> like: a) replication slot creation time, b) Wal reading c) Number of
> WAL records read d) subscription relation state change etc
> Couple of observation while we tested with this patch:
> 1) We noticed that the patch takes more time for finding the decoding
> start point.
> 2) Another observation was that the number of XLOG records read for
> identify the consistent point was significantly high with the v26_0001
> patch.
>
> HEAD
> postgres=# select avg(counttime)/1000 "avgtime(ms)",
> median(counttime)/1000 "median(ms)", min(counttime)/1000
> "mintime(ms)", max(counttime)/1000 "maxtime(ms)", logtype from test
> group by logtype;
>       avgtime(ms)       |       median(ms)       | mintime(ms) |
> maxtime(ms) |         logtype
> ------------------------+------------------------+-------------+-------------+--------------------------
>  0.00579245283018867920 | 0.00200000000000000000 |           0 |
>     1 | SNAPSHOT_BUILD
>      1.2246811320754717 | 0.98550000000000000000 |           0 |
>    37 | LOGICAL_SLOT_CREATION
>    171.0863283018867920 |   183.9120000000000000 |           0 |
>   408 | FIND_DECODING_STARTPOINT
>      2.0699433962264151 |     1.4380000000000000 |           1 |
>    49 | INIT_DECODING_CONTEXT
> (4 rows)
>
> HEAD + v26-0001 patch
> postgres=# select avg(counttime)/1000 "avgtime(ms)",
> median(counttime)/1000 "median(ms)", min(counttime)/1000
> "mintime(ms)", max(counttime)/1000 "maxtime(ms)", logtype from test
> group by logtype;
>       avgtime(ms)       |       median(ms)       | mintime(ms) |
> maxtime(ms) |         logtype
> ------------------------+------------------------+-------------+-------------+--------------------------
>  0.00588113207547169810 | 0.00500000000000000000 |           0 |
>     0 | SNAPSHOT_BUILD
>      1.1270962264150943 |     1.1000000000000000 |           0 |
>     2 | LOGICAL_SLOT_CREATION
>    301.1745528301886790 |   410.4870000000000000 |           0 |
>   427 | FIND_DECODING_STARTPOINT
>      1.4814660377358491 |     1.4530000000000000 |           1 |
>     9 | INIT_DECODING_CONTEXT
> (4 rows)
>
> In the above FIND_DECODING_STARTPOINT is very much higher with V26-0001 patch.
>
> HEAD
> FIND_DECODING_XLOG_RECORD_COUNT
> - average =  2762
> - median = 3362
>
> HEAD + reuse worker patch(v26_0001 patch)
> Where FIND_DECODING_XLOG_RECORD_COUNT
> - average =  4105
> - median = 5345
>
> Similarly Number of xlog records read is higher with v26_0001 patch.
>
> Steps to calculate the timing:
> -- first collect the necessary LOG from subscriber's log.
> cat *.log | grep -E
> '(LOGICAL_SLOT_CREATION|INIT_DECODING_CONTEXT|FIND_DECODING_STARTPOINT|SNAPSHOT_BUILD|FIND_DECODING_XLOG_RECORD_COUNT|LOGICAL_XLOG_READ|LOGICAL_DECODE_PROCESS_RECORD|LOGICAL_WAIT_TRANSACTION)'
> > grep.dat
>
> create table testv26(logtime varchar, pid varchar, level varchar,
> space varchar, logtype varchar, counttime int);
> -- then copy these datas into db table to count the avg number.
> COPY testv26 FROM '/home/logs/grep.dat' DELIMITER ' ';
>
> -- Finally, use the SQL to analyze the data:
> select avg(counttime)/1000 "avgtime(ms)", logtype from testv26 group by logtype;
>
> --- To get the number of xlog records read:
> select avg(counttime) from testv26 where logtype
> ='FIND_DECODING_XLOG_RECORD_COUNT' and counttime != 1;
>
> Thanks to Peter and Hou-san who helped in finding these out. We are
> parallely analysing this, @Melih Mutlu  posting this information so
> that it might help you too in analysing this issue.

I analysed further on why it needs to read a larger number of XLOG
records in some cases while creating the replication slot, here are my
thoughts:
Note: Tablesync worker needs to connect to the publisher and create
consistent point for the slots by reading the XLOG records. This
requires that all the open transactions and the transactions that are
created while creating consistent point should be committed.
I feel the creation of slots is better in few cases in Head because:
Publisher                        | Subscriber
------------------------------------------------------------
Begin txn1 transaction    |
Insert 1..1000 records    |
Commit                           |
Begin txn2 transaction    |
Insert 1..1000 records     |  Apply worker applies transaction txn1
                                        |  Start tablesync table t2
                                        |  create consistent point in
                                        | publisher before transaction txn3 is
                                        | started
commit                            |  We just need to wait till
                                        | transaction txn2 is finished.
Begin txn3 transaction    |
Insert 1..1000 records     |
commit                            |

In V26, this is happening in some cases:
Publisher                        | Subscriber
------------------------------------------------------------
Begin txn1 transaction    |
Insert 1..1000 records    |
Commit                           |
Begin txn2 transaction    |
Insert 1..1000 records     |  Apply worker applies transaction txn1
                                        |  Start tablesync table t2
commit                             |  Create consistent point
Begin txn3 transaction     |  (since transaction txn2 is committed
                                        |  and txn3 is started, we will
                                        |  need to wait
                                        |  for transaction txn3 to be
                                        |  committed)
Insert 1..1000 records     |
commit                             |

This is because In HEAD the tablesync worker will be started after one
commit, so we are able to create the consistent point before a new
transaction is started in some cases.
Create slot will be fastest if the tablesync worker is able to connect
to the publisher and create a consistent point before the new
transaction is started. The probability of this is better in HEAD for
this scenario as the new tablesync worker is started after commit and
the tablesync worker in HEAD has a better time window(because the
current transaction has just started) before another new transaction
is started. This probability is slightly lower with the V26 version.
I felt this issue is purely a timing issue in a few cases because of
the timing of the new transactions being created while creating the
slot.

Regards,
Vignesh

  155. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-08-11T12:24:03Z

    On Fri, 11 Aug 2023 at 16:26, vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 9 Aug 2023 at 09:51, vignesh C <vignesh21@gmail.com> wrote:
> >
> > Hi Melih,
> >
> > Here is a patch to help in getting the execution at various phases
> > like: a) replication slot creation time, b) Wal reading c) Number of
> > WAL records read d) subscription relation state change etc
> > Couple of observation while we tested with this patch:
> > 1) We noticed that the patch takes more time for finding the decoding
> > start point.
> > 2) Another observation was that the number of XLOG records read for
> > identify the consistent point was significantly high with the v26_0001
> > patch.
> >
> > HEAD
> > postgres=# select avg(counttime)/1000 "avgtime(ms)",
> > median(counttime)/1000 "median(ms)", min(counttime)/1000
> > "mintime(ms)", max(counttime)/1000 "maxtime(ms)", logtype from test
> > group by logtype;
> >       avgtime(ms)       |       median(ms)       | mintime(ms) |
> > maxtime(ms) |         logtype
> > ------------------------+------------------------+-------------+-------------+--------------------------
> >  0.00579245283018867920 | 0.00200000000000000000 |           0 |
> >     1 | SNAPSHOT_BUILD
> >      1.2246811320754717 | 0.98550000000000000000 |           0 |
> >    37 | LOGICAL_SLOT_CREATION
> >    171.0863283018867920 |   183.9120000000000000 |           0 |
> >   408 | FIND_DECODING_STARTPOINT
> >      2.0699433962264151 |     1.4380000000000000 |           1 |
> >    49 | INIT_DECODING_CONTEXT
> > (4 rows)
> >
> > HEAD + v26-0001 patch
> > postgres=# select avg(counttime)/1000 "avgtime(ms)",
> > median(counttime)/1000 "median(ms)", min(counttime)/1000
> > "mintime(ms)", max(counttime)/1000 "maxtime(ms)", logtype from test
> > group by logtype;
> >       avgtime(ms)       |       median(ms)       | mintime(ms) |
> > maxtime(ms) |         logtype
> > ------------------------+------------------------+-------------+-------------+--------------------------
> >  0.00588113207547169810 | 0.00500000000000000000 |           0 |
> >     0 | SNAPSHOT_BUILD
> >      1.1270962264150943 |     1.1000000000000000 |           0 |
> >     2 | LOGICAL_SLOT_CREATION
> >    301.1745528301886790 |   410.4870000000000000 |           0 |
> >   427 | FIND_DECODING_STARTPOINT
> >      1.4814660377358491 |     1.4530000000000000 |           1 |
> >     9 | INIT_DECODING_CONTEXT
> > (4 rows)
> >
> > In the above FIND_DECODING_STARTPOINT is very much higher with V26-0001 patch.
> >
> > HEAD
> > FIND_DECODING_XLOG_RECORD_COUNT
> > - average =  2762
> > - median = 3362
> >
> > HEAD + reuse worker patch(v26_0001 patch)
> > Where FIND_DECODING_XLOG_RECORD_COUNT
> > - average =  4105
> > - median = 5345
> >
> > Similarly Number of xlog records read is higher with v26_0001 patch.
> >
> > Steps to calculate the timing:
> > -- first collect the necessary LOG from subscriber's log.
> > cat *.log | grep -E
> > '(LOGICAL_SLOT_CREATION|INIT_DECODING_CONTEXT|FIND_DECODING_STARTPOINT|SNAPSHOT_BUILD|FIND_DECODING_XLOG_RECORD_COUNT|LOGICAL_XLOG_READ|LOGICAL_DECODE_PROCESS_RECORD|LOGICAL_WAIT_TRANSACTION)'
> > > grep.dat
> >
> > create table testv26(logtime varchar, pid varchar, level varchar,
> > space varchar, logtype varchar, counttime int);
> > -- then copy these datas into db table to count the avg number.
> > COPY testv26 FROM '/home/logs/grep.dat' DELIMITER ' ';
> >
> > -- Finally, use the SQL to analyze the data:
> > select avg(counttime)/1000 "avgtime(ms)", logtype from testv26 group by logtype;
> >
> > --- To get the number of xlog records read:
> > select avg(counttime) from testv26 where logtype
> > ='FIND_DECODING_XLOG_RECORD_COUNT' and counttime != 1;
> >
> > Thanks to Peter and Hou-san who helped in finding these out. We are
> > parallely analysing this, @Melih Mutlu  posting this information so
> > that it might help you too in analysing this issue.
>
> I analysed further on why it needs to read a larger number of XLOG
> records in some cases while creating the replication slot, here are my
> thoughts:
> Note: Tablesync worker needs to connect to the publisher and create
> consistent point for the slots by reading the XLOG records. This
> requires that all the open transactions and the transactions that are
> created while creating consistent point should be committed.
> I feel the creation of slots is better in few cases in Head because:
> Publisher                        | Subscriber
> ------------------------------------------------------------
> Begin txn1 transaction    |
> Insert 1..1000 records    |
> Commit                           |
> Begin txn2 transaction    |
> Insert 1..1000 records     |  Apply worker applies transaction txn1
>                                         |  Start tablesync table t2
>                                         |  create consistent point in
>                                         | publisher before transaction txn3 is
>                                         | started
> commit                            |  We just need to wait till
>                                         | transaction txn2 is finished.
> Begin txn3 transaction    |
> Insert 1..1000 records     |
> commit                            |
>
> In V26, this is happening in some cases:
> Publisher                        | Subscriber
> ------------------------------------------------------------
> Begin txn1 transaction    |
> Insert 1..1000 records    |
> Commit                           |
> Begin txn2 transaction    |
> Insert 1..1000 records     |  Apply worker applies transaction txn1
>                                         |  Start tablesync table t2
> commit                             |  Create consistent point
> Begin txn3 transaction     |  (since transaction txn2 is committed
>                                         |  and txn3 is started, we will
>                                         |  need to wait
>                                         |  for transaction txn3 to be
>                                         |  committed)
> Insert 1..1000 records     |
> commit                             |
>
> This is because In HEAD the tablesync worker will be started after one
> commit, so we are able to create the consistent point before a new
> transaction is started in some cases.
> Create slot will be fastest if the tablesync worker is able to connect
> to the publisher and create a consistent point before the new
> transaction is started. The probability of this is better in HEAD for
> this scenario as the new tablesync worker is started after commit and
> the tablesync worker in HEAD has a better time window(because the
> current transaction has just started) before another new transaction
> is started. This probability is slightly lower with the V26 version.
> I felt this issue is purely a timing issue in a few cases because of
> the timing of the new transactions being created while creating the
> slot.

I used the following steps to analyse this issue:
Logs can be captured by applying the patches at [1].

-- first collect the necessary information about from publisher's log
from the execution of HEAD:
cat *.log | grep FIND_DECODING_XLOG_RECORD_COUNT > grep_head.dat

-- first collect the necessary information about from publisher's log
from the execution of v26:
cat *.log | grep FIND_DECODING_XLOG_RECORD_COUNT > grep_v26.dat

-- then copy these datas into HEAD's db table to count the avg number.
COPY test_head FROM '/home/logs/grep_head.dat' DELIMITER ' ';

-- then copy these datas into the v26 db table to count the avg number.
COPY test_v26 FROM '/home/logs/grep_v26.dat' DELIMITER ' ';

Find the average of XLOG records read in HEAD:
postgres=# select avg(counttime) from test_head  where logtype
='FIND_DECODING_XLOG_RECORD_COUNT' and counttime != 1;
          avg
-----------------------
 1394.1100000000000000
(1 row)

Find the average of XLOG records read in V26:
postgres=# select avg(counttime) from test_v26 where logtype
='FIND_DECODING_XLOG_RECORD_COUNT' and counttime != 1;
          avg
-----------------------
 1900.4100000000000000
(1 row)

When analysing why create replication slot needs to read more records
in a few cases, I found a very interesting observation. I found that
with HEAD about 29% (29 out of 100 tables) of tables could find the
consistent point by reading the WAL records up to the next subsequent
COMMIT, whereas with V26 patch only 5% of tables could find the
consistent point by reading the WAL records up to next subsequent
commit. In these cases V26 patch had to read another transaction of
approximately > 1000 WAL records to reach the consistent point which
results in an increase of average for more records to be read with V26
version. For these I got the start lsn and consistent lsn from the log
files by matching the corresponding FIND_DECODING_XLOG_RECORD_COUNT, I
did a waldump of the WAL file and searched the records between start
lsn and consistent LSN in the WAL dump and confirmed that only one
COMMIT record had to be read to reach the consistent point. Details of
this information from the log of HEAD and V26 is attached.

The number of tables required to read less than 1 commit can be found
by the following:
-- I checked for 1000 WAL records because we are having 1000 inserts
in each transaction.
select count(counttime) from test_head  where logtype
='FIND_DECODING_XLOG_RECORD_COUNT' and counttime < 1000;
 count
-------
    29
(1 row)

select count(counttime) from test_v26  where logtype
='FIND_DECODING_XLOG_RECORD_COUNT' and counttime < 1000;
 count
-------
    5
(1 row)

Apart from these there were other instances where the V26 had to read
more COMMIT record in few cases.
The above is happening because as mentioned in [2]. i.e. in HEAD the
tablesync worker will be started after one commit, so we are able to
create the consistent point before a new transaction is started in
some cases. Create slot will be fastest if the tablesync worker is
able to connect to the publisher and create a consistent point before
the new transaction is started. The probability of this is better in
HEAD for this scenario as the new tablesync worker is started after
commit and the tablesync worker in HEAD has a better time
window(because the current transaction has just started) before
another new transaction  is started. This probability is slightly
lower with the V26 version. I felt this issue is purely a timing issue
in a few cases because of the timing of the new transactions being
created while creating the slot.
Since this is purely a timing issue as explained above in a few cases
because of the timing of the new transactions being created while
creating the slot, I felt we can ignore this.

[1] - https://www.postgresql.org/message-id/CALDaNm1TA068E2niJFUR9ig%2BYz3-ank%3Dj5%3Dj-2UocbzaDnQPrA%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/CALDaNm2k2z3Hpa3Omb_tpxWkyHnUvsWjJMbqDs-2uD2eLzemJQ%40mail.gmail.com

Regards,
Vignesh

  156. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Melih Mutlu <m.melihmutlu@gmail.com> — 2023-08-11T13:45:39Z

    Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 11 Ağu 2023 Cum, 01:26 tarihinde şunu
yazdı:

> No, I meant what I wrote there. When I ran the tests the HEAD included
> the v25-0001 refactoring patch, but v26 did not yet exist.
>
> For now, we are only performance testing the first
> "Reuse-Tablesyc-Workers" patch, but not yet including the second patch
> ("Reuse connection when...").
>
> Note that those "Reuse-Tablesyc-Workers" patches v24-0002 and v26-0001
> are equivalent because there are only cosmetic log message differences
> between them.
>

Ok, that's fair.



> So, my testing was with HEAD+v24-0002 (but not including v24-0003).
> Your same testing should be with HEAD+v26-0001 (but not including
> v26-0002).
>

That's actually what I did. I should have been more clear about what I
included in my previous email.With v26-0002, results are noticeably better
anyway.
I just rerun the test again against HEAD, HEAD+v26-0001 and additionally
HEAD+v26-0001+v26-0002 this time, for better comparison.

Here are my results with the same scripts you shared earlier (I obviously
only changed the number of inserts before each commit. ).
Note that this is when synchronous_commit = off.

100 inserts/tx
+-------------+-------+------+------+------+
|             | 2w    | 4w   | 8w   | 16w  |
+-------------+-------+------+------+------+
| v26-0002    | 10421 | 6472 | 6656 | 6566 |
+-------------+-------+------+------+------+
| improvement | 31%   | 12%  | 0%   | 5%   |
+-------------+-------+------+------+------+
| v26-0001    | 14585 | 7386 | 7129 | 7274 |
+-------------+-------+------+------+------+
| improvement | 9%    | 5%   | 12%  | 7%   |
+-------------+-------+------+------+------+
| HEAD        | 16130 | 7785 | 8147 | 7827 |
+-------------+-------+------+------+------+

1000 inserts/tx
+-------------+-------+------+------+------+
|             | 2w    | 4w   | 8w   | 16w  |
+-------------+-------+------+------+------+
| v26-0002    | 13796 | 6848 | 5942 | 6315 |
+-------------+-------+------+------+------+
| improvement | 9%    | 7%   | 10%  | 8%   |
+-------------+-------+------+------+------+
| v26-0001    | 14685 | 7325 | 6675 | 6719 |
+-------------+-------+------+------+------+
| improvement | 3%    | 0%   | 0%   | 2%   |
+-------------+-------+------+------+------+
| HEAD        | 15118 | 7354 | 6644 | 6890 |
+-------------+-------+------+------+------+

2000 inserts/tx
+-------------+-------+-------+------+------+
|             | 2w    | 4w    | 8w   | 16w  |
+-------------+-------+-------+------+------+
| v26-0002    | 22442 | 9944  | 6034 | 5829 |
+-------------+-------+-------+------+------+
| improvement | 5%    | 2%    | 4%   | 10%  |
+-------------+-------+-------+------+------+
| v26-0001    | 23632 | 10164 | 6311 | 6480 |
+-------------+-------+-------+------+------+
| improvement | 0%    | 0%    | 0%   | 0%   |
+-------------+-------+-------+------+------+
| HEAD        | 23667 | 10157 | 6285 | 6470 |
+-------------+-------+-------+------+------+

5000 inserts/tx
+-------------+-------+-------+-------+------+
|             | 2w    | 4w    | 8w    | 16w  |
+-------------+-------+-------+-------+------+
| v26-0002    | 41443 | 21385 | 10832 | 6146 |
+-------------+-------+-------+-------+------+
| improvement | 0%    | 0%    | 1%    | 16%  |
+-------------+-------+-------+-------+------+
| v26-0001    | 41293 | 21226 | 10814 | 6158 |
+-------------+-------+-------+-------+------+
| improvement | 0%    | 1%    | 1%    | 15%  |
+-------------+-------+-------+-------+------+
| HEAD        | 41503 | 21466 | 10943 | 7292 |
+-------------+-------+-------+-------+------+


Again, I couldn't reproduce the cases where you saw significantly degraded
performance. I wonder if I'm missing something. Did you do anything not
included in the test scripts you shared? Do you think v26-0001 will
perform 84% worse than HEAD, if you try again? I just want to be sure that
it was not a random thing.
Interestingly, I also don't see an improvement in above results as big as
in your results when inserts/tx ratio is smaller. Even though it certainly
is improved in such cases.

Thanks,
-- 
Melih Mutlu
Microsoft

  157. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-12T13:51:24Z

    On Fri, Aug 11, 2023 at 7:15 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Peter Smith <smithpb2250@gmail.com>, 11 Ağu 2023 Cum, 01:26 tarihinde şunu yazdı:
>>
>> No, I meant what I wrote there. When I ran the tests the HEAD included
>> the v25-0001 refactoring patch, but v26 did not yet exist.
>>
>> For now, we are only performance testing the first
>> "Reuse-Tablesyc-Workers" patch, but not yet including the second patch
>> ("Reuse connection when...").
>>
>> Note that those "Reuse-Tablesyc-Workers" patches v24-0002 and v26-0001
>> are equivalent because there are only cosmetic log message differences
>> between them.
>
>
> Ok, that's fair.
>
>
>>
>> So, my testing was with HEAD+v24-0002 (but not including v24-0003).
>> Your same testing should be with HEAD+v26-0001 (but not including v26-0002).
>
>
> That's actually what I did. I should have been more clear about what I included in my previous email.With v26-0002, results are noticeably better anyway.
> I just rerun the test again against HEAD, HEAD+v26-0001 and additionally HEAD+v26-0001+v26-0002 this time, for better comparison.
>
> Here are my results with the same scripts you shared earlier (I obviously only changed the number of inserts before each commit. ).
> Note that this is when synchronous_commit = off.
>
> 100 inserts/tx
> +-------------+-------+------+------+------+
> |             | 2w    | 4w   | 8w   | 16w  |
> +-------------+-------+------+------+------+
> | v26-0002    | 10421 | 6472 | 6656 | 6566 |
> +-------------+-------+------+------+------+
> | improvement | 31%   | 12%  | 0%   | 5%   |
> +-------------+-------+------+------+------+
> | v26-0001    | 14585 | 7386 | 7129 | 7274 |
> +-------------+-------+------+------+------+
> | improvement | 9%    | 5%   | 12%  | 7%   |
> +-------------+-------+------+------+------+
> | HEAD        | 16130 | 7785 | 8147 | 7827 |
> +-------------+-------+------+------+------+
>
> 1000 inserts/tx
> +-------------+-------+------+------+------+
> |             | 2w    | 4w   | 8w   | 16w  |
> +-------------+-------+------+------+------+
> | v26-0002    | 13796 | 6848 | 5942 | 6315 |
> +-------------+-------+------+------+------+
> | improvement | 9%    | 7%   | 10%  | 8%   |
> +-------------+-------+------+------+------+
> | v26-0001    | 14685 | 7325 | 6675 | 6719 |
> +-------------+-------+------+------+------+
> | improvement | 3%    | 0%   | 0%   | 2%   |
> +-------------+-------+------+------+------+
> | HEAD        | 15118 | 7354 | 6644 | 6890 |
> +-------------+-------+------+------+------+
>
> 2000 inserts/tx
> +-------------+-------+-------+------+------+
> |             | 2w    | 4w    | 8w   | 16w  |
> +-------------+-------+-------+------+------+
> | v26-0002    | 22442 | 9944  | 6034 | 5829 |
> +-------------+-------+-------+------+------+
> | improvement | 5%    | 2%    | 4%   | 10%  |
> +-------------+-------+-------+------+------+
> | v26-0001    | 23632 | 10164 | 6311 | 6480 |
> +-------------+-------+-------+------+------+
> | improvement | 0%    | 0%    | 0%   | 0%   |
> +-------------+-------+-------+------+------+
> | HEAD        | 23667 | 10157 | 6285 | 6470 |
> +-------------+-------+-------+------+------+
>
> 5000 inserts/tx
> +-------------+-------+-------+-------+------+
> |             | 2w    | 4w    | 8w    | 16w  |
> +-------------+-------+-------+-------+------+
> | v26-0002    | 41443 | 21385 | 10832 | 6146 |
> +-------------+-------+-------+-------+------+
> | improvement | 0%    | 0%    | 1%    | 16%  |
> +-------------+-------+-------+-------+------+
> | v26-0001    | 41293 | 21226 | 10814 | 6158 |
> +-------------+-------+-------+-------+------+
> | improvement | 0%    | 1%    | 1%    | 15%  |
> +-------------+-------+-------+-------+------+
> | HEAD        | 41503 | 21466 | 10943 | 7292 |
> +-------------+-------+-------+-------+------+
>
>
> Again, I couldn't reproduce the cases where you saw significantly degraded performance.
>

I am not surprised to see that you don't see regression because as per
Vignesh's analysis, this is purely a timing issue where sometimes
after the patch the slot creation can take more time because there is
a constant inflow of transactions on the publisher. I think we are
seeing it because this workload is predominantly just creating and
destroying slots. We can probably improve it later as discussed
earlier by using a single for multiple copies (especially for small
tables) or something like that.

-- 
With Regards,
Amit Kapila.

  158. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-14T10:07:43Z

    On Thu, Aug 10, 2023 at 10:15 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 9, 2023 at 8:28 AM Zhijie Hou (Fujitsu)
> <houzj.fnst@fujitsu.com> wrote:
> >
> > On Thursday, August 3, 2023 7:30 PM Melih Mutlu <m.melihmutlu@gmail.com>  wrote:
> >
> > > Right. I attached the v26 as you asked.
> >
> > Thanks for posting the patches.
> >
> > While reviewing the patch, I noticed one rare case that it's possible that there
> > are two table sync worker for the same table in the same time.
> >
> > The patch relies on LogicalRepWorkerLock to prevent concurrent access, but the
> > apply worker will start a new worker after releasing the lock. So, at the point[1]
> > where the lock is released and the new table sync worker has not been started,
> > it seems possible that another old table sync worker will be reused for the
> > same table.
> >
> >                                 /* Now safe to release the LWLock */
> >                                 LWLockRelease(LogicalRepWorkerLock);
> > *[1]
> >                                 /*
> >                                  * If there are free sync worker slot(s), start a new sync
> >                                  * worker for the table.
> >                                  */
> >                                 if (nsyncworkers < max_sync_workers_per_subscription)
> >                                 ...
> >                                                 logicalrep_worker_launch(MyLogicalRepWorker->dbid,
> >
>
> Yeah, this is a problem. I think one idea to solve this is by
> extending the lock duration till we launch the tablesync worker but we
> should also consider changing this locking scheme such that there is a
> better way to indicate that for a particular rel, tablesync is in
> progress. Currently, the code in TablesyncWorkerMain() also acquires
> the lock in exclusive mode even though the tablesync for a rel is in
> progress which I guess could easily heart us for larger values of
> max_logical_replication_workers. So, that could be another motivation
> to think for a different locking scheme.
>

Yet another problem is that currently apply worker maintains a hash
table for 'last_start_times' to avoid restarting the tablesync worker
immediately upon error. The same functionality is missing while
reusing the table sync worker. One possibility is to use a shared hash
table to remember start times but I think it may depend on what we
decide to solve the previous problem reported by Hou-San.

-- 
With Regards,
Amit Kapila.

  159. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2023-08-14T11:59:30Z

    On Thu, 10 Aug 2023 at 10:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 9, 2023 at 8:28 AM Zhijie Hou (Fujitsu)
> <houzj.fnst@fujitsu.com> wrote:
> >
> > On Thursday, August 3, 2023 7:30 PM Melih Mutlu <m.melihmutlu@gmail.com>  wrote:
> >
> > > Right. I attached the v26 as you asked.
> >
> > Thanks for posting the patches.
> >
> > While reviewing the patch, I noticed one rare case that it's possible that there
> > are two table sync worker for the same table in the same time.
> >
> > The patch relies on LogicalRepWorkerLock to prevent concurrent access, but the
> > apply worker will start a new worker after releasing the lock. So, at the point[1]
> > where the lock is released and the new table sync worker has not been started,
> > it seems possible that another old table sync worker will be reused for the
> > same table.
> >
> >                                 /* Now safe to release the LWLock */
> >                                 LWLockRelease(LogicalRepWorkerLock);
> > *[1]
> >                                 /*
> >                                  * If there are free sync worker slot(s), start a new sync
> >                                  * worker for the table.
> >                                  */
> >                                 if (nsyncworkers < max_sync_workers_per_subscription)
> >                                 ...
> >                                                 logicalrep_worker_launch(MyLogicalRepWorker->dbid,
> >
>
> Yeah, this is a problem. I think one idea to solve this is by
> extending the lock duration till we launch the tablesync worker but we
> should also consider changing this locking scheme such that there is a
> better way to indicate that for a particular rel, tablesync is in
> progress. Currently, the code in TablesyncWorkerMain() also acquires
> the lock in exclusive mode even though the tablesync for a rel is in
> progress which I guess could easily heart us for larger values of
> max_logical_replication_workers. So, that could be another motivation
> to think for a different locking scheme.

There are couple of ways in which this issue can be solved:
Approach #1) check that the reuse worker has not picked up this table
for table sync from logicalrep_worker_launch while holding a lock on
LogicalRepWorkerLock, if the reuse worker has already picked it up for
processing, simply ignore it and return, nothing has to be done by the
launcher in this case.
Approach #2) a) Applyworker to create a shared memory of all the
relations that need to be synced, b) tablesync worker to take a lock
on this shared memory and pick the next table to be
processed(tablesync worker need not get the subscription relations
again and again) c) tablesync worker to update the status in shared
memory for the relation(since the lock is held there will be no
concurrency issues), also mark the start time in the shared memory,
this will help in not to restart the failed table before
wal_retrieve_retry_interval has expired d) tablesync worker to sync
the table e) subscription relation will be marked as ready and the
tablesync worker to remove the entry from shared memory f) Applyworker
will periodically synchronize the shared memory relations to keep it
in sync with the fetched subscription relation tables  g) when apply
worker exits, the shared memory will be cleared.

Approach #2) will also help in solving the other issue reported by Amit at [1].
I felt we can use Approach #2 to solve the problem as it solves both
the reported issues and also there is an added advantage where the
re-use table sync worker need not scan the pg_subscription_rel to get
the non-ready table for every run, instead we can use the list
prepared by apply worker.
Thoughts?

[1] - https://www.postgresql.org/message-id/CAA4eK1KyHfVOVeio28p8CHDnuyKuej78cj_7U9mHAB4ictVQwQ%40mail.gmail.com

Regards,
Vignesh

  160. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-15T23:18:02Z

    Here is another review comment about patch v26-0001.

The tablesync worker processes include the 'relid' as part of their
name. See launcher.c:

snprintf(bgw.bgw_name, BGW_MAXLEN,
    "logical replication tablesync worker for subscription %u sync %u",
    subid,
    relid);

~~

And if that worker is "reused" by v26-0001 to process another relation
there is a LOG

if (reuse_worker)
    ereport(LOG,
        errmsg("logical replication table synchronization worker for
subscription \"%s\" will be reused to sync table \"%s\" with relid
%u.",
            MySubscription->name,
            get_rel_name(MyLogicalRepWorker->relid),
            MyLogicalRepWorker->relid));


AFAICT, when being "reused" the original process name remains
unchanged, and so I think it will continue to appear to any user
looking at it that the tablesync process is just taking a very long
time handling the original 'relid'.

Won't the stale process name cause confusion to the users?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  161. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-16T03:53:47Z

    On Fri, Aug 11, 2023 at 11:45 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Again, I couldn't reproduce the cases where you saw significantly degraded performance. I wonder if I'm missing something. Did you do anything not included in the test scripts you shared? Do you think v26-0001 will perform 84% worse than HEAD, if you try again? I just want to be sure that it was not a random thing.
> Interestingly, I also don't see an improvement in above results as big as in your results when inserts/tx ratio is smaller. Even though it certainly is improved in such cases.
>

TEST ENVIRONMENTS

I am running the tests on a high-spec machine:

-- NOTE: Nobody else is using this machine during our testing, so
there are no unexpected influences messing up the results.


Linix

Architecture:          x86_64
CPU(s):                120
Thread(s) per core:    2
Core(s) per socket:    15

              total        used        free      shared  buff/cache   available
Mem:           755G        5.7G        737G         49M         12G        748G
Swap:          4.0G          0B        4.0G

~~~

The results I am seeing are not random. HEAD+v26-0001 is consistently
worse than HEAD but only for some settings. With these settings, I see
bad results (i.e. worse than HEAD) consistently every time using the
dedicated test machine.

Hou-san also reproduced bad results using a different high-spec machine

Vignesh also reproduced bad results using just his laptop but in his
case, it did *not* occur every time. As discussed elsewhere the
problem is timing-related, so sometimes you may be lucky and sometimes
not.

~

I expect you are running everything correctly, but if you are using
just a laptop (like Vignesh) then like him you might need to try
multiple times before you can hit the problem happening in your
environment.

Anyway, in case there is some other reason you are not seeing the bad
results I have re-attached scripts and re-described every step below.

======

BUILDING

-- NOTE: I have a very minimal configuration without any
optimization/debug flags etc. See config.log

$ ./configure --prefix=/home/peter/pg_oss

-- NOTE: Of course, make sure to be running using the correct Postgres:

echo 'set environment variables for OSS work'
export PATH=/home/peter/pg_oss/bin:$PATH

-- NOTE: Be sure to do git stash or whatever so don't accidentally
build a patched version thinking it is the HEAD version
-- NOTE: Be sure to do a full clean build and apply (or don't apply
v26-0001) according to the test you wish to run.

STEPS
1. sudo make clean
2. make
3. sudo make install

======

SCRIPTS & STEPS

SCRIPTS
testrun.sh
do_one_test_setup.sh
do_one_test_PUB.sh
do_one_test_SUB.sh

---

STEPS

Step-1. Edit the testrun.sh

tables=( 100 )
workers=( 2 4 8 16 )
size="0"
prefix="0816headbusy" <-- edit to differentiate each test run

~

Step-2. Edit the do_one_test_PUB.sh
IF commit_counter = 1000 THEN <-- edit this if needed. I wanted 1000
inserts/tx so nothing to do

~

Step-3: Check nothing else is running. If yes, then clean it up
[peter@localhost testing_busy]$ ps -eaf | grep postgres
peter    111924 100103  0 19:31 pts/0    00:00:00 grep --color=auto postgres

~

Step-4: Run the tests
[peter@localhost testing_busy]$ ./testrun.sh
num_tables=100, size=0, num_workers=2, run #1 <-- check the echo
matched the config you set in the Setp-1
waiting for server to shut down.... done
server stopped
waiting for server to shut down.... done
server stopped
num_tables=100, size=0, num_workers=2, run #2
waiting for server to shut down.... done
server stopped
waiting for server to shut down.... done
server stopped
num_tables=100, size=0, num_workers=2, run #3
...

~

Step-5: Sanity check
When the test completes the current folder will be full of .log and .dat* files.
Check for sanity that no errors happened

[peter@localhost testing_busy]$ cat *.log | grep ERROR
[peter@localhost testing_busy]$

~

Step-6: Collect the results
The results are output (by the do_one_test_SUB.sh) into the *.dat_SUB files
Use grep to extract them

[peter@localhost testing_busy]$ cat 0816headbusy_100t_0_2w_*.dat_SUB |
grep RESULT | grep -v duration | awk '{print $3}'
11742.019
12157.355
11773.807
11582.981
12220.962
12546.325
12210.713
12614.892
12015.489
13527.05

Repeat grep for other files:
$ cat 0816headbusy_100t_0_4w_*.dat_SUB | grep RESULT | grep -v
duration | awk '{print $3}'
$ cat 0816headbusy_100t_0_8w_*.dat_SUB | grep RESULT | grep -v
duration | awk '{print $3}'
$ cat 0816headbusy_100t_0_16w_*.dat_SUB | grep RESULT | grep -v
duration | awk '{print $3}'

~

Step-7: Summarise the results
Now I just cut/paste the results from Step-6 into a spreadsheet and
report the median of the runs.

For example, for the above HEAD run, it was:
             2w        4w       8w      16w
1           11742   5996   1919   1582
2           12157   5960   1871   1469
3           11774   5926   2101   1571
4           11583   6155   1883   1671
5           12221   6310   1895   1707
6           12546   6166   1900   1470
7           12211   6114   2477   1587
8           12615   6173   2610   1715
9           12015   5869   2110   1673
10         13527   5913   2144   1227
Median 12184   6055   2010   1584

~

Step-8: REPEAT
-- repeat all above for different size transactions (editing do_one_test_PUB.sh)
-- repeat all above after rebuilding again with HEAD+v26-0001

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  162. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-21T07:56:27Z

    Hi Melih,

Last week we revisited your implementation of design#2. Vignesh rebased it,
and then made a few other changes.

PSA v28*

The patch changes include:
* changed the logic slightly by setting recv_immediately(new variable), if
this variable is set the main apply worker loop will not wait in this case.
* setting the relation state to ready immediately if there are no more
incremental changes to be synced.
* receive the incremental changes if applicable and set the relation state
to ready without waiting.
* reuse the worker if the worker is free before trying to start a new table
sync worker
* restarting the tablesync worker only after wal_retrieve_retry_interval

~

FWIW, we just wanted to share with you the performance measurements seen
using this design#2 patch set:

======

RESULTS (not busy tests)

------
10 empty tables
                2w      4w      8w      16w
HEAD:           125     119     140     133
HEAD+v28*:      92      93      123     134
%improvement:   27%     22%     12%     -1%
------
100 empty tables
                2w      4w      8w      16w
HEAD:           1037    843     1109    1155
HEAD+v28*:      591     625     2616    2569
%improvement:   43%     26%     -136%   -122%
------
1000 empty tables
                2w      4w      8w      16w
HEAD:           15874   10047   9919    10338
HEAD+v28*:      33673   12199   9094    9896
%improvement:   -112%   -21%    8%      4%
------
2000 empty tables
                2w      4w      8w      16w
HEAD:           45266   24216   19395   19820
HEAD+v28*:      88043   21550   21668   22607
%improvement:  -95%     11%     -12%    -14%

~~~

Note - the results were varying quite a lot in comparison to the HEAD
e.g. HEAD results are very consistent, but the v28* results observed are not
HEAD 1000 (2w): 15861, 15777, 16007, 15950, 15886, 15740, 15846, 15740,
15908, 15940
v28* 1000 (2w):  34214, 13679, 8792, 33289, 31976, 56071, 57042, 56163,
34058, 11969

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  163. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Peter Smith <smithpb2250@gmail.com> — 2023-08-21T07:58:25Z

    Oops - now with attachments

On Mon, Aug 21, 2023 at 5:56 PM Peter Smith <smithpb2250@gmail.com> wrote:

> Hi Melih,
>
> Last week we revisited your implementation of design#2. Vignesh rebased
> it, and then made a few other changes.
>
> PSA v28*
>
> The patch changes include:
> * changed the logic slightly by setting recv_immediately(new variable), if
> this variable is set the main apply worker loop will not wait in this case.
> * setting the relation state to ready immediately if there are no more
> incremental changes to be synced.
> * receive the incremental changes if applicable and set the relation state
> to ready without waiting.
> * reuse the worker if the worker is free before trying to start a new
> table sync worker
> * restarting the tablesync worker only after wal_retrieve_retry_interval
>
> ~
>
> FWIW, we just wanted to share with you the performance measurements seen
> using this design#2 patch set:
>
> ======
>
> RESULTS (not busy tests)
>
> ------
> 10 empty tables
>                 2w      4w      8w      16w
> HEAD:           125     119     140     133
> HEAD+v28*:      92      93      123     134
> %improvement:   27%     22%     12%     -1%
> ------
> 100 empty tables
>                 2w      4w      8w      16w
> HEAD:           1037    843     1109    1155
> HEAD+v28*:      591     625     2616    2569
> %improvement:   43%     26%     -136%   -122%
> ------
> 1000 empty tables
>                 2w      4w      8w      16w
> HEAD:           15874   10047   9919    10338
> HEAD+v28*:      33673   12199   9094    9896
> %improvement:   -112%   -21%    8%      4%
> ------
> 2000 empty tables
>                 2w      4w      8w      16w
> HEAD:           45266   24216   19395   19820
> HEAD+v28*:      88043   21550   21668   22607
> %improvement:  -95%     11%     -12%    -14%
>
> ~~~
>
> Note - the results were varying quite a lot in comparison to the HEAD
> e.g. HEAD results are very consistent, but the v28* results observed are
> not
> HEAD 1000 (2w): 15861, 15777, 16007, 15950, 15886, 15740, 15846, 15740,
> 15908, 15940
> v28* 1000 (2w):  34214, 13679, 8792, 33289, 31976, 56071, 57042, 56163,
> 34058, 11969
>
> ------
> Kind Regards,
> Peter Smith.
> Fujitsu Australia
>

  164. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-01-10T09:29:22Z

    Hi,

This patch is not applying on the HEAD. Please rebase and share the
updated patch.

Thanks and Regards
Shlok Kyal

On Wed, 10 Jan 2024 at 14:55, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Oops - now with attachments
>
> On Mon, Aug 21, 2023 at 5:56 PM Peter Smith <smithpb2250@gmail.com> wrote:
>>
>> Hi Melih,
>>
>> Last week we revisited your implementation of design#2. Vignesh rebased it, and then made a few other changes.
>>
>> PSA v28*
>>
>> The patch changes include:
>> * changed the logic slightly by setting recv_immediately(new variable), if this variable is set the main apply worker loop will not wait in this case.
>> * setting the relation state to ready immediately if there are no more incremental changes to be synced.
>> * receive the incremental changes if applicable and set the relation state to ready without waiting.
>> * reuse the worker if the worker is free before trying to start a new table sync worker
>> * restarting the tablesync worker only after wal_retrieve_retry_interval
>>
>> ~
>>
>> FWIW, we just wanted to share with you the performance measurements seen using this design#2 patch set:
>>
>> ======
>>
>> RESULTS (not busy tests)
>>
>> ------
>> 10 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           125     119     140     133
>> HEAD+v28*:      92      93      123     134
>> %improvement:   27%     22%     12%     -1%
>> ------
>> 100 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           1037    843     1109    1155
>> HEAD+v28*:      591     625     2616    2569
>> %improvement:   43%     26%     -136%   -122%
>> ------
>> 1000 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           15874   10047   9919    10338
>> HEAD+v28*:      33673   12199   9094    9896
>> %improvement:   -112%   -21%    8%      4%
>> ------
>> 2000 empty tables
>>                 2w      4w      8w      16w
>> HEAD:           45266   24216   19395   19820
>> HEAD+v28*:      88043   21550   21668   22607
>> %improvement:  -95%     11%     -12%    -14%
>>
>> ~~~
>>
>> Note - the results were varying quite a lot in comparison to the HEAD
>> e.g. HEAD results are very consistent, but the v28* results observed are not
>> HEAD 1000 (2w): 15861, 15777, 16007, 15950, 15886, 15740, 15846, 15740, 15908, 15940
>> v28* 1000 (2w):  34214, 13679, 8792, 33289, 31976, 56071, 57042, 56163, 34058, 11969
>>
>> ------
>> Kind Regards,
>> Peter Smith.
>> Fujitsu Australia

  165. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    Amit Kapila <amit.kapila16@gmail.com> — 2024-01-10T09:34:15Z

    On Wed, Jan 10, 2024 at 2:59 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> This patch is not applying on the HEAD. Please rebase and share the
> updated patch.
>

IIRC, there were some regressions observed with this patch. So, one
needs to analyze those as well. I think we should mark it as "Returned
with feedback".

-- 
With Regards,
Amit Kapila.

  166. Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

    vignesh C <vignesh21@gmail.com> — 2024-01-10T13:10:12Z

    On Wed, 10 Jan 2024 at 15:04, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 10, 2024 at 2:59 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > This patch is not applying on the HEAD. Please rebase and share the
> > updated patch.
> >
>
> IIRC, there were some regressions observed with this patch. So, one
> needs to analyze those as well. I think we should mark it as "Returned
> with feedback".

Thanks, I have updated the status to "Returned with feedback".
Feel free to post an updated version with the fix for the regression
and start a new entry for the same.

Regards,
Vignesh