Horton

Thread

Commits

  1. Improved CREATE SUBSCRIPTION message for clarity

  2. Fix the test case introduced by commit 8756930190.

  3. Raise a warning if there is a possibility of data from multiple origins.

  4. Add wait_for_subscription_sync for TAP tests.

  5. Bump catversion for commit d8cd0c6c95c0120168df93aae095df4e0682a08a.

  6. Allow users to skip logical replication of data having origin.

  7. Improve two comments related to a boolean DefElem's value

  8. doc: Fix man page whitespace issues

  1. Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-02-23T06:29:02Z

     Hi,

In logical replication, currently Walsender sends the data that is
generated locally and the data that are replicated from other
instances. This results in infinite recursion in circular logical
replication setup.
Here the user is trying to have a 2-way replication setup with node 1
publishing data to node2 and node2 publishing data to node1, so that
the user can perform dml operations from any node, it can act as a
2-way multi master replication setup.

This problem can be reproduced with the following steps:
-- Instance 1
create publication pub1 for table t1;
create table t1(c1 int);

-- Instance 2
create table t1(c1 int);
create publication pub2 for table t1;
create subscription sub1 CONNECTION 'dbname=postgres port=5432'
publication pub1;

-- Instance 1
create subscription sub2 CONNECTION 'dbname=postgres port=5433'
publication pub2; insert into t1 values(10);

In this scenario, the Walsender in publisher pub1 sends data to the
apply worker in subscriber sub1, the apply worker in sub1 maps the
data to local tables and applies the individual changes as they are
received. Then the Walsender in publisher pub2 sends data to the apply
worker in subscriber sub2, the apply worker in sub2 maps the data to
local tables and applies the individual changes as they are received.
This process repeats infinitely.

Currently we do not differentiate if the data is locally generated
data, or a replicated data and we send both the data which causes
infinite recursion.

We could see that the record count has increased significantly within sometime:
select count(*) from t1;
  count
--------------
 4000000
(1 row)

If the table had primary key constraint, we could notice that the
first insert is successful and when the same insert is sent back, the
insert fails because of constraint error:
2022-02-23 09:28:43.592 IST [14743] ERROR:  duplicate key value
violates unique constraint "t1_pkey"
2022-02-23 09:28:43.592 IST [14743] DETAIL:  Key (c1)=(10) already exists.
2022-02-23 09:28:43.592 IST [14743] CONTEXT:  processing remote data
during "INSERT" for replication target relation "public.t1" in
transaction 727 at 2022-02-23 09:28:43.406738+05:30
2022-02-23 09:28:43.593 IST [14678] LOG:  background worker "logical
replication worker" (PID 14743) exited with exit code 1
2022-02-23 09:28:48.608 IST [14745] LOG:  logical replication apply
worker for subscription "sub2" has started
2022-02-23 09:28:48.624 IST [14745] ERROR:  duplicate key value
violates unique constraint "t1_pkey"
2022-02-23 09:28:48.624 IST [14745] DETAIL:  Key (c1)=(10) already exists.
2022-02-23 09:28:48.624 IST [14745] CONTEXT:  processing remote data
during "INSERT" for replication target relation "public.t1" in
transaction 727 at 2022-02-23 09:28:43.406738+05:30
2022-02-23 09:28:48.626 IST [14678] LOG:  background worker "logical
replication worker" (PID 14745) exited with exit code 1

The same problem can occur in any circular node setup like 3 nodes,
4node etc like: a) node1 publishing to node2 b) node2 publishing to
node3 c) node3 publishing back to node1.

Here there are two problems for the user: a) incremental
synchronization of table sending both local data and replicated data
by walsender b) Table synchronization of table using copy command
sending both local data and replicated data

For the first problem "Incremental synchronization of table by
Walsender" can be solved by:
Currently the locally generated data does not have replication origin
associated and the data that has originated from another instance will
have a replication origin associated. We could use this information to
differentiate locally generated data and replicated data and send only
the locally generated data. This "only_local" could be provided as an
option while subscription is created:
ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
PUBLICATION pub1 with (only_local = on);

I have attached a basic patch for this, if the idea is accepted, I
will work further to test more scenarios, add documentation, and test
and post an updated patch.
For the second problem, Table synchronization of table including local
data and replicated data using copy command.

Let us consider the following scenario:
a) node1 publishing to node2 b) node2 publishing to node1. Here in
this case node1 will have replicated data from node2 and vice versa.

In the above if user wants to include node3 to subscribe data from
node2. Users will have to create a subscription in node3 to get the
data from node2. During table synchronization we send the complete
table data from node2 to node3. Node2 will have local data from node2
and also replicated data from node1. Currently we don't have an option
to differentiate between the locally generated data and replicated
data in the heap which will cause infinite recursion as described
above.

To handle this if user has specified only_local option, we could throw
a warning or error out while creating subscription in this case, we
could have a column srreplicateddata in pg_subscription_rel which
could indicate if the table has any replicated data or not:
postgres=# select * from pg_subscription_rel;
 srsubid | srrelid | srsubstate | srsublsn  | srreplicateddata
---------+---------+------------+-----------+------------------
   16389 |   16384 | r          | 0/14A4640 |        t
   16389 |   16385 | r          | 0/14A4690 |        f
(1 row)

In the above example, srreplicateddata with true indicates, tabel t1
whose relid is 16384 has replicated data and the other row having
srreplicateddata  as false indicates table t2 whose relid is 16385
does not have replicated data.
When creating a new subscription, the subscriber will connect to the
publisher and check if the relation has replicated data by checking
srreplicateddata in pg_subscription_rel table.
If the table has any replicated data, log a warning or error for this.

Also, we could document the steps on how to handle the initial sync like:
a) Complete the ongoing transactions on this table in the replication
setup nodes i.e. node1 and node2 in the above case,  so that the table
data is consistent, b) Once there are no ongoing transaction, Copy the
table data using copy command from any one of the nodes, c) create
subscription with copy_data option as off d) Perform further
transactions on the table e) All the further transactions performed
will be handled by the walsender which will take care of skipping
replicated data and sending only the local data. i.e. node2 will send
the locally generated data to node3.

I'm not sure if there is any other better way to handle this. If there
is a better way, we could handle it accordingly.
Thoughts?

Regards,
Vignesh

  2. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-03-01T10:42:27Z

    Hi Vignesh,

> In logical replication, currently Walsender sends the data that is
> generated locally and the data that are replicated from other
> instances. This results in infinite recursion in circular logical
> replication setup.

Thank you for good explanation. I understand that this fix can be used
for a bidirectional replication.

> Here there are two problems for the user: a) incremental
> synchronization of table sending both local data and replicated data
> by walsender b) Table synchronization of table using copy command
> sending both local data and replicated data

So you wanted to solve these two problem and currently focused on
the first one, right? We can check one by one.

> For the first problem "Incremental synchronization of table by
> Walsender" can be solved by:
> Currently the locally generated data does not have replication origin
> associated and the data that has originated from another instance will
> have a replication origin associated. We could use this information to
> differentiate locally generated data and replicated data and send only
> the locally generated data. This "only_local" could be provided as an
> option while subscription is created:
> ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> PUBLICATION pub1 with (only_local = on);

Sounds good, but I cannot distinguish whether the assumption will keep.

I played with your patch, but it could not be applied to current master.
I tested from bd74c40 and I confirmed infinite loop was not appeared.

local_only could not be set from ALTER SUBSCRIPTION command.
Is it expected?

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  3. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-02T03:26:57Z

    On Tue, Mar 1, 2022 at 4:12 PM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Hi Vignesh,
>
> > In logical replication, currently Walsender sends the data that is
> > generated locally and the data that are replicated from other
> > instances. This results in infinite recursion in circular logical
> > replication setup.
>
> Thank you for good explanation. I understand that this fix can be used
> for a bidirectional replication.

Once these issues are resolved, it can be used for bi-directional
logical replication.

> > Here there are two problems for the user: a) incremental
> > synchronization of table sending both local data and replicated data
> > by walsender b) Table synchronization of table using copy command
> > sending both local data and replicated data
>
> So you wanted to solve these two problem and currently focused on
> the first one, right? We can check one by one.

Yes.

> > For the first problem "Incremental synchronization of table by
> > Walsender" can be solved by:
> > Currently the locally generated data does not have replication origin
> > associated and the data that has originated from another instance will
> > have a replication origin associated. We could use this information to
> > differentiate locally generated data and replicated data and send only
> > the locally generated data. This "only_local" could be provided as an
> > option while subscription is created:
> > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > PUBLICATION pub1 with (only_local = on);
>
> Sounds good, but I cannot distinguish whether the assumption will keep.

Replication origin is created by the apply worker and it will be used
for all the transactions received from the walsender. I feel the
replication origin will be present always.

> I played with your patch, but it could not be applied to current master.
> I tested from bd74c40 and I confirmed infinite loop was not appeared.

I will post an updated version for this soon.

> local_only could not be set from ALTER SUBSCRIPTION command.
> Is it expected?

I wanted to get the opinion from others too just to make sure the
approach is right. I will fix this including the documentation, test,
etc in the later versions.

Regards,
Vignesh

  4. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-02T10:28:55Z

    On Wed, Feb 23, 2022 at 11:59 AM vignesh C <vignesh21@gmail.com> wrote:
>
...
...
> I have attached a basic patch for this, if the idea is accepted, I
> will work further to test more scenarios, add documentation, and test
> and post an updated patch.
> For the second problem, Table synchronization of table including local
> data and replicated data using copy command.
>
> Let us consider the following scenario:
> a) node1 publishing to node2 b) node2 publishing to node1. Here in
> this case node1 will have replicated data from node2 and vice versa.
>
> In the above if user wants to include node3 to subscribe data from
> node2. Users will have to create a subscription in node3 to get the
> data from node2. During table synchronization we send the complete
> table data from node2 to node3. Node2 will have local data from node2
> and also replicated data from node1. Currently we don't have an option
> to differentiate between the locally generated data and replicated
> data in the heap which will cause infinite recursion as described
> above.
>
> To handle this if user has specified only_local option, we could throw
> a warning or error out while creating subscription in this case, we
> could have a column srreplicateddata in pg_subscription_rel which
> could indicate if the table has any replicated data or not:
> postgres=# select * from pg_subscription_rel;
>  srsubid | srrelid | srsubstate | srsublsn  | srreplicateddata
> ---------+---------+------------+-----------+------------------
>    16389 |   16384 | r          | 0/14A4640 |        t
>    16389 |   16385 | r          | 0/14A4690 |        f
> (1 row)
>
> In the above example, srreplicateddata with true indicates, tabel t1
> whose relid is 16384 has replicated data and the other row having
> srreplicateddata  as false indicates table t2 whose relid is 16385
> does not have replicated data.
> When creating a new subscription, the subscriber will connect to the
> publisher and check if the relation has replicated data by checking
> srreplicateddata in pg_subscription_rel table.
> If the table has any replicated data, log a warning or error for this.
>

If you want to give the error in this case, then I think we need to
provide an option to the user to allow copy. One possibility could be
to extend existing copy_data option as 'false', 'true', 'force'. For
'false', there shouldn't be any change, for 'true', if 'only_local'
option is also set and the new column indicates replicated data then
give an error, for 'force', we won't give an error even if the
conditions as mentioned for 'true' case are met, rather we will allow
copy in this case.

> Also, we could document the steps on how to handle the initial sync like:
> a) Complete the ongoing transactions on this table in the replication
> setup nodes i.e. node1 and node2 in the above case,  so that the table
> data is consistent, b) Once there are no ongoing transaction, Copy the
> table data using copy command from any one of the nodes, c) create
> subscription with copy_data option as off d) Perform further
> transactions on the table e) All the further transactions performed
> will be handled by the walsender which will take care of skipping
> replicated data and sending only the local data. i.e. node2 will send
> the locally generated data to node3.
>
> I'm not sure if there is any other better way to handle this.
>

I could think of the below options for users to set up bi-directional
replication for the same table.

Option-1:
There is no pre-existing data in the tables that are going to
participate in bi-directional replication. In such a case, Users can
create pub/sub (with only_local option as proposed by you) on both
nodes before starting any writes on tables. This will allow
bi-directional replication for the required tables. Now, if the user
wants one of the nodes to join at a later point, then the strategy in
Option-2/3 could be used.

Option-2:
One of the nodes (say node-1) has some pre-existing data and another
node (say node-2) doesn't have any pre-existing data. In this case,
the user can set up pub/sub (with only_local and copy_data as 'false'
options) for node-1 first before any of the operations on node-2.
Then, it can set up pub/sub on node-2. This will allow bi-directional
replication for the required tables.

Option-3:
Both the nodes have some pre-existing data. I think the easiest option
could be to truncate data on one of the nodes and set up pub/sub on
both nodes. See, one way to achieve it among two nodes as below:

Node-1:
Table t1 has data
1, 2, 3, 4

Publication for t1, pub1: Create Publication pub1 For Table t1;

Node-2:
Table t1 has data
5, 6, 7, 8
Publication for t1, pub1_2: Create Publication pub1_2 For Table t1;

Now, Create Subscription for pub1 on node1: Create Subscription sub1_2
Connection '<node-1 details>' Publication pub1 WITH (only_local =
true);

Node-1:
Begin;
# Disallow truncates to be published
Alter Publication pub1 Set (publish = 'insert, update, delete');
Truncate t1;
Create Subscription sub1 Connection '<node-2 details>' Publication
pub1 WITH (only_local = true, copy_data = 'force');
Alter Publication pub1 Set (publish = 'insert, update, delete, truncate');
Commit;

I think this will allow the bi-directional replication between two
nodes. In this scheme, the user needs to manually perform some steps
including truncate of the table on one of the nodes which she might or
might not like but at least there will be a way to set up a
bi-directional replication on two nodes for same table operations
which is not possible now.

I think one can even imagine using and extending this functionality so
that users don't need to perform TRUNCATE on one of the nodes. Say, in
the above case for tablesync phase, we make both nodes to start a
transaction, create a slot on another node (with USE_SNAPSHOT option),
and then allow copy from another node. I think it will be important to
allow copy on each node once the slots are created and the initial
snapshot is established.

For more than two nodes, I think we can suggest having either of the
option-1 or 2 for setup. But, there could be other ways as well
depending on how the user wants to do the setup.

Thoughts?

-- 
With Regards,
Amit Kapila.

  5. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-02T16:12:11Z

    On Tue, Mar 1, 2022 at 4:12 PM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Hi Vignesh,
>
> > In logical replication, currently Walsender sends the data that is
> > generated locally and the data that are replicated from other
> > instances. This results in infinite recursion in circular logical
> > replication setup.
>
> Thank you for good explanation. I understand that this fix can be used
> for a bidirectional replication.
>
> > Here there are two problems for the user: a) incremental
> > synchronization of table sending both local data and replicated data
> > by walsender b) Table synchronization of table using copy command
> > sending both local data and replicated data
>
> So you wanted to solve these two problem and currently focused on
> the first one, right? We can check one by one.
>
> > For the first problem "Incremental synchronization of table by
> > Walsender" can be solved by:
> > Currently the locally generated data does not have replication origin
> > associated and the data that has originated from another instance will
> > have a replication origin associated. We could use this information to
> > differentiate locally generated data and replicated data and send only
> > the locally generated data. This "only_local" could be provided as an
> > option while subscription is created:
> > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > PUBLICATION pub1 with (only_local = on);
>
> Sounds good, but I cannot distinguish whether the assumption will keep.
>
> I played with your patch, but it could not be applied to current master.
> I tested from bd74c40 and I confirmed infinite loop was not appeared.
Rebased the patch on top of head

> local_only could not be set from ALTER SUBSCRIPTION command.
> Is it expected?
Modified

Thanks for the comments, the attached patch has the changes for the same.

Regards,
Vignesh

  6. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T03:37:02Z

    FYI, the v2 patch did not apply to HEAD

[postgres@CentOS7-x64 oss_postgres_misc]$ git apply
../patches_misc/v2-0001-Skip-replication-of-non-local-data.patch
--verbose
...
error: patch failed: src/backend/replication/slotfuncs.c:231
error: src/backend/replication/slotfuncs.c: patch does not apply

------
Kind Regards,
Peter Smith.
Fujitsu Australia.

  7. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-03-07T03:55:33Z

    On Wed, Feb 23, 2022 at 11:59 AM vignesh C <vignesh21@gmail.com> wrote:
> Here there are two problems for the user: a) incremental
> synchronization of table sending both local data and replicated data
> by walsender b) Table synchronization of table using copy command
> sending both local data and replicated data
>
> For the first problem "Incremental synchronization of table by
> Walsender" can be solved by:
> Currently the locally generated data does not have replication origin
> associated and the data that has originated from another instance will
> have a replication origin associated. We could use this information to
> differentiate locally generated data and replicated data and send only
> the locally generated data. This "only_local" could be provided as an
> option while subscription is created:
> ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> PUBLICATION pub1 with (only_local = on);

I haven't yet gone through the patch, but I have a question about the
idea.  Suppose I want to set up a logical replication like,
node1->node2->node3->node1.  So how would I create the subscriber at
node1?  only_local=on or off?.  I mean on node1, I want the changes
from node3 which are generated on node3 or which are replicated from
node2 but I do not want changes that are replicated from node1 itself?
 So if I set only_local=on then node1 will not get the changes
replicated from node2, is that right? and If I set only_local=off then
it will create the infinite loop again?  So how are we protecting
against this case?

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  8. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-07T04:35:09Z

    On Mon, Mar 7, 2022 at 9:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Feb 23, 2022 at 11:59 AM vignesh C <vignesh21@gmail.com> wrote:
> > Here there are two problems for the user: a) incremental
> > synchronization of table sending both local data and replicated data
> > by walsender b) Table synchronization of table using copy command
> > sending both local data and replicated data
> >
> > For the first problem "Incremental synchronization of table by
> > Walsender" can be solved by:
> > Currently the locally generated data does not have replication origin
> > associated and the data that has originated from another instance will
> > have a replication origin associated. We could use this information to
> > differentiate locally generated data and replicated data and send only
> > the locally generated data. This "only_local" could be provided as an
> > option while subscription is created:
> > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > PUBLICATION pub1 with (only_local = on);
>
> I haven't yet gone through the patch, but I have a question about the
> idea.  Suppose I want to set up a logical replication like,
> node1->node2->node3->node1.  So how would I create the subscriber at
> node1?  only_local=on or off?.  I mean on node1, I want the changes
> from node3 which are generated on node3 or which are replicated from
> node2 but I do not want changes that are replicated from node1 itself?
>  So if I set only_local=on then node1 will not get the changes
> replicated from node2, is that right? and If I set only_local=off then
> it will create the infinite loop again?  So how are we protecting
> against this case?
>

In the above topology if you want local changes from both node3 and
node2 then I think the way to get that would be you have to create two
subscriptions on node1. The first one points to node2 (with
only_local=off) and the second one points to node3 (with only_local
=off). Will that address your case or am I missing something?

-- 
With Regards,
Amit Kapila.

  9. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-07T04:45:46Z

    On Mon, Mar 7, 2022 at 10:05 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 9:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Feb 23, 2022 at 11:59 AM vignesh C <vignesh21@gmail.com> wrote:
> > > Here there are two problems for the user: a) incremental
> > > synchronization of table sending both local data and replicated data
> > > by walsender b) Table synchronization of table using copy command
> > > sending both local data and replicated data
> > >
> > > For the first problem "Incremental synchronization of table by
> > > Walsender" can be solved by:
> > > Currently the locally generated data does not have replication origin
> > > associated and the data that has originated from another instance will
> > > have a replication origin associated. We could use this information to
> > > differentiate locally generated data and replicated data and send only
> > > the locally generated data. This "only_local" could be provided as an
> > > option while subscription is created:
> > > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > > PUBLICATION pub1 with (only_local = on);
> >
> > I haven't yet gone through the patch, but I have a question about the
> > idea.  Suppose I want to set up a logical replication like,
> > node1->node2->node3->node1.  So how would I create the subscriber at
> > node1?  only_local=on or off?.  I mean on node1, I want the changes
> > from node3 which are generated on node3 or which are replicated from
> > node2 but I do not want changes that are replicated from node1 itself?
> >  So if I set only_local=on then node1 will not get the changes
> > replicated from node2, is that right? and If I set only_local=off then
> > it will create the infinite loop again?  So how are we protecting
> > against this case?
> >
>
> In the above topology if you want local changes from both node3 and
> node2 then I think the way to get that would be you have to create two
> subscriptions on node1. The first one points to node2 (with
> only_local=off) and the second one points to node3 (with only_local
> =off).
>

Sorry, I intend to say 'only_local=on' at both places in my previous email.

-- 
With Regards,
Amit Kapila.

  10. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T04:56:25Z

    Hi Vignesh, I also have not looked at the patch yet, but I have what
seems like a very fundamental (and possibly dumb) question...

Basically, I do not understand the choice of syntax for setting things up.

IMO that "only-local" option sounds very similar to the other
PUBLICATION ("publish") options which decide the kinds of things that
will be published. So it feels more natural for me to think of the
publisher as being the one to decide what will be published.

e.g.

option 1:
CREATE PUBLICATION p1 FOR TABLE t1;
CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);

option 2:
CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;

~~

IIUC the patch is using option 1. My first impression was it feels
back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
publish.

So, why does the patch use syntax option 1?

------
Kind Regards,
Peter Smith
Fujitsu Australia

  11. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T05:06:53Z

    On Mon, Mar 7, 2022 at 3:56 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh, I also have not looked at the patch yet, but I have what
> seems like a very fundamental (and possibly dumb) question...
>
> Basically, I do not understand the choice of syntax for setting things up.
>
> IMO that "only-local" option sounds very similar to the other
> PUBLICATION ("publish") options which decide the kinds of things that
> will be published. So it feels more natural for me to think of the
> publisher as being the one to decide what will be published.
>
> e.g.
>
> option 1:
> CREATE PUBLICATION p1 FOR TABLE t1;
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
>
> option 2:
> CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
>

Sorry, I mean to write WITH.

option 2:
CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'only_local');
CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;

> ~~
>
> IIUC the patch is using option 1. My first impression was it feels
> back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> publish.
>
> So, why does the patch use syntax option 1?
>
> ------
> Kind Regards,
> Peter Smith
> Fujitsu Australia

  12. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-07T05:20:35Z

    On Mon, Mar 7, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh, I also have not looked at the patch yet, but I have what
> seems like a very fundamental (and possibly dumb) question...
>
> Basically, I do not understand the choice of syntax for setting things up.
>
> IMO that "only-local" option sounds very similar to the other
> PUBLICATION ("publish") options which decide the kinds of things that
> will be published. So it feels more natural for me to think of the
> publisher as being the one to decide what will be published.
>
> e.g.
>
> option 1:
> CREATE PUBLICATION p1 FOR TABLE t1;
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
>
> option 2:
> CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
>
> ~~
>
> IIUC the patch is using option 1. My first impression was it feels
> back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> publish.
>
> So, why does the patch use syntax option 1?

I felt the advantage with keeping it at the subscription side is that,
the subscriber from one node can subscribe with only_local option on
and a different subscriber from a different node can subscribe with
only_local option as off. This might not be possible with having the
option at publisher side. Having it at the subscriber side might give
more flexibility for the user.

Regards,
Vignesh

  13. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-07T05:44:31Z

    On Tue, Mar 1, 2022 at 4:12 PM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Hi Vignesh,
>
> > In logical replication, currently Walsender sends the data that is
> > generated locally and the data that are replicated from other
> > instances. This results in infinite recursion in circular logical
> > replication setup.
>
> Thank you for good explanation. I understand that this fix can be used
> for a bidirectional replication.
>
> > Here there are two problems for the user: a) incremental
> > synchronization of table sending both local data and replicated data
> > by walsender b) Table synchronization of table using copy command
> > sending both local data and replicated data
>
> So you wanted to solve these two problem and currently focused on
> the first one, right? We can check one by one.
>
> > For the first problem "Incremental synchronization of table by
> > Walsender" can be solved by:
> > Currently the locally generated data does not have replication origin
> > associated and the data that has originated from another instance will
> > have a replication origin associated. We could use this information to
> > differentiate locally generated data and replicated data and send only
> > the locally generated data. This "only_local" could be provided as an
> > option while subscription is created:
> > ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=5433'
> > PUBLICATION pub1 with (only_local = on);
>
> Sounds good, but I cannot distinguish whether the assumption will keep.
>
> I played with your patch, but it could not be applied to current master.
> I tested from bd74c40 and I confirmed infinite loop was not appeared.
>
> local_only could not be set from ALTER SUBSCRIPTION command.
> Is it expected?

I felt changing only_local option might be useful for the user while
modifying the subscription like setting it with a different set of
publications. Changes for this are included in the v2 patch attached
at [1].
[2] - https://www.postgresql.org/message-id/CALDaNm0WSo5369pr2eN1obTGBeiJU9cQdF6Ju1sC4hMQNy5BfQ%40mail.gmail.com

Regards,
Vignesh

  14. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-03-07T06:12:50Z

    Dear Peter,

> > So, why does the patch use syntax option 1?

IMU it might be useful for the following case.

Assuming that multi-master configuration with node1, node2.
Node1 has a publication pub1 and a subscription sub2, node2 has pub2 and sub1.

From that situation, please consider that new node node3 is added
that subscribe some changes from node2.

If the feature is introduced as option1, new publication must be defined in node2.
If that is introduced as option2, however, maybe pub2 can be reused.
i.e. multiple declaration of publications can be avoided.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  15. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T06:14:48Z

    On Mon, Mar 7, 2022 at 4:20 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh, I also have not looked at the patch yet, but I have what
> > seems like a very fundamental (and possibly dumb) question...
> >
> > Basically, I do not understand the choice of syntax for setting things up.
> >
> > IMO that "only-local" option sounds very similar to the other
> > PUBLICATION ("publish") options which decide the kinds of things that
> > will be published. So it feels more natural for me to think of the
> > publisher as being the one to decide what will be published.
> >
> > e.g.
> >
> > option 1:
> > CREATE PUBLICATION p1 FOR TABLE t1;
> > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
> >
> > option 2:
> > CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
> >
> > ~~
> >
> > IIUC the patch is using option 1. My first impression was it feels
> > back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> > publish.
> >
> > So, why does the patch use syntax option 1?
>
> I felt the advantage with keeping it at the subscription side is that,
> the subscriber from one node can subscribe with only_local option on
> and a different subscriber from a different node can subscribe with
> only_local option as off. This might not be possible with having the
> option at publisher side. Having it at the subscriber side might give
> more flexibility for the user.
>

OK.  Option 2 needs two publications for that scenario. IMO it's more
intuitive this way, but maybe you wanted to avoid the extra
publications?

node0:
CREATE PUBLICATION p1 FOR TABLE t1;
CREATE PUBLICATION p1_local FOR TABLE t1 WITH (publish = 'only_local');

node1: CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1_local;
node2: CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1;

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  16. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T06:20:03Z

    On Mon, Mar 7, 2022 at 5:12 PM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Peter,
>
> > > So, why does the patch use syntax option 1?
>
> IMU it might be useful for the following case.
>
> Assuming that multi-master configuration with node1, node2.
> Node1 has a publication pub1 and a subscription sub2, node2 has pub2 and sub1.
>
> From that situation, please consider that new node node3 is added
> that subscribe some changes from node2.
>
> If the feature is introduced as option1, new publication must be defined in node2.
> If that is introduced as option2, however, maybe pub2 can be reused.
> i.e. multiple declaration of publications can be avoided.
>

Yes. Thanks for the example. I had the same observation in my last post  [1]

------
[1] https://www.postgresql.org/message-id/CAHut%2BPtRxiQR_4UFLNThg-NNRV447FvwtcR-BvqMzjyMJXKwfw%40mail.gmail.com

Kind Regards,
Peter Smith
Fujitsu Australia.

  17. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-07T07:17:43Z

    On Mon, Mar 7, 2022 at 11:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 4:20 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, Mar 7, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Vignesh, I also have not looked at the patch yet, but I have what
> > > seems like a very fundamental (and possibly dumb) question...
> > >
> > > Basically, I do not understand the choice of syntax for setting things up.
> > >
> > > IMO that "only-local" option sounds very similar to the other
> > > PUBLICATION ("publish") options which decide the kinds of things that
> > > will be published. So it feels more natural for me to think of the
> > > publisher as being the one to decide what will be published.
> > >
> > > e.g.
> > >
> > > option 1:
> > > CREATE PUBLICATION p1 FOR TABLE t1;
> > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
> > >
> > > option 2:
> > > CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
> > >
> > > ~~
> > >
> > > IIUC the patch is using option 1. My first impression was it feels
> > > back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> > > publish.
> > >
> > > So, why does the patch use syntax option 1?
> >
> > I felt the advantage with keeping it at the subscription side is that,
> > the subscriber from one node can subscribe with only_local option on
> > and a different subscriber from a different node can subscribe with
> > only_local option as off. This might not be possible with having the
> > option at publisher side. Having it at the subscriber side might give
> > more flexibility for the user.
> >
>
> OK.  Option 2 needs two publications for that scenario. IMO it's more
> intuitive this way, but maybe you wanted to avoid the extra
> publications?

Yes, I wanted to avoid the extra publication creation that you pointed
out. Option 1 can handle this scenario without creating the extra
publications:
node0: CREATE PUBLICATION p1 FOR TABLE t1;
node1: CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = on);
node2:  CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = off);

I'm ok with both the approaches, now that this scenario can be handled
by using both the options. i.e providing only_local option as an
option while creating publication or providing only_local option as an
option while creating subscription as Peter has pointed out at [1].
option 1:
CREATE PUBLICATION p1 FOR TABLE t1;
CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);

option 2:
CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'only_local');
CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;

Shall we get a few opinions on this and take it in that direction?

[1] - https://www.postgresql.org/message-id/CAHut%2BPsAWaETh9VMymbBfMrqiE1KuqMq%2BwpBg0s7eMzwLATr%2Bw%40mail.gmail.com

Regards,
Vignesh

  18. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-03-07T07:37:46Z

    Dear Vignesh,

> I felt changing only_local option might be useful for the user while
> modifying the subscription like setting it with a different set of
> publications. Changes for this are included in the v2 patch attached
> at [1].

+1, thanks. I'll post if I notice something to say.

> Shall we get a few opinions on this and take it in that direction?

I prefer subscriber-option, but I also think both are reasonable.
+1 about asking other reviewers.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  19. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-03-07T07:40:44Z

    On Mon, Mar 7, 2022 at 10:15 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > I haven't yet gone through the patch, but I have a question about the
> > > idea.  Suppose I want to set up a logical replication like,
> > > node1->node2->node3->node1.  So how would I create the subscriber at
> > > node1?  only_local=on or off?.  I mean on node1, I want the changes
> > > from node3 which are generated on node3 or which are replicated from
> > > node2 but I do not want changes that are replicated from node1 itself?
> > >  So if I set only_local=on then node1 will not get the changes
> > > replicated from node2, is that right? and If I set only_local=off then
> > > it will create the infinite loop again?  So how are we protecting
> > > against this case?
> > >
> >
> > In the above topology if you want local changes from both node3 and
> > node2 then I think the way to get that would be you have to create two
> > subscriptions on node1. The first one points to node2 (with
> > only_local=off) and the second one points to node3 (with only_local
> > =off).
> >
>
> Sorry, I intend to say 'only_local=on' at both places in my previous email.

Hmm okay, so for this topology we will have to connect node1 directly
to node2 as well as to node3 but can not cascade the changes.  I was
wondering can it be done without using the extra connection between
node2 to node1?  I mean instead of making this a boolean flag that
whether we want local change or remote change, can't we control the
changes based on the origin id?  Such that node1 will get the local
changes of node3 but with using the same subscription it will get
changes from node3 which are originated from node2 but it will not
receive the changes which are originated from node1.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  20. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T08:14:39Z

    On Mon, Mar 7, 2022 at 6:17 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 11:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Mon, Mar 7, 2022 at 4:20 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, Mar 7, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Vignesh, I also have not looked at the patch yet, but I have what
> > > > seems like a very fundamental (and possibly dumb) question...
> > > >
> > > > Basically, I do not understand the choice of syntax for setting things up.
> > > >
> > > > IMO that "only-local" option sounds very similar to the other
> > > > PUBLICATION ("publish") options which decide the kinds of things that
> > > > will be published. So it feels more natural for me to think of the
> > > > publisher as being the one to decide what will be published.
> > > >
> > > > e.g.
> > > >
> > > > option 1:
> > > > CREATE PUBLICATION p1 FOR TABLE t1;
> > > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
> > > >
> > > > option 2:
> > > > CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> > > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
> > > >
> > > > ~~
> > > >
> > > > IIUC the patch is using option 1. My first impression was it feels
> > > > back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> > > > publish.
> > > >
> > > > So, why does the patch use syntax option 1?
> > >
> > > I felt the advantage with keeping it at the subscription side is that,
> > > the subscriber from one node can subscribe with only_local option on
> > > and a different subscriber from a different node can subscribe with
> > > only_local option as off. This might not be possible with having the
> > > option at publisher side. Having it at the subscriber side might give
> > > more flexibility for the user.
> > >
> >
> > OK.  Option 2 needs two publications for that scenario. IMO it's more
> > intuitive this way, but maybe you wanted to avoid the extra
> > publications?
>
> Yes, I wanted to avoid the extra publication creation that you pointed
> out. Option 1 can handle this scenario without creating the extra
> publications:
> node0: CREATE PUBLICATION p1 FOR TABLE t1;
> node1: CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = on);
> node2:  CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = off);
>
> I'm ok with both the approaches, now that this scenario can be handled
> by using both the options. i.e providing only_local option as an
> option while creating publication or providing only_local option as an
> option while creating subscription as Peter has pointed out at [1].
> option 1:
> CREATE PUBLICATION p1 FOR TABLE t1;
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
>
> option 2:
> CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'only_local');
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
>
> Shall we get a few opinions on this and take it in that direction?
>
> [1] - https://www.postgresql.org/message-id/CAHut%2BPsAWaETh9VMymbBfMrqiE1KuqMq%2BwpBg0s7eMzwLATr%2Bw%40mail.gmail.com
>
> Regards,
> Vignesh

BTW here is a counter-example to your scenario from earlier.

Let's say I have a publication p1 and p2 and want to subscribe to p1
with only_local=true, and p2 with only_local = false;

Using the current OPtion 1 syntax you cannot do this with a single
subscription because the option is tied to the subscription.
But using syntax Option 2 you may be able to do it.

Option 1:
CREATE PUBLICATION p1 FOR TABLE t1;
CREATE PUBLICATION p2 FOR TABLE t2;
CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 WITH (local_only = true);
CREATE SUBSCRIPTION s2 ... FOR PUBLICATION p1 WITH (local_only = false);

Option 2:
CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'local_only');
CREATE PUBLICATION p2 FOR TABLE t2;
CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1, p2;

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  21. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-07T08:54:43Z

    On Mon, Mar 7, 2022 at 12:48 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 11:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Mon, Mar 7, 2022 at 4:20 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, Mar 7, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Vignesh, I also have not looked at the patch yet, but I have what
> > > > seems like a very fundamental (and possibly dumb) question...
> > > >
> > > > Basically, I do not understand the choice of syntax for setting things up.
> > > >
> > > > IMO that "only-local" option sounds very similar to the other
> > > > PUBLICATION ("publish") options which decide the kinds of things that
> > > > will be published. So it feels more natural for me to think of the
> > > > publisher as being the one to decide what will be published.
> > > >
> > > > e.g.
> > > >
> > > > option 1:
> > > > CREATE PUBLICATION p1 FOR TABLE t1;
> > > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
> > > >
> > > > option 2:
> > > > CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> > > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
> > > >
> > > > ~~
> > > >
> > > > IIUC the patch is using option 1. My first impression was it feels
> > > > back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> > > > publish.
> > > >
> > > > So, why does the patch use syntax option 1?
> > >
> > > I felt the advantage with keeping it at the subscription side is that,
> > > the subscriber from one node can subscribe with only_local option on
> > > and a different subscriber from a different node can subscribe with
> > > only_local option as off. This might not be possible with having the
> > > option at publisher side. Having it at the subscriber side might give
> > > more flexibility for the user.
> > >
> >
> > OK.  Option 2 needs two publications for that scenario. IMO it's more
> > intuitive this way, but maybe you wanted to avoid the extra
> > publications?
>
> Yes, I wanted to avoid the extra publication creation that you pointed
> out. Option 1 can handle this scenario without creating the extra
> publications:
> node0: CREATE PUBLICATION p1 FOR TABLE t1;
> node1: CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = on);
> node2:  CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = off);
>
> I'm ok with both the approaches, now that this scenario can be handled
> by using both the options. i.e providing only_local option as an
> option while creating publication or providing only_local option as an
> option while creating subscription as Peter has pointed out at [1].
> option 1:
> CREATE PUBLICATION p1 FOR TABLE t1;
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
>
> option 2:
> CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'only_local');
> CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
>
> Shall we get a few opinions on this and take it in that direction?
>

Allowing multiple publications will lead to a lot of duplicate data in
catalogs like pg_publication_rel, so, I don't see how option-2 can be
beneficial?

-- 
With Regards,
Amit Kapila.

  22. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-07T08:57:47Z

    Hi Vignesh,

Here are some review comments for patch v2.

======

1. Question about syntax

I already posted some questions about why the syntax is on the CREATE
SUBSCRCRIBER side.
IMO "local_only" is a publisher option, so it seemed more natural to
me for it to be specified as a "publish" option.

Ref [1] my original question + suggestion for Option 2
Ref [2] some other examples of subscribing to multiple-publishers

Anyway, +1 to see what other people think.

~~~

2. ALTER

(related also to the question about syntax)

If subscribing to multiple publications then ALTER is going to change
the 'local_only' for all of them, which might not be what you want
(??)

~~~

3. subscription_parameter

(related also to the question about syntax)

CREATE SUBSCRIPTION subscription_name
    CONNECTION 'conninfo'
    PUBLICATION publication_name [, ...]
    [ WITH ( subscription_parameter [= value] [, ... ] ) ]

~

That WITH is for *subscription* options, not the publication options.

So IMO 'local_only' intuitively seems like "local" means local where
the subscriber is.

So, if the Option 1 syntax is chosen (see comment #1) then I think the
option name maybe should change to be something more like
'publish_local_only' or something similar to be more clear what local
actually means.

~~~

4. contrib/test_decoding/test_decoding.c

@@ -484,6 +487,16 @@ pg_decode_filter(LogicalDecodingContext *ctx,
  return false;
 }

+static bool
+pg_decode_filter_remotedata(LogicalDecodingContext *ctx,
+   RepOriginId origin_id)
+{
+ TestDecodingData *data = ctx->output_plugin_private;
+
+ if (data->only_local && origin_id != InvalidRepOriginId)
+ return true;
+ return false;
+}

4a. Maybe needs function comment.

4b. Missing blank line following this function

~~~

5. General - please check all of the patch.

There seems inconsistency with the member names, local variable names,
parameter names etc. There are all variations of:

- only_local
- onlylocaldata
- onlylocal_data
- etc

Please try using the same name everywhere for everything if possible.

~~~

6. src/backend/replication/logical/decode.c - FilterRemoteOriginData

@@ -585,7 +594,8 @@ logicalmsg_decode(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
  message = (xl_logical_message *) XLogRecGetData(r);

  if (message->dbId != ctx->slot->data.database ||
- FilterByOrigin(ctx, origin_id))
+ FilterByOrigin(ctx, origin_id) ||
+ FilterRemoteOriginData(ctx, origin_id))
  return;

I noticed that every call to FilterRemoteOriginData has an associated
preceding call to FilterByOrigin. It might be worth just combining the
logic into FilterByOrigin. Then none of that calling code (9 x places)
would need to change at all.

~~~

7. src/backend/replication/logical/logical.c  - CreateInitDecodingContext

@@ -451,6 +453,8 @@ CreateInitDecodingContext(const char *plugin,
  */
  ctx->twophase &= slot->data.two_phase;

+ ctx->onlylocal_data &= slot->data.onlylocal_data;

The equivalent 'twophase' option had a big comment. Probably this new
option should also have a similar comment?

~~~

8. src/backend/replication/logical/logical.c - filter_remotedata_cb_wrapper

+bool
+filter_remotedata_cb_wrapper(LogicalDecodingContext *ctx,
+    RepOriginId origin_id)
+{
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+ bool ret;
+
+ Assert(!ctx->fast_forward);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "filter_remoteorigin";

There is no consistency between the function and the name:

"filter_remoteorigin" versus filter_remotedata_cb.

A similar inconsistency for this is elsewhere. See review comment #9

~~~

9. src/backend/replication/pgoutput/pgoutput.c

@@ -215,6 +217,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
  cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
  cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
  cb->filter_by_origin_cb = pgoutput_origin_filter;
+ cb->filter_remotedata_cb = pgoutput_remoteorigin_filter;

Inconsistent names for the member and function.

filter_remotedata_cb VS pgoutput_remoteorigin_filter.

~~~

10. src/backend/replication/pgoutput/pgoutput.c

@@ -1450,6 +1465,16 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
  return false;
 }

+static bool
+pgoutput_remoteorigin_filter(LogicalDecodingContext *ctx,
+ RepOriginId origin_id)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+
+ if (data->onlylocal_data && origin_id != InvalidRepOriginId)
+ return true;
+ return false;
+}
 /*
  * Shutdown the output plugin.
  *

10a. Add a function comment.

10b. Missing blank line after the function

~~~

11. src/backend/replication/slotfuncs.c - pg_create_logical_replication_slot

@@ -171,6 +174,7 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS)
  Name plugin = PG_GETARG_NAME(1);
  bool temporary = PG_GETARG_BOOL(2);
  bool two_phase = PG_GETARG_BOOL(3);
+ bool onlylocal_data = PG_GETARG_BOOL(4);
  Datum result;
  TupleDesc tupdesc;
  HeapTuple tuple;


Won't there be some PG Docs needing to be updated now there is another
parameter?

~~~

12. src/include/catalog/pg_proc.dat - pg_get_replication_slots

I did not see any update for pg_get_replication_slots,  but you added
the 4th parameter elsewhere. Is something missing here?

~~~

13. src/include/replication/logical.h

@@ -99,6 +99,8 @@ typedef struct LogicalDecodingContext
  */
  bool twophase_opt_given;

+ bool onlylocal_data;
+

I think the new member needs some comment.

~~~

14. src/include/replication/walreceiver.h

@@ -183,6 +183,7 @@ typedef struct
  bool streaming; /* Streaming of large transactions */
  bool twophase; /* Streaming of two-phase transactions at
  * prepare time */
+ bool onlylocal_data;
  } logical;
  } proto;
 } WalRcvStreamOptions;

I think the new member needs some comment.

~~~

15. src/test/regress/sql/subscription.sql

ALTER SUBSCRIPTION test missing?

------
[1] https://www.postgresql.org/message-id/CAHut%2BPsAWaETh9VMymbBfMrqiE1KuqMq%2BwpBg0s7eMzwLATr%2Bw%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPvQonJd5epJBM0Yfh1499mL9kTL9a%3DGrMhvnL6Ok05zqw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  23. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-07T09:31:15Z

    On Mon, Mar 7, 2022 at 1:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 10:15 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > I haven't yet gone through the patch, but I have a question about the
> > > > idea.  Suppose I want to set up a logical replication like,
> > > > node1->node2->node3->node1.  So how would I create the subscriber at
> > > > node1?  only_local=on or off?.  I mean on node1, I want the changes
> > > > from node3 which are generated on node3 or which are replicated from
> > > > node2 but I do not want changes that are replicated from node1 itself?
> > > >  So if I set only_local=on then node1 will not get the changes
> > > > replicated from node2, is that right? and If I set only_local=off then
> > > > it will create the infinite loop again?  So how are we protecting
> > > > against this case?
> > > >
> > >
> > > In the above topology if you want local changes from both node3 and
> > > node2 then I think the way to get that would be you have to create two
> > > subscriptions on node1. The first one points to node2 (with
> > > only_local=off) and the second one points to node3 (with only_local
> > > =off).
> > >
> >
> > Sorry, I intend to say 'only_local=on' at both places in my previous email.
>
> Hmm okay, so for this topology we will have to connect node1 directly
> to node2 as well as to node3 but can not cascade the changes.  I was
> wondering can it be done without using the extra connection between
> node2 to node1?  I mean instead of making this a boolean flag that
> whether we want local change or remote change, can't we control the
> changes based on the origin id?  Such that node1 will get the local
> changes of node3 but with using the same subscription it will get
> changes from node3 which are originated from node2 but it will not
> receive the changes which are originated from node1.
>

Good point. I think we can provide that as an additional option to
give more flexibility but we won't be able to use it for initial sync
where we can't differentiate between data from different origins.
Also, I think as origins are internally generated, we may need some
better way to expose it to users so that they can specify it as an
option. Isn't it better to provide first some simple way like a
boolean option so that users have some way to replicate the same table
data among different nodes without causing an infinite loop and then
extend it as you are suggesting or may be in some other ways as well?

-- 
With Regards,
Amit Kapila.

  24. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-03-07T09:55:18Z

    On Mon, Mar 7, 2022 at 3:01 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 1:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Mon, Mar 7, 2022 at 10:15 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > I haven't yet gone through the patch, but I have a question about the
> > > > > idea.  Suppose I want to set up a logical replication like,
> > > > > node1->node2->node3->node1.  So how would I create the subscriber at
> > > > > node1?  only_local=on or off?.  I mean on node1, I want the changes
> > > > > from node3 which are generated on node3 or which are replicated from
> > > > > node2 but I do not want changes that are replicated from node1 itself?
> > > > >  So if I set only_local=on then node1 will not get the changes
> > > > > replicated from node2, is that right? and If I set only_local=off then
> > > > > it will create the infinite loop again?  So how are we protecting
> > > > > against this case?
> > > > >
> > > >
> > > > In the above topology if you want local changes from both node3 and
> > > > node2 then I think the way to get that would be you have to create two
> > > > subscriptions on node1. The first one points to node2 (with
> > > > only_local=off) and the second one points to node3 (with only_local
> > > > =off).
> > > >
> > >
> > > Sorry, I intend to say 'only_local=on' at both places in my previous email.
> >
> > Hmm okay, so for this topology we will have to connect node1 directly
> > to node2 as well as to node3 but can not cascade the changes.  I was
> > wondering can it be done without using the extra connection between
> > node2 to node1?  I mean instead of making this a boolean flag that
> > whether we want local change or remote change, can't we control the
> > changes based on the origin id?  Such that node1 will get the local
> > changes of node3 but with using the same subscription it will get
> > changes from node3 which are originated from node2 but it will not
> > receive the changes which are originated from node1.
> >
>
> Good point. I think we can provide that as an additional option to
> give more flexibility but we won't be able to use it for initial sync
> where we can't differentiate between data from different origins.
> Also, I think as origins are internally generated, we may need some
> better way to expose it to users so that they can specify it as an
> option. Isn't it better to provide first some simple way like a
> boolean option so that users have some way to replicate the same table
> data among different nodes without causing an infinite loop and then
> extend it as you are suggesting or may be in some other ways as well?

Yeah, that makes sense that first we provide some simple mechanism to
enable it and we can extend it later.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  25. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-07T10:53:17Z

    On Mon, Mar 7, 2022 at 1:45 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 6:17 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, Mar 7, 2022 at 11:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > On Mon, Mar 7, 2022 at 4:20 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Mon, Mar 7, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > > >
> > > > > Hi Vignesh, I also have not looked at the patch yet, but I have what
> > > > > seems like a very fundamental (and possibly dumb) question...
> > > > >
> > > > > Basically, I do not understand the choice of syntax for setting things up.
> > > > >
> > > > > IMO that "only-local" option sounds very similar to the other
> > > > > PUBLICATION ("publish") options which decide the kinds of things that
> > > > > will be published. So it feels more natural for me to think of the
> > > > > publisher as being the one to decide what will be published.
> > > > >
> > > > > e.g.
> > > > >
> > > > > option 1:
> > > > > CREATE PUBLICATION p1 FOR TABLE t1;
> > > > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
> > > > >
> > > > > option 2:
> > > > > CREATE PUBLICATION p1 FOR TABLE t1 WEHRE (publish = 'only_local');
> > > > > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
> > > > >
> > > > > ~~
> > > > >
> > > > > IIUC the patch is using option 1. My first impression was it feels
> > > > > back-to-front for the SUBSCRIPTION telling the PUBLICATION what to
> > > > > publish.
> > > > >
> > > > > So, why does the patch use syntax option 1?
> > > >
> > > > I felt the advantage with keeping it at the subscription side is that,
> > > > the subscriber from one node can subscribe with only_local option on
> > > > and a different subscriber from a different node can subscribe with
> > > > only_local option as off. This might not be possible with having the
> > > > option at publisher side. Having it at the subscriber side might give
> > > > more flexibility for the user.
> > > >
> > >
> > > OK.  Option 2 needs two publications for that scenario. IMO it's more
> > > intuitive this way, but maybe you wanted to avoid the extra
> > > publications?
> >
> > Yes, I wanted to avoid the extra publication creation that you pointed
> > out. Option 1 can handle this scenario without creating the extra
> > publications:
> > node0: CREATE PUBLICATION p1 FOR TABLE t1;
> > node1: CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = on);
> > node2:  CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 with (only_local = off);
> >
> > I'm ok with both the approaches, now that this scenario can be handled
> > by using both the options. i.e providing only_local option as an
> > option while creating publication or providing only_local option as an
> > option while creating subscription as Peter has pointed out at [1].
> > option 1:
> > CREATE PUBLICATION p1 FOR TABLE t1;
> > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true);
> >
> > option 2:
> > CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'only_local');
> > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1;
> >
> > Shall we get a few opinions on this and take it in that direction?
> >
> > [1] - https://www.postgresql.org/message-id/CAHut%2BPsAWaETh9VMymbBfMrqiE1KuqMq%2BwpBg0s7eMzwLATr%2Bw%40mail.gmail.com
> >
> > Regards,
> > Vignesh
>
> BTW here is a counter-example to your scenario from earlier.
>
> Let's say I have a publication p1 and p2 and want to subscribe to p1
> with only_local=true, and p2 with only_local = false;
>
> Using the current OPtion 1 syntax you cannot do this with a single
> subscription because the option is tied to the subscription.
> But using syntax Option 2 you may be able to do it.
>
> Option 1:
> CREATE PUBLICATION p1 FOR TABLE t1;
> CREATE PUBLICATION p2 FOR TABLE t2;
> CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1 WITH (local_only = true);
> CREATE SUBSCRIPTION s2 ... FOR PUBLICATION p1 WITH (local_only = false);
>
> Option 2:
> CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'local_only');
> CREATE PUBLICATION p2 FOR TABLE t2;
> CREATE SUBSCRIPTION s1 ... FOR PUBLICATION p1, p2;

I felt having multiple publications will create duplicate entries in
the system table, Amit also has pointed this at [1]. Also enhancing
this approach to support filtering based on replication origin which
is suggested by dilip at [2] is also on the client side and also the
initial check to handle the copy_data specified by Amit at [3] will be
done by the client side. Based on the above I feel the existing
approach is better. I might be missing something here.

[1] - https://www.postgresql.org/message-id/CAA4eK1LgCVv8u-fOsMPbGC96sWXhT3EKOBAeFW3g84otjStztw%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/CAFiTN-tKbjHDjAFNnqRoR8u1B%2Bfs0wunGz%3D3wp0iU-sUaxZJTQ%40mail.gmail.com
[3] - https://www.postgresql.org/message-id/CAA4eK1%2Bco2cd8a6okgUD_pcFEHcc7mVc0k_RE2%3D6ahyv3WPRMg%40mail.gmail.com

Regards,
Vignesh

  26. Re: Handle infinite recursion in logical replication setup

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2022-03-07T11:30:56Z

    Hi Vignesh,
I agree with Peter's comment that the changes to
FilterRemoteOriginData() should be part of FilterByOrigin()

Further, I wonder why "onlylocal_data" is a replication slot's
property. A replication slot tracks the progress of replication and it
may be used by different receivers with different options. I could
start one receiver which wants only local data, say using
"pg_logical_slot_get_changes" and later start another receiver which
fetches all the data starting from where the first receiver left. This
option prevents such flexibility.

As discussed earlier in the thread, local_only can be property of
publication or subscription, depending upon the use case, but I can't
see any reason that it should be tied to a replication slot.

I have a similar question for "two_phase" but the ship has sailed and
probably it makes some sense there which I don't know.

As for publication vs subscription, I think both are useful cases.
1. It will be a publication's property, if we want the node to not
publish any data that it receives from other nodes for a given set of
tables.
2. It will be the subscription's property, if we want the subscription
to decide whether it wants to fetch the data changed on only upstream
or other nodes as well.

Maybe we want to add it to both and use the stricter of both. If a
publication has local_only, it publishes only local changes. If the
subscription has local only then WAL sender sends only local changes.

--
Best Wishes,
Ashutosh

  27. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-07T12:20:59Z

    On Mon, Mar 7, 2022 at 5:01 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> Hi Vignesh,
> I agree with Peter's comment that the changes to
> FilterRemoteOriginData() should be part of FilterByOrigin()
>
> Further, I wonder why "onlylocal_data" is a replication slot's
> property. A replication slot tracks the progress of replication and it
> may be used by different receivers with different options. I could
> start one receiver which wants only local data, say using
> "pg_logical_slot_get_changes" and later start another receiver which
> fetches all the data starting from where the first receiver left. This
> option prevents such flexibility.
>
> As discussed earlier in the thread, local_only can be property of
> publication or subscription, depending upon the use case, but I can't
> see any reason that it should be tied to a replication slot.
>

I thought it should be similar to 'streaming' option of subscription
but may be Vignesh has some other reason which makes it different.

> I have a similar question for "two_phase" but the ship has sailed and
> probably it makes some sense there which I don't know.
>

two_phase is different from some of the other subscription options
like 'streaming' such that it can be enabled only at the time of slot
and subscription creation, we can't change/specify it via
pg_logical_slot_get_changes. This is to avoid the case where we won't
know at the time of the commit prepared whether the prepare for the
transaction has already been sent. For the same reason, we need to
also know the 'two_phase_at' information.

> As for publication vs subscription, I think both are useful cases.
> 1. It will be a publication's property, if we want the node to not
> publish any data that it receives from other nodes for a given set of
> tables.
> 2. It will be the subscription's property, if we want the subscription
> to decide whether it wants to fetch the data changed on only upstream
> or other nodes as well.
>

I think it could be useful to allow it via both publication and
subscription but I guess it is better to provide it via one way
initially just to keep things simple and give users some way to deal
with such cases. I would prefer to allow it via subscription initially
for the reasons specified by Vignesh in his previous email [1]. Now,
if we think those all are ignorable things and it is more important to
allow this option first by publication or we must allow it via both
publication and subscription then it makes sense to change it.


[1] - https://www.postgresql.org/message-id/CALDaNm3jkotRhKfCqu5CXOf36_yiiW_cYE5%3DbG%3Dj6N3gOWJkqw%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  28. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-07T16:27:03Z

    On Mon, Mar 7, 2022 at 5:51 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 5:01 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > Hi Vignesh,
> > I agree with Peter's comment that the changes to
> > FilterRemoteOriginData() should be part of FilterByOrigin()
> >
> > Further, I wonder why "onlylocal_data" is a replication slot's
> > property. A replication slot tracks the progress of replication and it
> > may be used by different receivers with different options. I could
> > start one receiver which wants only local data, say using
> > "pg_logical_slot_get_changes" and later start another receiver which
> > fetches all the data starting from where the first receiver left. This
> > option prevents such flexibility.
> >
> > As discussed earlier in the thread, local_only can be property of
> > publication or subscription, depending upon the use case, but I can't
> > see any reason that it should be tied to a replication slot.
> >
>
> I thought it should be similar to 'streaming' option of subscription
> but may be Vignesh has some other reason which makes it different.

Yes, this can be removed from the replication slot. It is my mistake
that I have made while making the code similar to two-phase, I'm
working on making the changes for this. I will fix and post an updated
version for this.

Regards,
Vignesh

  29. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-08T05:00:45Z

    IIUC the new option may be implemented subscriber-side and/or
publisher-side and/or both, and the subscriber-side option may be
"enhanced" in future to prevent cycles. And probably there are more
features I don't know about or that have not yet been thought of.

~~

Even if the plan is only to implement just one part now and then add
more later, I think there still should be some consideration for what
you expect all possible future options to look like, because that may
affect current implementation choices.

The point is:

- we should take care so don't accidentally end up with an option that
turned out to be inconsistent looking on the subscriber-side /
publisher-side.

- we should try to avoid accidentally painting ourselves into a corner
(e.g. stuck with a boolean option that cannot be enhanced later on)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  30. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-08T05:21:08Z

    On Tue, Mar 8, 2022 at 10:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> IIUC the new option may be implemented subscriber-side and/or
> publisher-side and/or both, and the subscriber-side option may be
> "enhanced" in future to prevent cycles. And probably there are more
> features I don't know about or that have not yet been thought of.
>
> ~~
>
> Even if the plan is only to implement just one part now and then add
> more later, I think there still should be some consideration for what
> you expect all possible future options to look like, because that may
> affect current implementation choices.
>
> The point is:
>
> - we should take care so don't accidentally end up with an option that
> turned out to be inconsistent looking on the subscriber-side /
> publisher-side.
>
> - we should try to avoid accidentally painting ourselves into a corner
> (e.g. stuck with a boolean option that cannot be enhanced later on)
>

Agreed. I think it is important to see we shouldn't do something which
is not extendable in future or paint us in the corner. But, as of now,
with the current proposal, the main thing we should consider is
whether exposing the boolean option is okay or shall we do something
different so that we can extend it later to specific origin ids?

-- 
With Regards,
Amit Kapila.

  31. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-03-08T06:46:37Z

    On Tue, Mar 8, 2022 at 4:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Mar 8, 2022 at 10:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > IIUC the new option may be implemented subscriber-side and/or
> > publisher-side and/or both, and the subscriber-side option may be
> > "enhanced" in future to prevent cycles. And probably there are more
> > features I don't know about or that have not yet been thought of.
> >
> > ~~
> >
> > Even if the plan is only to implement just one part now and then add
> > more later, I think there still should be some consideration for what
> > you expect all possible future options to look like, because that may
> > affect current implementation choices.
> >
> > The point is:
> >
> > - we should take care so don't accidentally end up with an option that
> > turned out to be inconsistent looking on the subscriber-side /
> > publisher-side.
> >
> > - we should try to avoid accidentally painting ourselves into a corner
> > (e.g. stuck with a boolean option that cannot be enhanced later on)
> >
>
> Agreed. I think it is important to see we shouldn't do something which
> is not extendable in future or paint us in the corner. But, as of now,
> with the current proposal, the main thing we should consider is
> whether exposing the boolean option is okay or shall we do something
> different so that we can extend it later to specific origin ids?
>

I was wondering, assuming later there is an enhancement to detect and
prevent cycles using origin ids, then what really is the purpose of
the option?

I can imagine if a cycle happens should probably log some one-time
WARNING (just to bring it to the user's attention in case it was
caused by some accidental misconfiguration) but apart from that why
would this need to be an option at all - i.e. is there a use-case
where a user would NOT want to prevent a recursive error?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  32. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-08T07:54:47Z

    On Tue, Mar 8, 2022 at 12:17 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Mar 8, 2022 at 4:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Mar 8, 2022 at 10:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > IIUC the new option may be implemented subscriber-side and/or
> > > publisher-side and/or both, and the subscriber-side option may be
> > > "enhanced" in future to prevent cycles. And probably there are more
> > > features I don't know about or that have not yet been thought of.
> > >
> > > ~~
> > >
> > > Even if the plan is only to implement just one part now and then add
> > > more later, I think there still should be some consideration for what
> > > you expect all possible future options to look like, because that may
> > > affect current implementation choices.
> > >
> > > The point is:
> > >
> > > - we should take care so don't accidentally end up with an option that
> > > turned out to be inconsistent looking on the subscriber-side /
> > > publisher-side.
> > >
> > > - we should try to avoid accidentally painting ourselves into a corner
> > > (e.g. stuck with a boolean option that cannot be enhanced later on)
> > >
> >
> > Agreed. I think it is important to see we shouldn't do something which
> > is not extendable in future or paint us in the corner. But, as of now,
> > with the current proposal, the main thing we should consider is
> > whether exposing the boolean option is okay or shall we do something
> > different so that we can extend it later to specific origin ids?
> >
>
> I was wondering, assuming later there is an enhancement to detect and
> prevent cycles using origin ids, then what really is the purpose of
> the option?
>

We can't use origin ids during the initial sync as we can't
differentiate between data from local or remote node. Also, how will
we distinguish between different origins? AFAIU, all the changes
applied on any particular node use the same origin.

> I can imagine if a cycle happens should probably log some one-time
> WARNING (just to bring it to the user's attention in case it was
> caused by some accidental misconfiguration) but apart from that why
> would this need to be an option at all - i.e. is there a use-case
> where a user would NOT want to prevent a recursive error?
>

I think there will be plenty of such setups where there won't be a
need for any such option like where users won't need bi-directional
replication for the same table.

-- 
With Regards,
Amit Kapila.

  33. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-08T14:38:00Z

    On Mon, Mar 7, 2022 at 2:28 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are some review comments for patch v2.
>
> ======
>
> 1. Question about syntax
>
> I already posted some questions about why the syntax is on the CREATE
> SUBSCRCRIBER side.
> IMO "local_only" is a publisher option, so it seemed more natural to
> me for it to be specified as a "publish" option.
>
> Ref [1] my original question + suggestion for Option 2
> Ref [2] some other examples of subscribing to multiple-publishers
>
> Anyway, +1 to see what other people think.
>

I feel we can support it in the subscriber side first and then extend
it to the publisher side as being discussed in [1]. I have retained it
as it is.

> ~~~
>
> 2. ALTER
>
> (related also to the question about syntax)
>
> If subscribing to multiple publications then ALTER is going to change
> the 'local_only' for all of them, which might not be what you want
> (??)
>

 I feel we can support it in the subscriber side first and then extend
it to the publisher side as being discussed in [1]. When it is
extended to the publisher side, it will get handled. I have retained
it as it is.

> ~~~
>
> 3. subscription_parameter
>
> (related also to the question about syntax)
>
> CREATE SUBSCRIPTION subscription_name
>     CONNECTION 'conninfo'
>     PUBLICATION publication_name [, ...]
>     [ WITH ( subscription_parameter [= value] [, ... ] ) ]
>
> ~
>
> That WITH is for *subscription* options, not the publication options.
>
> So IMO 'local_only' intuitively seems like "local" means local where
> the subscriber is.
>
> So, if the Option 1 syntax is chosen (see comment #1) then I think the
> option name maybe should change to be something more like
> 'publish_local_only' or something similar to be more clear what local
> actually means.
>

Changed it to publish_local_only

> ~~~
>
> 4. contrib/test_decoding/test_decoding.c
>
> @@ -484,6 +487,16 @@ pg_decode_filter(LogicalDecodingContext *ctx,
>   return false;
>  }
>
> +static bool
> +pg_decode_filter_remotedata(LogicalDecodingContext *ctx,
> +   RepOriginId origin_id)
> +{
> + TestDecodingData *data = ctx->output_plugin_private;
> +
> + if (data->only_local && origin_id != InvalidRepOriginId)
> + return true;
> + return false;
> +}
>
> 4a. Maybe needs function comment.

Modified

> 4b. Missing blank line following this function
>

Modified

> ~~~
>
> 5. General - please check all of the patch.
>
> There seems inconsistency with the member names, local variable names,
> parameter names etc. There are all variations of:
>
> - only_local
> - onlylocaldata
> - onlylocal_data
> - etc
>
> Please try using the same name everywhere for everything if possible.
>

I have changed it to only_local wherever possible.

> ~~~
>
> 6. src/backend/replication/logical/decode.c - FilterRemoteOriginData
>
> @@ -585,7 +594,8 @@ logicalmsg_decode(LogicalDecodingContext *ctx,
> XLogRecordBuffer *buf)
>   message = (xl_logical_message *) XLogRecGetData(r);
>
>   if (message->dbId != ctx->slot->data.database ||
> - FilterByOrigin(ctx, origin_id))
> + FilterByOrigin(ctx, origin_id) ||
> + FilterRemoteOriginData(ctx, origin_id))
>   return;
>
> I noticed that every call to FilterRemoteOriginData has an associated
> preceding call to FilterByOrigin. It might be worth just combining the
> logic into FilterByOrigin. Then none of that calling code (9 x places)
> would need to change at all.

Modified

> ~~~
>
> 7. src/backend/replication/logical/logical.c  - CreateInitDecodingContext
>
> @@ -451,6 +453,8 @@ CreateInitDecodingContext(const char *plugin,
>   */
>   ctx->twophase &= slot->data.two_phase;
>
> + ctx->onlylocal_data &= slot->data.onlylocal_data;
>
> The equivalent 'twophase' option had a big comment. Probably this new
> option should also have a similar comment?

These change is not required anymore, the comment no more applies. I
have not made any change for this.

> ~~~
>
> 8. src/backend/replication/logical/logical.c - filter_remotedata_cb_wrapper
>
> +bool
> +filter_remotedata_cb_wrapper(LogicalDecodingContext *ctx,
> +    RepOriginId origin_id)
> +{
> + LogicalErrorCallbackState state;
> + ErrorContextCallback errcallback;
> + bool ret;
> +
> + Assert(!ctx->fast_forward);
> +
> + /* Push callback + info on the error context stack */
> + state.ctx = ctx;
> + state.callback_name = "filter_remoteorigin";
>
> There is no consistency between the function and the name:
>
> "filter_remoteorigin" versus filter_remotedata_cb.
>
> A similar inconsistency for this is elsewhere. See review comment #9

Modified it to filter_remote_origin_cb_wrapper and changed to
*_remotedata_* to *_remote_origin_*

> ~~~
>
> 9. src/backend/replication/pgoutput/pgoutput.c
>
> @@ -215,6 +217,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
>   cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
>   cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
>   cb->filter_by_origin_cb = pgoutput_origin_filter;
> + cb->filter_remotedata_cb = pgoutput_remoteorigin_filter;
>
> Inconsistent names for the member and function.
>
> filter_remotedata_cb VS pgoutput_remoteorigin_filter.

Modified it to filter_remote_origin_cb and changed to *_remotedata_*
to *_remote_origin_*

> ~~~
>
> 10. src/backend/replication/pgoutput/pgoutput.c
>
> @@ -1450,6 +1465,16 @@ pgoutput_origin_filter(LogicalDecodingContext *ctx,
>   return false;
>  }
>
> +static bool
> +pgoutput_remoteorigin_filter(LogicalDecodingContext *ctx,
> + RepOriginId origin_id)
> +{
> + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
> +
> + if (data->onlylocal_data && origin_id != InvalidRepOriginId)
> + return true;
> + return false;
> +}
>  /*
>   * Shutdown the output plugin.
>   *
>
> 10a. Add a function comment.

Modified

> 10b. Missing blank line after the function

Modified

> ~~~
>
> 11. src/backend/replication/slotfuncs.c - pg_create_logical_replication_slot
>
> @@ -171,6 +174,7 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS)
>   Name plugin = PG_GETARG_NAME(1);
>   bool temporary = PG_GETARG_BOOL(2);
>   bool two_phase = PG_GETARG_BOOL(3);
> + bool onlylocal_data = PG_GETARG_BOOL(4);
>   Datum result;
>   TupleDesc tupdesc;
>   HeapTuple tuple;
>
>
> Won't there be some PG Docs needing to be updated now there is another
> parameter?

This change is not required anymore, the comment no longer applies. I
have not made any changes for this.

> ~~~
>
> 12. src/include/catalog/pg_proc.dat - pg_get_replication_slots
>
> I did not see any update for pg_get_replication_slots,  but you added
> the 4th parameter elsewhere. Is something missing here?

This change is not required anymore, the comment no longer applies. I
have not made any changes for this.

> ~~~
>
> 13. src/include/replication/logical.h
>
> @@ -99,6 +99,8 @@ typedef struct LogicalDecodingContext
>   */
>   bool twophase_opt_given;
>
> + bool onlylocal_data;
> +
>
> I think the new member needs some comment.

Modified

> ~~~
>
> 14. src/include/replication/walreceiver.h
>
> @@ -183,6 +183,7 @@ typedef struct
>   bool streaming; /* Streaming of large transactions */
>   bool twophase; /* Streaming of two-phase transactions at
>   * prepare time */
> + bool onlylocal_data;
>   } logical;
>   } proto;
>  } WalRcvStreamOptions;
>
> I think the new member needs some comment.

Modified

> ~~~
>
> 15. src/test/regress/sql/subscription.sql
>
> ALTER SUBSCRIPTION test missing?

Added

Thanks for the comments, the attached v3 patch has the fixes for the same.
[1] - https://www.postgresql.org/message-id/CAA4eK1%2BMtz%2BStvNNtTg9%3D9BTq8%3DpMu-V5i4yWqs%3DKJUh0Z_L4g%40mail.gmail.com

Regards,
Vignesh

  34. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-08T14:46:24Z

    On Mon, Mar 7, 2022 at 9:57 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Mar 7, 2022 at 5:51 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Mar 7, 2022 at 5:01 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > Hi Vignesh,
> > > I agree with Peter's comment that the changes to
> > > FilterRemoteOriginData() should be part of FilterByOrigin()
> > >
> > > Further, I wonder why "onlylocal_data" is a replication slot's
> > > property. A replication slot tracks the progress of replication and it
> > > may be used by different receivers with different options. I could
> > > start one receiver which wants only local data, say using
> > > "pg_logical_slot_get_changes" and later start another receiver which
> > > fetches all the data starting from where the first receiver left. This
> > > option prevents such flexibility.
> > >
> > > As discussed earlier in the thread, local_only can be property of
> > > publication or subscription, depending upon the use case, but I can't
> > > see any reason that it should be tied to a replication slot.
> > >
> >
> > I thought it should be similar to 'streaming' option of subscription
> > but may be Vignesh has some other reason which makes it different.
>
> Yes, this can be removed from the replication slot. It is my mistake
> that I have made while making the code similar to two-phase, I'm
> working on making the changes for this. I will fix and post an updated
> version for this.

I have made the changes for this, the changes for this are available
in the v3 patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0JcV-7iQZhyy3kehnWTy6x%3Dz%2BsX6u6Df%2B%2By8z33pz%2BBw%40mail.gmail.com

Regards,
Vignesh

  35. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-03-11T10:58:25Z

    Hi Vegnesh,

While considering about second problem, I was very confusing about it.
I'm happy if you answer my question.

> To handle this if user has specified only_local option, we could throw
> a warning or error out while creating subscription in this case, we
> could have a column srreplicateddata in pg_subscription_rel which
> could indicate if the table has any replicated data or not:
> postgres=# select * from pg_subscription_rel;
>  srsubid | srrelid | srsubstate | srsublsn  | srreplicateddata
> ---------+---------+------------+-----------+------------------
>    16389 |   16384 | r          | 0/14A4640 |        t
>    16389 |   16385 | r          | 0/14A4690 |        f
> (1 row)
> In the above example, srreplicateddata with true indicates, tabel t1
> whose relid is 16384 has replicated data and the other row having
> srreplicateddata  as false indicates table t2 whose relid is 16385
> does not have replicated data.
> When creating a new subscription, the subscriber will connect to the
> publisher and check if the relation has replicated data by checking
> srreplicateddata in pg_subscription_rel table.
> If the table has any replicated data, log a warning or error for this.

IIUC srreplicateddata represents whether the subscribed data is not
generated from the publisher, but another node.
My first impression was that the name 'srreplicateddata' is not friendly
because all subscribed data is replicated from publisher.
Also I was not sure how value of the column was set.
IIUC a filtering by replication origins is done in publisher node
and subscriber node cannot know
whether some data are really filtered or not.
If we distinguish by subscriber option publish_local_only,
it cannot reproduce your example because same subscriber have different 'srreplicateddata'.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  36. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-12T05:16:19Z

    On Wed, Mar 2, 2022 at 3:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Feb 23, 2022 at 11:59 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> ...
> ...
> > I have attached a basic patch for this, if the idea is accepted, I
> > will work further to test more scenarios, add documentation, and test
> > and post an updated patch.
> > For the second problem, Table synchronization of table including local
> > data and replicated data using copy command.
> >
> > Let us consider the following scenario:
> > a) node1 publishing to node2 b) node2 publishing to node1. Here in
> > this case node1 will have replicated data from node2 and vice versa.
> >
> > In the above if user wants to include node3 to subscribe data from
> > node2. Users will have to create a subscription in node3 to get the
> > data from node2. During table synchronization we send the complete
> > table data from node2 to node3. Node2 will have local data from node2
> > and also replicated data from node1. Currently we don't have an option
> > to differentiate between the locally generated data and replicated
> > data in the heap which will cause infinite recursion as described
> > above.
> >
> > To handle this if user has specified only_local option, we could throw
> > a warning or error out while creating subscription in this case, we
> > could have a column srreplicateddata in pg_subscription_rel which
> > could indicate if the table has any replicated data or not:
> > postgres=# select * from pg_subscription_rel;
> >  srsubid | srrelid | srsubstate | srsublsn  | srreplicateddata
> > ---------+---------+------------+-----------+------------------
> >    16389 |   16384 | r          | 0/14A4640 |        t
> >    16389 |   16385 | r          | 0/14A4690 |        f
> > (1 row)
> >
> > In the above example, srreplicateddata with true indicates, tabel t1
> > whose relid is 16384 has replicated data and the other row having
> > srreplicateddata  as false indicates table t2 whose relid is 16385
> > does not have replicated data.
> > When creating a new subscription, the subscriber will connect to the
> > publisher and check if the relation has replicated data by checking
> > srreplicateddata in pg_subscription_rel table.
> > If the table has any replicated data, log a warning or error for this.
> >
>
> If you want to give the error in this case, then I think we need to
> provide an option to the user to allow copy. One possibility could be
> to extend existing copy_data option as 'false', 'true', 'force'. For
> 'false', there shouldn't be any change, for 'true', if 'only_local'
> option is also set and the new column indicates replicated data then
> give an error, for 'force', we won't give an error even if the
> conditions as mentioned for 'true' case are met, rather we will allow
> copy in this case.

When a subscription is created with publish_local_only and copy_data,
it will connect to the publisher and check if the published tables
have also been subscribed from other nodes by checking if the entry is
present in pg_subscription_rel and throw an error if present. The
attached v4-0002-Support-force-option-for-copy_data-check-and-thro.patch
has the implementation for the same.
Thoughts?

Regards,
Vignesh

  37. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-12T06:55:27Z

    On Fri, Mar 11, 2022 at 4:28 PM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Hi Vegnesh,
>
> While considering about second problem, I was very confusing about it.
> I'm happy if you answer my question.
>
> > To handle this if user has specified only_local option, we could throw
> > a warning or error out while creating subscription in this case, we
> > could have a column srreplicateddata in pg_subscription_rel which
> > could indicate if the table has any replicated data or not:
> > postgres=# select * from pg_subscription_rel;
> >  srsubid | srrelid | srsubstate | srsublsn  | srreplicateddata
> > ---------+---------+------------+-----------+------------------
> >    16389 |   16384 | r          | 0/14A4640 |        t
> >    16389 |   16385 | r          | 0/14A4690 |        f
> > (1 row)
> > In the above example, srreplicateddata with true indicates, tabel t1
> > whose relid is 16384 has replicated data and the other row having
> > srreplicateddata  as false indicates table t2 whose relid is 16385
> > does not have replicated data.
> > When creating a new subscription, the subscriber will connect to the
> > publisher and check if the relation has replicated data by checking
> > srreplicateddata in pg_subscription_rel table.
> > If the table has any replicated data, log a warning or error for this.
>
> IIUC srreplicateddata represents whether the subscribed data is not
> generated from the publisher, but another node.
> My first impression was that the name 'srreplicateddata' is not friendly
> because all subscribed data is replicated from publisher.
> Also I was not sure how value of the column was set.
> IIUC a filtering by replication origins is done in publisher node
> and subscriber node cannot know
> whether some data are really filtered or not.
> If we distinguish by subscriber option publish_local_only,
> it cannot reproduce your example because same subscriber have different 'srreplicateddata'.

Let's consider an existing Multi master logical replication setup
between Node1 and Node2 that is created using the following steps:
a) Node1 - Publication publishing employee table - pub1
b) Node2 - Subscription subscribing from publication pub1 with
publish_local_only - sub1_pub1_node1
c) Node2 - Publication publishing employee table - pub2
d) Node1 - Subscription subscribing from publication pub2 with
publish_local_only - sub2_pub2_node2

To create a subscription in node3, we will be using the following steps:
a) Node2 - Publication publishing employee table. - pub3
b) Node3 - Subscription subscribing from publication in Node2 with
publish_local_only - sub3_pub3_node2

When we create a subscription in Node3, Node3 will connect to
Node2(this will not be done in Node3) and check if the employee table
is present in pg_subscription_rel, in our case Node2 will have
employee table present in pg_subscription_rel (sub1_pub1_node1
subscribing to employee table from pub1 in Node1). As employee table
is being subscribed in node2 from node1, we will throw an error like
below:
postgres=# create subscription sub2 CONNECTION 'dbname =postgres port
= 9999' publication pub2 with (publish_local_only=on);
ERROR:  CREATE/ALTER SUBSCRIPTION with publish_local_only and
copy_data as true is not allowed when the publisher might have
replicated data, table:public.t1 might have replicated data in the
publisher
HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force

I was initially planning to add srreplicateddata field but I have
changed it slightly to keep the design simple. Now we just check if
the relation is present in pg_subscription_rel and throw an error if
copy_data and publish_local_only option is specified. The changes for
the same are available at [1].

[1] - https://www.postgresql.org/message-id/CALDaNm0V%2B%3Db%3DCeZJNAAUO2PmSXH5QzNX3jADXb-0hGO_jVj0vA%40mail.gmail.com
Thoughts?

Regards,
Vignesh

  38. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-03-15T01:39:49Z

    Dear Vignesh,

Thank you for updating your patch!

> Let's consider an existing Multi master logical replication setup
> between Node1 and Node2 that is created using the following steps:
> a) Node1 - Publication publishing employee table - pub1
> b) Node2 - Subscription subscribing from publication pub1 with
> publish_local_only - sub1_pub1_node1
> c) Node2 - Publication publishing employee table - pub2
> d) Node1 - Subscription subscribing from publication pub2 with
> publish_local_only - sub2_pub2_node2
> 
> To create a subscription in node3, we will be using the following steps:
> a) Node2 - Publication publishing employee table. - pub3
> b) Node3 - Subscription subscribing from publication in Node2 with
> publish_local_only - sub3_pub3_node2
> 
> When we create a subscription in Node3, Node3 will connect to
> Node2(this will not be done in Node3) and check if the employee table
> is present in pg_subscription_rel, in our case Node2 will have
> employee table present in pg_subscription_rel (sub1_pub1_node1
> subscribing to employee table from pub1 in Node1). As employee table
> is being subscribed in node2 from node1, we will throw an error like
> below:
> postgres=# create subscription sub2 CONNECTION 'dbname =postgres port
> = 9999' publication pub2 with (publish_local_only=on);
> ERROR:  CREATE/ALTER SUBSCRIPTION with publish_local_only and
> copy_data as true is not allowed when the publisher might have
> replicated data, table:public.t1 might have replicated data in the
> publisher
> HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force

Thanks for kind explanation. 
I read above and your doc in 0002, and I put some comments.

1. alter_subscription.sgml

```
-        <term><literal>copy_data</literal> (<type>boolean</type>)</term>
+        <term><literal>copy_data</literal> (<type>boolean</type> | <literal>force</literal>)</term>
```

I thought that it should be written as enum. For example, huge_pages GUC parameter
can accept {on, off, try}, and it has been written as enum.

2. create_subscription.sgml

```
-        <term><literal>copy_data</literal> (<type>boolean</type>)</term>
+        <term><literal>copy_data</literal> (<type>boolean</type> | <literal>force</literal>)</term>
```

Same as above.

3. create_subscription.sgml

```
+
+         <para>
+          If the publication tables were also subscribing data in the publisher
+          from other publishers, it will affect the
+          <command>CREATE SUBSCRIPTION</command> based on the value specified
+          for <literal>publish_local_only</literal> option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>
```

I seeked docs, but the words " publication tables " have not seen.
How about "tables in the publication"?

4. create_subscription.sgml - about your example 

In the first section, we should describe about 2-nodes case more detail
like Amit mentioned in [1]. I thought that Option-3 can be resolved by defining
subscriptions in both nodes with publish_local_only = true and copy_data = force.

> I was initially planning to add srreplicateddata field but I have
> changed it slightly to keep the design simple. Now we just check if
> the relation is present in pg_subscription_rel and throw an error if
> copy_data and publish_local_only option is specified. The changes for
> the same are available at [1].
> 
> [1] -
> https://www.postgresql.org/message-id/CALDaNm0V%2B%3Db%3DCeZJNAAU
> O2PmSXH5QzNX3jADXb-0hGO_jVj0vA%40mail.gmail.com
> Thoughts?

Actually I doubted that the column is really needed, so it is good.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  39. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-03-15T04:25:21Z

    Dear Vignesh,

> Thanks for kind explanation.
> I read above and your doc in 0002, and I put some comments.

I forgot a comment about 0002 doc.

5. create_subscription.sgml - about your example

Three possibilities were listed in the doc,
but I was not sure about b) case.
In the situation Node1 and Node2 have already become multi-master,
and data has already synced at that time.
If so, how do we realize that "there is data present only in one Node"?
Case a) and c) seem reasonable.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  40. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-16T05:45:09Z

    On Tue, Mar 15, 2022 at 7:09 AM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Vignesh,
>
> Thank you for updating your patch!
>
> > Let's consider an existing Multi master logical replication setup
> > between Node1 and Node2 that is created using the following steps:
> > a) Node1 - Publication publishing employee table - pub1
> > b) Node2 - Subscription subscribing from publication pub1 with
> > publish_local_only - sub1_pub1_node1
> > c) Node2 - Publication publishing employee table - pub2
> > d) Node1 - Subscription subscribing from publication pub2 with
> > publish_local_only - sub2_pub2_node2
> >
> > To create a subscription in node3, we will be using the following steps:
> > a) Node2 - Publication publishing employee table. - pub3
> > b) Node3 - Subscription subscribing from publication in Node2 with
> > publish_local_only - sub3_pub3_node2
> >
> > When we create a subscription in Node3, Node3 will connect to
> > Node2(this will not be done in Node3) and check if the employee table
> > is present in pg_subscription_rel, in our case Node2 will have
> > employee table present in pg_subscription_rel (sub1_pub1_node1
> > subscribing to employee table from pub1 in Node1). As employee table
> > is being subscribed in node2 from node1, we will throw an error like
> > below:
> > postgres=# create subscription sub2 CONNECTION 'dbname =postgres port
> > = 9999' publication pub2 with (publish_local_only=on);
> > ERROR:  CREATE/ALTER SUBSCRIPTION with publish_local_only and
> > copy_data as true is not allowed when the publisher might have
> > replicated data, table:public.t1 might have replicated data in the
> > publisher
> > HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force
>
> Thanks for kind explanation.
> I read above and your doc in 0002, and I put some comments.
>
> 1. alter_subscription.sgml
>
> ```
> -        <term><literal>copy_data</literal> (<type>boolean</type>)</term>
> +        <term><literal>copy_data</literal> (<type>boolean</type> | <literal>force</literal>)</term>
> ```
>
> I thought that it should be written as enum. For example, huge_pages GUC parameter
> can accept {on, off, try}, and it has been written as enum.

Modified

> 2. create_subscription.sgml
>
> ```
> -        <term><literal>copy_data</literal> (<type>boolean</type>)</term>
> +        <term><literal>copy_data</literal> (<type>boolean</type> | <literal>force</literal>)</term>
> ```
>
> Same as above.

Modified

> 3. create_subscription.sgml
>
> ```
> +
> +         <para>
> +          If the publication tables were also subscribing data in the publisher
> +          from other publishers, it will affect the
> +          <command>CREATE SUBSCRIPTION</command> based on the value specified
> +          for <literal>publish_local_only</literal> option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
> ```
>
> I seeked docs, but the words " publication tables " have not seen.
> How about "tables in the publication"?

Modified

> 4. create_subscription.sgml - about your example
>
> In the first section, we should describe about 2-nodes case more detail
> like Amit mentioned in [1]. I thought that Option-3 can be resolved by defining
> subscriptions in both nodes with publish_local_only = true and copy_data = force.

I thought existing information is enough because we have mentioned
that node1 and node2 have bidirectional replication setup done and
both the table data will be replicated and synchronized as and when
the DML operations are happening. In option-3 we need to create a
subscription with copy_data as force to one node and copy_data as
false to another node because both nodes will be having the same data,
copying the data just from one of the nodes should be enough.

Thanks for the comments, the attached v5 patch has the changes for the same.

Regards,
Vignesh

  41. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-03-16T05:47:16Z

    On Tue, Mar 15, 2022 at 9:55 AM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Vignesh,
>
> > Thanks for kind explanation.
> > I read above and your doc in 0002, and I put some comments.
>
> I forgot a comment about 0002 doc.
>
> 5. create_subscription.sgml - about your example
>
> Three possibilities were listed in the doc,
> but I was not sure about b) case.
> In the situation Node1 and Node2 have already become multi-master,
> and data has already synced at that time.
> If so, how do we realize that "there is data present only in one Node"?
> Case a) and c) seem reasonable.

Your point is valid, modified.

The changes for the same are available int the v5 patch available at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com

Regards,
Vignesh

  42. Re: Handle infinite recursion in logical replication setup

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2022-03-30T13:52:12Z

    Hi Vignesh,
Some review comments on 0001

On Wed, Mar 16, 2022 at 11:17 AM vignesh C <vignesh21@gmail.com> wrote:

>
> The changes for the same are available int the v5 patch available at [1].
> [1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com
>

     cb->truncate_cb = pg_decode_truncate;
     cb->commit_cb = pg_decode_commit_txn;
     cb->filter_by_origin_cb = pg_decode_filter;
+    cb->filter_remote_origin_cb = pg_decode_filter_remote_origin;

Why do we need a new hook? Can we use filter_by_origin_cb? Also it looks like
implementation of pg_decode_filter and pg_decode_filter_remote_origin is same,
unless my eyes are deceiving me.

-      <literal>binary</literal>, <literal>streaming</literal>, and
-      <literal>disable_on_error</literal>.
+      <literal>binary</literal>, <literal>streaming</literal>,
+      <literal>disable_on_error</literal> and
+      <literal>publish_local_only</literal>.

"publish_local_only" as a "subscription" option looks odd. Should it be
"subscribe_local_only"?


+       <varlistentry>
+        <term><literal>publish_local_only</literal>
(<type>boolean</type>)</term>
+        <listitem>
+         <para>
+          Specifies whether the subscription should subscribe only to the
+          locally generated changes or subscribe to both the locally generated
+          changes and the replicated changes that was generated from other
+          nodes in the publisher. The default is <literal>false</literal>.

This description to uses verb "subscribe" instead of "publish".

@@ -936,6 +951,13 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
                         = true;
                 }

+                if (IsSet(opts.specified_opts, SUBOPT_PUBLISH_LOCAL_ONLY))
+                {
+                    values[Anum_pg_subscription_sublocal - 1] =
+                        BoolGetDatum(opts.streaming);

should this be opts.sublocal?

     cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
     cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
     cb->filter_by_origin_cb = pgoutput_origin_filter;
+    cb->filter_remote_origin_cb = pgoutput_remote_origin_filter;

pgoutput_origin_filter just returns false now. I think we should just enhance
that function and reuse the callback, instead of adding a new callback.

--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -99,6 +99,8 @@ typedef struct LogicalDecodingContext
      */
     bool        twophase_opt_given;

+    bool        only_local;        /* publish only locally generated data */
+

If we get rid of the new callback, we won't need this new member as well.
Anyway the filtering happens within the output plugin. There is nothing that
core is required to do here.

--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -183,6 +183,7 @@ typedef struct
             bool        streaming;    /* Streaming of large transactions */
             bool        twophase;    /* Streaming of two-phase transactions at
                                      * prepare time */
+            bool        only_local; /* publish only locally generated data */

Are we using this anywhere. I couldn't spot any usage of this member. I might
be missing something though.



-- 
Best Wishes,
Ashutosh Bapat

  43. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-31T03:44:53Z

    On Wed, Mar 30, 2022 at 7:40 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
> >
> > The changes for the same are available int the v5 patch available at [1].
> > [1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com
> >
>
>      cb->truncate_cb = pg_decode_truncate;
>      cb->commit_cb = pg_decode_commit_txn;
>      cb->filter_by_origin_cb = pg_decode_filter;
> +    cb->filter_remote_origin_cb = pg_decode_filter_remote_origin;
>
> Why do we need a new hook? Can we use filter_by_origin_cb?
>

I also think there is no need for a new hook in this case but I might
also be missing something that Vignesh has in his mind.

> Also it looks like
> implementation of pg_decode_filter and pg_decode_filter_remote_origin is same,
> unless my eyes are deceiving me.
>
> -      <literal>binary</literal>, <literal>streaming</literal>, and
> -      <literal>disable_on_error</literal>.
> +      <literal>binary</literal>, <literal>streaming</literal>,
> +      <literal>disable_on_error</literal> and
> +      <literal>publish_local_only</literal>.
>
> "publish_local_only" as a "subscription" option looks odd. Should it be
> "subscribe_local_only"?
>

I also think "subscribe_local_only" fits better here.

-- 
With Regards,
Amit Kapila.

  44. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-03-31T09:21:54Z

    On Thu, Mar 31, 2022 at 9:14 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Mar 30, 2022 at 7:40 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > The changes for the same are available int the v5 patch available at [1].
> > > [1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com
> > >
> >
> >      cb->truncate_cb = pg_decode_truncate;
> >      cb->commit_cb = pg_decode_commit_txn;
> >      cb->filter_by_origin_cb = pg_decode_filter;
> > +    cb->filter_remote_origin_cb = pg_decode_filter_remote_origin;
> >
> > Why do we need a new hook? Can we use filter_by_origin_cb?
> >
>
> I also think there is no need for a new hook in this case but I might
> also be missing something that Vignesh has in his mind.
>
> > Also it looks like
> > implementation of pg_decode_filter and pg_decode_filter_remote_origin is same,
> > unless my eyes are deceiving me.
> >
> > -      <literal>binary</literal>, <literal>streaming</literal>, and
> > -      <literal>disable_on_error</literal>.
> > +      <literal>binary</literal>, <literal>streaming</literal>,
> > +      <literal>disable_on_error</literal> and
> > +      <literal>publish_local_only</literal>.
> >
> > "publish_local_only" as a "subscription" option looks odd. Should it be
> > "subscribe_local_only"?
> >
>
> I also think "subscribe_local_only" fits better here.
>

About 0002 patch,
Commit message:
------
If a subscription is created to Node1 from Node3 with publish_local_only
and copy_data as ON, then throw an error so that user can handle
creation of subscription with table having consistent data.
------

Do you want to refer to Node2 instead of Node3 here as Node3 doesn't
make sense in the description?

Also, you haven't explained anywhere in the patch why
'publish_local_only' (or whatever we call it) won't work for initial
sync? IIUC, it is because we can identify origin changes only based on
WAL and initial sync directly copies data from the heap. Am, I missing
something here?

-- 
With Regards,
Amit Kapila.

  45. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-01T07:04:21Z

    On Wed, Mar 30, 2022 at 7:22 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> Hi Vignesh,
> Some review comments on 0001
>
> On Wed, Mar 16, 2022 at 11:17 AM vignesh C <vignesh21@gmail.com> wrote:
>
> >
> > The changes for the same are available int the v5 patch available at [1].
> > [1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com
> >
>
>      cb->truncate_cb = pg_decode_truncate;
>      cb->commit_cb = pg_decode_commit_txn;
>      cb->filter_by_origin_cb = pg_decode_filter;
> +    cb->filter_remote_origin_cb = pg_decode_filter_remote_origin;
>
> Why do we need a new hook? Can we use filter_by_origin_cb? Also it looks like
> implementation of pg_decode_filter and pg_decode_filter_remote_origin is same,
> unless my eyes are deceiving me.

I have used filter_by_origin_cb for the implementation, removed
filter_remote_origin_cb

> -      <literal>binary</literal>, <literal>streaming</literal>, and
> -      <literal>disable_on_error</literal>.
> +      <literal>binary</literal>, <literal>streaming</literal>,
> +      <literal>disable_on_error</literal> and
> +      <literal>publish_local_only</literal>.
>
> "publish_local_only" as a "subscription" option looks odd. Should it be
> "subscribe_local_only"?

Modified

>
> +       <varlistentry>
> +        <term><literal>publish_local_only</literal>
> (<type>boolean</type>)</term>
> +        <listitem>
> +         <para>
> +          Specifies whether the subscription should subscribe only to the
> +          locally generated changes or subscribe to both the locally generated
> +          changes and the replicated changes that was generated from other
> +          nodes in the publisher. The default is <literal>false</literal>.
>
> This description to uses verb "subscribe" instead of "publish".

Modified

> @@ -936,6 +951,13 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>                          = true;
>                  }
>
> +                if (IsSet(opts.specified_opts, SUBOPT_PUBLISH_LOCAL_ONLY))
> +                {
> +                    values[Anum_pg_subscription_sublocal - 1] =
> +                        BoolGetDatum(opts.streaming);
>
> should this be opts.sublocal?

Yes you are right, Modified

>      cb->commit_prepared_cb = pgoutput_commit_prepared_txn;
>      cb->rollback_prepared_cb = pgoutput_rollback_prepared_txn;
>      cb->filter_by_origin_cb = pgoutput_origin_filter;
> +    cb->filter_remote_origin_cb = pgoutput_remote_origin_filter;
>
> pgoutput_origin_filter just returns false now. I think we should just enhance
> that function and reuse the callback, instead of adding a new callback.

Modified

> --- a/src/include/replication/logical.h
> +++ b/src/include/replication/logical.h
> @@ -99,6 +99,8 @@ typedef struct LogicalDecodingContext
>       */
>      bool        twophase_opt_given;
>
> +    bool        only_local;        /* publish only locally generated data */
> +
>
> If we get rid of the new callback, we won't need this new member as well.
> Anyway the filtering happens within the output plugin. There is nothing that
> core is required to do here.

Modified

> --- a/src/include/replication/walreceiver.h
> +++ b/src/include/replication/walreceiver.h
> @@ -183,6 +183,7 @@ typedef struct
>              bool        streaming;    /* Streaming of large transactions */
>              bool        twophase;    /* Streaming of two-phase transactions at
>                                       * prepare time */
> +            bool        only_local; /* publish only locally generated data */
>
> Are we using this anywhere. I couldn't spot any usage of this member. I might
> be missing something though.

This is set in ApplyWorkerMain before calling libpqrcv_startstreaming.
This will be used in building the START_REPLICATION command.
Thanks for the comments, the attached v6 patch has the changes for the same.

Regards,
Vignesh

  46. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-01T07:15:50Z

    On Thu, Mar 31, 2022 at 2:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Mar 31, 2022 at 9:14 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Mar 30, 2022 at 7:40 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > > >
> > > > The changes for the same are available int the v5 patch available at [1].
> > > > [1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com
> > > >
> > >
> > >      cb->truncate_cb = pg_decode_truncate;
> > >      cb->commit_cb = pg_decode_commit_txn;
> > >      cb->filter_by_origin_cb = pg_decode_filter;
> > > +    cb->filter_remote_origin_cb = pg_decode_filter_remote_origin;
> > >
> > > Why do we need a new hook? Can we use filter_by_origin_cb?
> > >
> >
> > I also think there is no need for a new hook in this case but I might
> > also be missing something that Vignesh has in his mind.
> >
> > > Also it looks like
> > > implementation of pg_decode_filter and pg_decode_filter_remote_origin is same,
> > > unless my eyes are deceiving me.
> > >
> > > -      <literal>binary</literal>, <literal>streaming</literal>, and
> > > -      <literal>disable_on_error</literal>.
> > > +      <literal>binary</literal>, <literal>streaming</literal>,
> > > +      <literal>disable_on_error</literal> and
> > > +      <literal>publish_local_only</literal>.
> > >
> > > "publish_local_only" as a "subscription" option looks odd. Should it be
> > > "subscribe_local_only"?
> > >
> >
> > I also think "subscribe_local_only" fits better here.
> >
>
> About 0002 patch,
> Commit message:
> ------
> If a subscription is created to Node1 from Node3 with publish_local_only
> and copy_data as ON, then throw an error so that user can handle
> creation of subscription with table having consistent data.
> ------
> Do you want to refer to Node2 instead of Node3 here as Node3 doesn't
> make sense in the description?

It should be Node3, I have added more details in the commit message
mentioning about the scenario.

> Also, you haven't explained anywhere in the patch why
> 'publish_local_only' (or whatever we call it) won't work for initial
> sync? IIUC, it is because we can identify origin changes only based on
> WAL and initial sync directly copies data from the heap. Am, I missing
> something here?

I have added the explanation where we are throwing the error.

The v6 patch at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm16eBx2BjLFjfFHSU4pb25HmgV--M692OPgH_91Dwn%3D2g%40mail.gmail.com

Regards,
Vignesh

  47. Re: Handle infinite recursion in logical replication setup

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2022-04-04T13:20:08Z

    I didn't find this in https://commitfest.postgresql.org/37/. Is this
somewhere in the commitfest?

On Fri, Apr 1, 2022 at 12:46 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, Mar 31, 2022 at 2:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Mar 31, 2022 at 9:14 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Mar 30, 2022 at 7:40 PM Ashutosh Bapat
> > > <ashutosh.bapat.oss@gmail.com> wrote:
> > > > >
> > > > > The changes for the same are available int the v5 patch available at [1].
> > > > > [1] - https://www.postgresql.org/message-id/CALDaNm3wCf0YcvVo%2BgHMGpupk9K6WKJxCyLUvhPC2GkPKRZUWA%40mail.gmail.com
> > > > >
> > > >
> > > >      cb->truncate_cb = pg_decode_truncate;
> > > >      cb->commit_cb = pg_decode_commit_txn;
> > > >      cb->filter_by_origin_cb = pg_decode_filter;
> > > > +    cb->filter_remote_origin_cb = pg_decode_filter_remote_origin;
> > > >
> > > > Why do we need a new hook? Can we use filter_by_origin_cb?
> > > >
> > >
> > > I also think there is no need for a new hook in this case but I might
> > > also be missing something that Vignesh has in his mind.
> > >
> > > > Also it looks like
> > > > implementation of pg_decode_filter and pg_decode_filter_remote_origin is same,
> > > > unless my eyes are deceiving me.
> > > >
> > > > -      <literal>binary</literal>, <literal>streaming</literal>, and
> > > > -      <literal>disable_on_error</literal>.
> > > > +      <literal>binary</literal>, <literal>streaming</literal>,
> > > > +      <literal>disable_on_error</literal> and
> > > > +      <literal>publish_local_only</literal>.
> > > >
> > > > "publish_local_only" as a "subscription" option looks odd. Should it be
> > > > "subscribe_local_only"?
> > > >
> > >
> > > I also think "subscribe_local_only" fits better here.
> > >
> >
> > About 0002 patch,
> > Commit message:
> > ------
> > If a subscription is created to Node1 from Node3 with publish_local_only
> > and copy_data as ON, then throw an error so that user can handle
> > creation of subscription with table having consistent data.
> > ------
> > Do you want to refer to Node2 instead of Node3 here as Node3 doesn't
> > make sense in the description?
>
> It should be Node3, I have added more details in the commit message
> mentioning about the scenario.
>
> > Also, you haven't explained anywhere in the patch why
> > 'publish_local_only' (or whatever we call it) won't work for initial
> > sync? IIUC, it is because we can identify origin changes only based on
> > WAL and initial sync directly copies data from the heap. Am, I missing
> > something here?
>
> I have added the explanation where we are throwing the error.
>
> The v6 patch at [1] has the changes for the same.
> [1] - https://www.postgresql.org/message-id/CALDaNm16eBx2BjLFjfFHSU4pb25HmgV--M692OPgH_91Dwn%3D2g%40mail.gmail.com
>
> Regards,
> Vignesh



-- 
Best Wishes,
Ashutosh Bapat

  48. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-05T00:51:20Z

    Here are my comments for the latest patch v6-0001.

(I will post my v6-0002 review comments separately)

PATCH v6-0001 comments
======================

1.1 General - Option name

I still feel like the option name is not ideal. Unfortunately, this is
important because any name change would impact lots of these patch
files and docs, struct members etc.

It was originally called "local_only", but I thought that as a
SUBSCRIPTION option that was confusing because "local" means local to
what? Really it is local to the publisher, not local to the
subscriber, so that name seemed misleading.

Then I suggested "publish_local_only". Although that resolved the
ambiguity problem, other people thought it seemed odd to have the
"publish" prefix for a subscription-side option.

So now it is changed again to "subscribe_local_only" -- It's getting
better but still, it is implied that the "local" means local to the
publisher except there is nothing in the option name really to convey
that meaning. IMO we here all understand the meaning of this option
mostly by familiarity with this discussion thread, but I think a user
coming to this for the first time will still be confused by the name.

Below are some other name ideas. Maybe they are not improvements, but
it might help other people to come up with something better.

subscribe_publocal_only = true/false
origin = pub_only/any
adjacent_data_only = true/false
direct_subscriptions_only = true/false
...


(FYI, the remainder of these review comments will assume the option is
still called "subscribe_local_only")

~~~

1.2 General - inconsistent members and args

IMO the struct members and args should also be named for close
consistency with whatever the option name is.

Currently the option is called "subscription_local_only".  So I think
the members/args would be better to be called "local_only" instead of
"only_local".

~~~

1.3 Commit message - wrong option name

The commit message refers to the option name as "publish_local_only"
instead of the option name that is currently implemented.

~~~

1.4 Commit message - wording

The wording seems a bit off. Below is suggested simpler wording which
I AFAIK conveys the same information.

BEFORE
Add an option publish_local_only which will subscribe only to the locally
generated data in the publisher node. If subscriber is created with this
option, publisher will skip publishing the data that was subscribed
from other nodes. It can be created using following syntax:
ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999'
PUBLICATION pub1 with (publish_local_only = on);

SUGGESTION
This patch adds a new SUBSCRIPTION boolean option
"subscribe_local_only". The default is false. When a SUBSCRIPTION is
created with this option enabled, the publisher will only publish data
that originated at the publisher node.
Usage:
CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999'
PUBLICATION pub1 with (subscribe_local_only = true);

~~~

1.5 doc/src/sgml/ref/create_subscription.sgml - "generated" changes.

+         <para>
+          Specifies whether the subscription will request the publisher to send
+          locally generated changes or both the locally generated changes and
+          the replicated changes that was generated from other nodes. The
+          default is <literal>false</literal>.
+         </para>

For some reason, it seemed a bit strange to me to use the term
"generated" changes. Maybe better to refer to the origin of changes?

SUGGESTION
Specifies whether the publisher should send only changes that
originated locally at the publisher node, or send any publisher node
changes regardless of their origin. The default is false.

~~~

1.6 src/backend/replication/pgoutput/pgoutput.c -
LOGICALREP_PROTO_TWOPHASE_VERSION_NUM

@@ -496,6 +509,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
  else
  ctx->twophase_opt_given = true;

+ if (data->only_local && data->protocol_version <
LOGICALREP_PROTO_TWOPHASE_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support
subscribe_local_only, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_TWOPHASE_VERSION_NUM)));

I thought this code should not be using
LOGICALREP_PROTO_TWOPHASE_VERSION_NUM. Shouldn't there be some newly
introduced constant like LOGICALREP_PROTO_LOCALONLY_VERSION_NUM which
you will use here?

~~~

1.7 src/bin/pg_dump/pg_dump.c - 150000

@@ -4451,11 +4452,13 @@ getSubscriptions(Archive *fout)
  if (fout->remoteVersion >= 150000)
  appendPQExpBufferStr(query,
  " s.subtwophasestate,\n"
- " s.subdisableonerr\n");
+ " s.subdisableonerr,\n"
+ " s.sublocal\n");
  else
  appendPQExpBuffer(query,
    " '%c' AS subtwophasestate,\n"
-   " false AS subdisableonerr\n",
+   " false AS subdisableonerr,\n"
+   " false AS s.sublocal\n",
    LOGICALREP_TWOPHASE_STATE_DISABLED);

I think this local_only feature is unlikely to get into the PG15
release, so this code should be split out into a separate condition
because later will need to change to say >= 160000.

~~~

1.8 src/bin/pg_dump/pg_dump.c - dumpSubscription

@@ -4585,6 +4591,9 @@ dumpSubscription(Archive *fout, const
SubscriptionInfo *subinfo)
  if (strcmp(subinfo->subdisableonerr, "t") == 0)
  appendPQExpBufferStr(query, ", disable_on_error = true");

+ if (strcmp(subinfo->sublocal, "f") != 0)
+ appendPQExpBufferStr(query, ", subscribe_local_only = on");
+

I felt it is more natural to say "if it is true set to true", instead
of "if it is not false set to on".

SUGGESTION
if (strcmp(subinfo->sublocal, "t") == 0)
appendPQExpBufferStr(query, ", subscribe_local_only = true");

~~~

1.9 src/bin/psql/describe.c - 150000

@@ -6318,9 +6318,11 @@ describeSubscriptions(const char *pattern, bool verbose)
  if (pset.sversion >= 150000)
  appendPQExpBuffer(&buf,
    ", subtwophasestate AS \"%s\"\n"
-   ", subdisableonerr AS \"%s\"\n",
+   ", subdisableonerr AS \"%s\"\n"
+   ", sublocal AS \"%s\"\n",
    gettext_noop("Two phase commit"),
-   gettext_noop("Disable on error"));
+   gettext_noop("Disable on error"),
+   gettext_noop("Only local"));

I think this local_only feature is unlikely to get into the PG15
release, so this code should be split out into a separate condition
because later will need to change to say >= 160000.

~~~

1.10 src/bin/psql/describe.c - describeSubscriptions column

@@ -6318,9 +6318,11 @@ describeSubscriptions(const char *pattern, bool verbose)
  if (pset.sversion >= 150000)
  appendPQExpBuffer(&buf,
    ", subtwophasestate AS \"%s\"\n"
-   ", subdisableonerr AS \"%s\"\n",
+   ", subdisableonerr AS \"%s\"\n"
+   ", sublocal AS \"%s\"\n",
    gettext_noop("Two phase commit"),
-   gettext_noop("Disable on error"));
+   gettext_noop("Disable on error"),
+   gettext_noop("Only local"));

I think the column name here should be more consistent with the option
name. e.g. it should be "Local only", not "Only local".

~~~

1.11 src/bin/psql/tab-complete.c - whitespace

@@ -3167,7 +3167,7 @@ psql_completion(const char *text, int start, int end)
  /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
  else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
  COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
-   "enabled", "slot_name", "streaming",
+   "enabled",  "slot_name", "streaming", "subscribe_local_only",

The patch accidentally added a space char before the "slot_name".

~~~

1.12 src/include/replication/walreceiver.h - "generated"

@@ -183,6 +183,7 @@ typedef struct
  bool streaming; /* Streaming of large transactions */
  bool twophase; /* Streaming of two-phase transactions at
  * prepare time */
+ bool only_local; /* publish only locally generated data */

This is a similar review comment as #1.5 about saying the word "generated".
Maybe there is another way to word this?

~~~

1.13 src/test/regress/sql/subscription.sql - missing test case

Isn't there a missing test case for ensuring that the new option is boolean?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  49. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-05T01:54:37Z

    On Mon, Apr 4, 2022 at 6:50 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> I didn't find this in https://commitfest.postgresql.org/37/. Is this
> somewhere in the commitfest?

I missed adding it earlier, I have added it to commitfest today:
https://commitfest.postgresql.org/38/3610/

Regards,
Vignesh

  50. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-05T02:36:18Z

    On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my comments for the latest patch v6-0001.
>
> (I will post my v6-0002 review comments separately)
>
> PATCH v6-0001 comments
> ======================
>
> 1.1 General - Option name
>
> I still feel like the option name is not ideal. Unfortunately, this is
> important because any name change would impact lots of these patch
> files and docs, struct members etc.
>
> It was originally called "local_only", but I thought that as a
> SUBSCRIPTION option that was confusing because "local" means local to
> what? Really it is local to the publisher, not local to the
> subscriber, so that name seemed misleading.
>
> Then I suggested "publish_local_only". Although that resolved the
> ambiguity problem, other people thought it seemed odd to have the
> "publish" prefix for a subscription-side option.
>
> So now it is changed again to "subscribe_local_only" -- It's getting
> better but still, it is implied that the "local" means local to the
> publisher except there is nothing in the option name really to convey
> that meaning. IMO we here all understand the meaning of this option
> mostly by familiarity with this discussion thread, but I think a user
> coming to this for the first time will still be confused by the name.
>
> Below are some other name ideas. Maybe they are not improvements, but
> it might help other people to come up with something better.
>
> subscribe_publocal_only = true/false
> origin = pub_only/any
> adjacent_data_only = true/false
> direct_subscriptions_only = true/false
> ...
>

I think local_only with a description should be okay considering other
current options. Some of the options like slot_name, binary, etc. are
publisher-specific which becomes clear only after reading the
description. I am not sure adding pub*/sub* to it is much helpful. So,
+1 to a simple boolean like 'local_only', 'data_local', or something
like that.

-- 
With Regards,
Amit Kapila.

  51. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-05T02:47:04Z

    Here are my comments for the latest patch v6-0002.

PATCH v6-0002 comments
======================

2.1 General - should this be an independent patch?

In many ways, I think most of this patch is unrelated to the other
"local_only" patch (v6-0001).

For example, IIUC even in the current HEAD, we could consider it to be
a user error if multiple SUBSCRIPTIONS or multiple PUBLICATIONS of the
same SUBSCRIPTION are replicating to the same TABLE on the same node
and using "copy_data = on".

So I think it would be ok to throw an ERROR if such a copy_data clash
is detected, and then the user will have to change to use "copy_data =
off" for some/all of them to avoid data duplication.

The "local_only" option only adds some small logic to this new ERROR,
but it's not really a prerequisite at all.

e.g. this whole ERROR part of the patch can be a separate thread.

~~~

2.2 General - can we remove the "force" enum?

Now, because I consider the clashing "copy_data = on" ERROR to be a
user error, I think that is something that the user can already take
care of themselves just using the copy_data = off.

I did not really like the modifying of the "copy_data" option from
just boolean to some kind of hybrid boolean + "force".

a) It smells a bit off to me. IMO replication is supposed to end up
with the same (replicated) data on the standby machine but this
"force" mode seems to be just helping the user to break that concept
and say - "I know what I'm doing, and I don't care if I get lots of
duplicated data in the replica table - just let me do it"...

b) It also somehow feels like the new "force" was introduced mostly to
make the code ERROR handling implementation simpler, rather than to
make the life of the end-user better. Yes, if force is removed maybe
the copy-clash-detection-code will need to be internally quite more
complex than it is now, but that is how it should be, instead of
putting extra burden on the user (e.g. by complicating the PG docs and
giving them yet more alternatives to configure). I think any clashing
copy_data options really is a user error, but also I think the current
boolean copy_data true/false already gives the user a way to fix it.

c) Actually, your new error hint messages are similar to my
perspective: They already say "Use CREATE/ALTER SUBSCRIPTION with
copy_data = off or force". All I am saying is remove the "force", and
the user can still fix the problem just by using "copy_data = off"
appropriately.

======

So (from above) I am not much in favour of the copy_data becoming a
hybrid enum and using "force", yet that is what most of this patch is
implementing. Anyway, the remainder of my review comments here are for
the code in its current form. Maybe if "force" can be removed most of
the following comments end up being redundant.

======

2.3 Commit message - wording

This message is difficult to understand.

I think that the long sentence "Now when user is trying..." can be
broken into more manageable parts.

This part "and throw an error so that user can handle the initial copy
data." also seemed a bit vague.

~~~

2.4 Commit message - more functions

"This patch does couple of things:"

IIUC, there seems a third thing implemented by this patch but not
described by the comment. I think it also adds support for ALTER
SUBSCRIPTION SET PUBLICATION WITH (subscribe_local_only)

~~~

2.5 doc/src/sgml/ref/create_subscription.sgml - wording

@@ -161,6 +161,13 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
           the replicated changes that was generated from other nodes. The
           default is <literal>false</literal>.
          </para>
+         <para>
+          If the tables in the publication were also subscribing to the data in
+          the publisher from other publishers, it will affect the
+          <command>CREATE SUBSCRIPTION</command> based on the value specified
+          for <literal>copy_data</literal> option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>

Is there is a simpler way to express all that?

SUGGESTION
There is some interation between the option "subscribe_local_only" and
option "copy_data". Refer to the <xref
linkend="sql-createsubscription-notes" /> for details.

~~~

2.6 doc/src/sgml/ref/create_subscription.sgml - whitespace

@@ -213,18 +220,28 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
        </varlistentry>

        <varlistentry>
-        <term><literal>copy_data</literal> (<type>boolean</type>)</term>
+        <term><literal>copy_data</literal> (<type>enum</type>)</term>
         <listitem>
          <para>
           Specifies whether to copy pre-existing data in the publications
-          that are being subscribed to when the replication starts.
-          The default is <literal>true</literal>.
+          that are being subscribed to when the replication starts. This
+          parameter may be either <literal>true</literal>,
+          <literal>false</literal> or <literal>force</literal>. The default is
+          <literal>true</literal>.

That last line has trailing whitespace.

~~~

2.7 doc/src/sgml/ref/create_subscription.sgml - wording

+         <para>
+          If the tables in the publication were also subscribing to the data in
+          the publisher from other publishers, it will affect the
+          <command>CREATE SUBSCRIPTION</command> based on the value specified
+          for <literal>subscribe_local_only</literal> option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>

This is similar to review comment #2.5 which I thought could be
written in a simpler way.

~~~

2.8 doc/src/sgml/ref/create_subscription.sgml

@@ -375,6 +392,42 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   Let's consider an existing Multi master logical replication setup between
+   Node1 and Node2 that is created using the following steps:
+   a) Node1 - Publication publishing employee table.
+   b) Node2 - Subscription subscribing from publication pub1 with
+   <literal>subscribe_local_only</literal>.
+   c) Node2 - Publication publishing employee table.
+   d) Node1 - Subscription subscribing from publication pub2 with
+   <literal>subscribe_local_only</literal>.
+   Now when user is trying to add another node Node3 to the above Multi master
+   logical replication setup, user will have to create one subscription
+   subscribing from Node1 and another subscription subscribing from Node2 in
+   Node3 using <literal>subscribe_local_only</literal> option and
+   <literal>copy_data</literal> as <literal>true</literal>, while
+   the subscription is created, server will identify that Node2 is subscribing
+   from Node1 and Node1 is subscribing from Node2 and throw an error like:
+   <programlisting>
+postgres=# CREATE SUBSCRIPTION mysub CONNECTION 'host=192.168.1.50
port=5432 user=foo dbname=foodb'
+           PUBLICATION mypublication, insert_only with
(subscribe_local_only=on);
+ERROR:  CREATE/ALTER SUBSCRIPTION with subscribe_local_only and
copy_data as true is not allowed when the publisher might have
replicated data, table:public.t1 might have replicated data in the
publisher
+HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force
+   </programlisting>
+   In this scenario user can solve this based on one of the 2 possibilities,
+   a) If there are no data present in Node1 and Node2, then the user can create
+   the subscriptions to Node1 and Node2 with
+   <literal>subscribe_local_only</literal> as <literal>true</literal> and
+   <literal>copy_data</literal> as <literal>false</literal>. b) If the data is
+   present, then the user can create subscription with
+   <literal>copy_data</literal> as <literal>force</literal> on Node1 and
+   <literal>copy_data</literal> as <literal>false</literal> on Node2, before
+   allowing any operations on the respective tables of Node1 and Node2, in this
+   case <literal>copy_data</literal> is <literal>false</literal> on Node2
+   because the data will be replicated to each other and available on both the
+   nodes.
+  </para>
+

That is a large slab of text in the Notes, so not very easy to digest it.

I'm not sure what to suggest for this -
- Perhaps the a,b,c,d should all be "lists" so it renders differently?
- It almost seems like too much information to be included in this
"Notes" section, Maybe it needs its own full page in PG Docs "Logical
Replication" to discuss this topic.

~~~

2.9 src/backend/commands/subscriptioncmds.c - IS_COPY_DATA_VALID

@@ -69,6 +69,18 @@
 /* check if the 'val' has 'bits' set */
 #define IsSet(val, bits)  (((val) & (bits)) == (bits))

+#define IS_COPY_DATA_VALID(copy_data) (copy_data != COPY_DATA_OFF)

The macro seems misnamed because "off" is also "valid". It seems like
it should be called something different like IS_COPY_DATA, or
IS_COPY_DATA_ON_OR_FORCE, etc.

~~~

2.10 src/backend/commands/subscriptioncmds.c - DefGetCopyData

+ /*
+ * The set of strings accepted here should match up with
+ * the grammar's opt_boolean_or_string production.
+ */
+ if (pg_strcasecmp(sval, "true") == 0)
+ return COPY_DATA_ON;
+ if (pg_strcasecmp(sval, "false") == 0)
+ return COPY_DATA_OFF;
+ if (pg_strcasecmp(sval, "on") == 0)
+ return COPY_DATA_ON;
+ if (pg_strcasecmp(sval, "off") == 0)
+ return COPY_DATA_OFF;
+ if (pg_strcasecmp(sval, "force") == 0)
+ return COPY_DATA_FORCE;

Maybe combine the return for true/on and false/off?

~~~

2.11 src/backend/commands/subscriptioncmds.c - fetch_table_list

+ appendStringInfoString(&cmd,
+    "SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename,
PS.srsubstate as replicated\n"
+    "FROM pg_publication P,\n"
+    "LATERAL pg_get_publication_tables(P.pubname) GPT\n"
+    "LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
+    "pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
+    "WHERE C.oid = GPT.relid AND P.pubname in (");
+

That blank line is not needed (it was not there previously) because
the next get_publications_str is a continuation of this SQL.

~~~

2.12 src/backend/commands/subscriptioncmds.c - fetch_table_list comment

+ * It is quite possible that subscriber has not yet pulled data to
+ * the tables, but in ideal cases the table data will be subscribed.
+ * Too keep the code simple it is not checked if the subscriber table
+ * has pulled the data or not.
+ */

Typo "Too" -> "To"

~~~

2.13 src/backend/commands/subscriptioncmds.c - force

+ if (copydata == COPY_DATA_ON && only_local && !slot_attisnull(slot, 3))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("CREATE/ALTER SUBSCRIPTION with subscribe_local_only and
copy_data as true is not allowed when the publisher might have
replicated data, table:%s.%s might have replicated data in the
publisher",
+    nspname, relname),
+ errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));

AFAIK this is the only code of this patch that is distinguishing
between "force" and the "on". As I wrote at the beginning of this post
I feel you can keep this ERROR (maybe the way to detect it needs to be
more complex) and the user can fix it just by using "copy_data = off"
appropriately. So then copy_data does not need to be changed.

~~~

2.14 src/test/regress/sql/subscription.sql - missing tests

The new copy_data is not really an enum true/false/force like the PG
docs claims.

It seems more like some kind of hybrid boolean+force. So if that is
how it is going to be then there are missing test cases to make sure
that values like "on"/"off"/"0"/"1" still are working.

~~~

2.15 src/test/subscription/t/032_circular.pl

@@ -65,6 +69,25 @@ $node_A->safe_psql('postgres', "
  PUBLICATION tap_pub_B
  WITH (subscribe_local_only = on, copy_data = off)");

+($result, $stdout, $stderr) = $node_A->psql('postgres',  "
+        CREATE SUBSCRIPTION tap_sub_A1
+        CONNECTION '$node_B_connstr application_name=$appname_A'
+        PUBLICATION tap_pub_B
+        WITH (subscribe_local_only = on, copy_data = on)");
+like(
+        $stderr,
+        qr/ERROR:  CREATE\/ALTER SUBSCRIPTION with
subscribe_local_only and copy_data as true is not allowed when the
publisher might have replicated data/,
+        "Create subscription with subscribe_local_only and copy_data
having replicated table in publisher");
+
+$node_A->safe_psql('postgres',  "
+        CREATE SUBSCRIPTION tap_sub_A2
+        CONNECTION '$node_B_connstr application_name=$appname_A'
+        PUBLICATION tap_pub_B
+        WITH (subscribe_local_only = on, copy_data = force)");
+
+$node_A->safe_psql('postgres',  "
+        DROP SUBSCRIPTION tap_sub_A2");
+

Maybe underneath it is the same, but from the outside, this looks like
a slightly different scenario from what is mentioned everywhere else
in the patch.

I think it would be better to create a new Node_C (aka Node3) so then
the TAP test can use the same example that you give in the commit
message and the PG docs notes.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  52. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-05T03:09:01Z

    On Tue, Apr 5, 2022 at 12:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my comments for the latest patch v6-0001.
> >
> > (I will post my v6-0002 review comments separately)
> >
> > PATCH v6-0001 comments
> > ======================
> >
> > 1.1 General - Option name
> >
> > I still feel like the option name is not ideal. Unfortunately, this is
> > important because any name change would impact lots of these patch
> > files and docs, struct members etc.
> >
> > It was originally called "local_only", but I thought that as a
> > SUBSCRIPTION option that was confusing because "local" means local to
> > what? Really it is local to the publisher, not local to the
> > subscriber, so that name seemed misleading.
> >
> > Then I suggested "publish_local_only". Although that resolved the
> > ambiguity problem, other people thought it seemed odd to have the
> > "publish" prefix for a subscription-side option.
> >
> > So now it is changed again to "subscribe_local_only" -- It's getting
> > better but still, it is implied that the "local" means local to the
> > publisher except there is nothing in the option name really to convey
> > that meaning. IMO we here all understand the meaning of this option
> > mostly by familiarity with this discussion thread, but I think a user
> > coming to this for the first time will still be confused by the name.
> >
> > Below are some other name ideas. Maybe they are not improvements, but
> > it might help other people to come up with something better.
> >
> > subscribe_publocal_only = true/false
> > origin = pub_only/any
> > adjacent_data_only = true/false
> > direct_subscriptions_only = true/false
> > ...
> >
>
> I think local_only with a description should be okay considering other
> current options. Some of the options like slot_name, binary, etc. are
> publisher-specific which becomes clear only after reading the
> description. I am not sure adding pub*/sub* to it is much helpful. So,
> +1 to a simple boolean like 'local_only', 'data_local', or something
> like that.

+1 to use a simple boolean option.

Thanks for the examples of the other options which are also really
publisher-specific. Seen in that light, and with a clear description,
probably the original "local_only"  was fine after all.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  53. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-05T06:15:50Z

    On Tue, Apr 5, 2022 at 8:17 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my comments for the latest patch v6-0002.
>
> PATCH v6-0002 comments
> ======================
>
> 2.1 General - should this be an independent patch?
>

I think both the patches are dependent and might get committed
together if the concept proved to be useful and doesn't have flaws.

> In many ways, I think most of this patch is unrelated to the other
> "local_only" patch (v6-0001).
>
> For example, IIUC even in the current HEAD, we could consider it to be
> a user error if multiple SUBSCRIPTIONS or multiple PUBLICATIONS of the
> same SUBSCRIPTION are replicating to the same TABLE on the same node
> and using "copy_data = on".
>
> So I think it would be ok to throw an ERROR if such a copy_data clash
> is detected, and then the user will have to change to use "copy_data =
> off" for some/all of them to avoid data duplication.
>
> The "local_only" option only adds some small logic to this new ERROR,
> but it's not really a prerequisite at all.
>
> e.g. this whole ERROR part of the patch can be a separate thread.
>
> ~~~
>
> 2.2 General - can we remove the "force" enum?
>
> Now, because I consider the clashing "copy_data = on" ERROR to be a
> user error, I think that is something that the user can already take
> care of themselves just using the copy_data = off.
>
> I did not really like the modifying of the "copy_data" option from
> just boolean to some kind of hybrid boolean + "force".
>
> a) It smells a bit off to me. IMO replication is supposed to end up
> with the same (replicated) data on the standby machine but this
> "force" mode seems to be just helping the user to break that concept
> and say - "I know what I'm doing, and I don't care if I get lots of
> duplicated data in the replica table - just let me do it"...
>
> b) It also somehow feels like the new "force" was introduced mostly to
> make the code ERROR handling implementation simpler, rather than to
> make the life of the end-user better. Yes, if force is removed maybe
> the copy-clash-detection-code will need to be internally quite more
> complex than it is now, but that is how it should be, instead of
> putting extra burden on the user (e.g. by complicating the PG docs and
> giving them yet more alternatives to configure). I think any clashing
> copy_data options really is a user error, but also I think the current
> boolean copy_data true/false already gives the user a way to fix it.
>
> c) Actually, your new error hint messages are similar to my
> perspective: They already say "Use CREATE/ALTER SUBSCRIPTION with
> copy_data = off or force". All I am saying is remove the "force", and
> the user can still fix the problem just by using "copy_data = off"
> appropriately.
>

How will it work if there is more than one table? copy_data = "off"
means it won't copy any of the other tables in the corresponding
publication(s). I think it is primarily to give the user some option
where she understands the logical replication setup better and
understands that this won't be a problem.

-- 
With Regards,
Amit Kapila.

  54. Re: Handle infinite recursion in logical replication setup

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2022-04-05T13:36:18Z

    On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my comments for the latest patch v6-0001.
>
> (I will post my v6-0002 review comments separately)
>
> PATCH v6-0001 comments
> ======================
>
> 1.1 General - Option name
>
> I still feel like the option name is not ideal. Unfortunately, this is
> important because any name change would impact lots of these patch
> files and docs, struct members etc.
>
> It was originally called "local_only", but I thought that as a
> SUBSCRIPTION option that was confusing because "local" means local to
> what? Really it is local to the publisher, not local to the
> subscriber, so that name seemed misleading.
>
> Then I suggested "publish_local_only". Although that resolved the
> ambiguity problem, other people thought it seemed odd to have the
> "publish" prefix for a subscription-side option.
>
> So now it is changed again to "subscribe_local_only" -- It's getting
> better but still, it is implied that the "local" means local to the
> publisher except there is nothing in the option name really to convey
> that meaning. IMO we here all understand the meaning of this option
> mostly by familiarity with this discussion thread, but I think a user
> coming to this for the first time will still be confused by the name.
>
> Below are some other name ideas. Maybe they are not improvements, but
> it might help other people to come up with something better.
>
> subscribe_publocal_only = true/false
> origin = pub_only/any
> adjacent_data_only = true/false
> direct_subscriptions_only = true/false
> ...

FWIW, The subscriber wants "changes originated on publisher". From
that angle origin = publisher/any looks attractive. It also leaves
open the possibility that the subscriber may ask changes from a set of
origins or even non-publisher origins.

-- 
Best Wishes,
Ashutosh Bapat

  55. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-04-06T04:06:52Z

    On Tue, Apr 5, 2022 at 7:06 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Below are some other name ideas. Maybe they are not improvements, but
> > it might help other people to come up with something better.
> >
> > subscribe_publocal_only = true/false
> > origin = pub_only/any
> > adjacent_data_only = true/false
> > direct_subscriptions_only = true/false
> > ...
>
> FWIW, The subscriber wants "changes originated on publisher". From
> that angle origin = publisher/any looks attractive. It also leaves
> open the possibility that the subscriber may ask changes from a set of
> origins or even non-publisher origins.
>

So, how are you imagining extending it for multiple origins? I think
we can have multiple origins data on a node when there are multiple
subscriptions pointing to the different or same node. The origin names
are internally generated names but are present in
pg_replication_origin. So, are we going to ask the user to check that
table and specify it as an option? But, note, that the user may not be
able to immediately recognize which origin data is useful for her.
Surely, with some help or advice in docs, she may be able to identify
it but that doesn't seem so straightforward. The other point is that
such an option won't be applicable for initial sync as we can't
differentiate between data from a local or remote node.

-- 
With Regards,
Amit Kapila.

  56. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-07T04:09:23Z

    FYI, here is a test script that is using the current patch (v6) to
demonstrate a way to share table data between different numbers of
nodes (up to 5 of them here).

The script starts off with just 2-way sharing (nodes N1, N2),
then expands to 3-way sharing (nodes N1, N2, N3),
then 4-way sharing (nodes N1, N2, N3, N4),
then 5-way sharing (nodes N1, N2, N3, N4, N5).

As an extra complication, for this test, all 5 nodes have different
initial table data, which gets replicated to the others whenever each
new node joins the existing share group.

PSA.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  57. RE: Handle infinite recursion in logical replication setup

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2022-04-07T06:03:34Z

    Dear Peter,

> FYI, here is a test script that is using the current patch (v6) to
> demonstrate a way to share table data between different numbers of
> nodes (up to 5 of them here).

Thanks for sharing your script! It's very helpful for us.

While reading your script, however, I had a question about it.
Line 121-122, you defined subscriptions for 2-nodes cluster:

psql -p $port_N1 -c "create subscription sub12 connection 'port=$port_N2' publication pub2 with ($copy_force);"
psql -p $port_N2 -c "create subscription sub21 connection 'port=$port_N1' publication pub1 with ($copy_force);"

But I was not sure it works well.
N2 already have shared data from N1 when subscription sub21 is created.
Did you assume that the initial copying is not so quick and
data synchronization will be not done when creating sub21?

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  58. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-07T06:13:56Z

    On Thu, Apr 7, 2022 at 4:03 PM kuroda.hayato@fujitsu.com
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Peter,
>
> > FYI, here is a test script that is using the current patch (v6) to
> > demonstrate a way to share table data between different numbers of
> > nodes (up to 5 of them here).
>
> Thanks for sharing your script! It's very helpful for us.
>
> While reading your script, however, I had a question about it.
> Line 121-122, you defined subscriptions for 2-nodes cluster:
>
> psql -p $port_N1 -c "create subscription sub12 connection 'port=$port_N2' publication pub2 with ($copy_force);"
> psql -p $port_N2 -c "create subscription sub21 connection 'port=$port_N1' publication pub1 with ($copy_force);"
>
> But I was not sure it works well.
> N2 already have shared data from N1 when subscription sub21 is created.
> Did you assume that the initial copying is not so quick and
> data synchronization will be not done when creating sub21?

Oops. Good catch.

Although the 2-way test was working OK for me, I think that it worked
only because of lucky timing. e.g. When I put a delay between those 2
subscriptions then the 2nd one would cause the PK violation that
probably you were anticipating would happen.

I have modified the 2-way example to use the same truncate pattern as others.

PSA the fixed test.sh script and accompanying files.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  59. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-07T23:07:45Z

    Hi Vignesh, FYI the patch is recently broken again and is failing on cfbot [1].

------
[1] http://cfbot.cputube.org/patch_38_3610.log

Kind Regards,
Peter Smith.
Fujitsu Australia

  60. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-08T01:11:50Z

    On Fri, Apr 8, 2022 at 4:38 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh, FYI the patch is recently broken again and is failing on cfbot [1].
>
> ------
> [1] http://cfbot.cputube.org/patch_38_3610.log

I'm working on fixing a few review comments, I will be posting an
updated version to handle this today.

Regards,
Vignesh

  61. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-08T07:30:04Z

    On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my comments for the latest patch v6-0001.
>
> (I will post my v6-0002 review comments separately)
>
> PATCH v6-0001 comments
> ======================
>
> 1.1 General - Option name
>
> I still feel like the option name is not ideal. Unfortunately, this is
> important because any name change would impact lots of these patch
> files and docs, struct members etc.
>
> It was originally called "local_only", but I thought that as a
> SUBSCRIPTION option that was confusing because "local" means local to
> what? Really it is local to the publisher, not local to the
> subscriber, so that name seemed misleading.
>
> Then I suggested "publish_local_only". Although that resolved the
> ambiguity problem, other people thought it seemed odd to have the
> "publish" prefix for a subscription-side option.
>
> So now it is changed again to "subscribe_local_only" -- It's getting
> better but still, it is implied that the "local" means local to the
> publisher except there is nothing in the option name really to convey
> that meaning. IMO we here all understand the meaning of this option
> mostly by familiarity with this discussion thread, but I think a user
> coming to this for the first time will still be confused by the name.
>
> Below are some other name ideas. Maybe they are not improvements, but
> it might help other people to come up with something better.
>
> subscribe_publocal_only = true/false
> origin = pub_only/any
> adjacent_data_only = true/false
> direct_subscriptions_only = true/false
> ...
>
>
> (FYI, the remainder of these review comments will assume the option is
> still called "subscribe_local_only")

Modified to local_only

> ~~~
>
> 1.2 General - inconsistent members and args
>
> IMO the struct members and args should also be named for close
> consistency with whatever the option name is.
>
> Currently the option is called "subscription_local_only".  So I think
> the members/args would be better to be called "local_only" instead of
> "only_local".

Modified

> ~~~
>
> 1.3 Commit message - wrong option name
>
> The commit message refers to the option name as "publish_local_only"
> instead of the option name that is currently implemented.

Modified

> ~~~
>
> 1.4 Commit message - wording
>
> The wording seems a bit off. Below is suggested simpler wording which
> I AFAIK conveys the same information.
>
> BEFORE
> Add an option publish_local_only which will subscribe only to the locally
> generated data in the publisher node. If subscriber is created with this
> option, publisher will skip publishing the data that was subscribed
> from other nodes. It can be created using following syntax:
> ex: CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999'
> PUBLICATION pub1 with (publish_local_only = on);
>
> SUGGESTION
> This patch adds a new SUBSCRIPTION boolean option
> "subscribe_local_only". The default is false. When a SUBSCRIPTION is
> created with this option enabled, the publisher will only publish data
> that originated at the publisher node.
> Usage:
> CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999'
> PUBLICATION pub1 with (subscribe_local_only = true);

Modified

> ~~~
>
> 1.5 doc/src/sgml/ref/create_subscription.sgml - "generated" changes.
>
> +         <para>
> +          Specifies whether the subscription will request the publisher to send
> +          locally generated changes or both the locally generated changes and
> +          the replicated changes that was generated from other nodes. The
> +          default is <literal>false</literal>.
> +         </para>
>
> For some reason, it seemed a bit strange to me to use the term
> "generated" changes. Maybe better to refer to the origin of changes?
>
> SUGGESTION
> Specifies whether the publisher should send only changes that
> originated locally at the publisher node, or send any publisher node
> changes regardless of their origin. The default is false.

Modified

> ~~~
>
> 1.6 src/backend/replication/pgoutput/pgoutput.c -
> LOGICALREP_PROTO_TWOPHASE_VERSION_NUM
>
> @@ -496,6 +509,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
> OutputPluginOptions *opt,
>   else
>   ctx->twophase_opt_given = true;
>
> + if (data->only_local && data->protocol_version <
> LOGICALREP_PROTO_TWOPHASE_VERSION_NUM)
> + ereport(ERROR,
> + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> + errmsg("requested proto_version=%d does not support
> subscribe_local_only, need %d or higher",
> + data->protocol_version, LOGICALREP_PROTO_TWOPHASE_VERSION_NUM)));
>
> I thought this code should not be using
> LOGICALREP_PROTO_TWOPHASE_VERSION_NUM. Shouldn't there be some newly
> introduced constant like LOGICALREP_PROTO_LOCALONLY_VERSION_NUM which
> you will use here?

Modified, I have set LOGICALREP_PROTO_LOCALONLY_VERSION_NUM to same
value as LOGICALREP_PROTO_TWOPHASE_VERSION_NUM, will increment this
once server version is changed.

> ~~~
>
> 1.7 src/bin/pg_dump/pg_dump.c - 150000
>
> @@ -4451,11 +4452,13 @@ getSubscriptions(Archive *fout)
>   if (fout->remoteVersion >= 150000)
>   appendPQExpBufferStr(query,
>   " s.subtwophasestate,\n"
> - " s.subdisableonerr\n");
> + " s.subdisableonerr,\n"
> + " s.sublocal\n");
>   else
>   appendPQExpBuffer(query,
>     " '%c' AS subtwophasestate,\n"
> -   " false AS subdisableonerr\n",
> +   " false AS subdisableonerr,\n"
> +   " false AS s.sublocal\n",
>     LOGICALREP_TWOPHASE_STATE_DISABLED);
>
> I think this local_only feature is unlikely to get into the PG15
> release, so this code should be split out into a separate condition
> because later will need to change to say >= 160000.

 I have split the condition and checked it with 150000, this will be
changed later to 160000 after the branch is created.

> ~~~
>
> 1.8 src/bin/pg_dump/pg_dump.c - dumpSubscription
>
> @@ -4585,6 +4591,9 @@ dumpSubscription(Archive *fout, const
> SubscriptionInfo *subinfo)
>   if (strcmp(subinfo->subdisableonerr, "t") == 0)
>   appendPQExpBufferStr(query, ", disable_on_error = true");
>
> + if (strcmp(subinfo->sublocal, "f") != 0)
> + appendPQExpBufferStr(query, ", subscribe_local_only = on");
> +
>
> I felt it is more natural to say "if it is true set to true", instead
> of "if it is not false set to on".
>
> SUGGESTION
> if (strcmp(subinfo->sublocal, "t") == 0)
> appendPQExpBufferStr(query, ", subscribe_local_only = true");

Modified

> ~~~
>
> 1.9 src/bin/psql/describe.c - 150000
>
> @@ -6318,9 +6318,11 @@ describeSubscriptions(const char *pattern, bool verbose)
>   if (pset.sversion >= 150000)
>   appendPQExpBuffer(&buf,
>     ", subtwophasestate AS \"%s\"\n"
> -   ", subdisableonerr AS \"%s\"\n",
> +   ", subdisableonerr AS \"%s\"\n"
> +   ", sublocal AS \"%s\"\n",
>     gettext_noop("Two phase commit"),
> -   gettext_noop("Disable on error"));
> +   gettext_noop("Disable on error"),
> +   gettext_noop("Only local"));
>
> I think this local_only feature is unlikely to get into the PG15
> release, so this code should be split out into a separate condition
> because later will need to change to say >= 160000.

I have split the condition and checked it with 150000, this will be
changed later to 160000 after the branch is created.

> ~~
>
> 1.10 src/bin/psql/describe.c - describeSubscriptions column
>
> @@ -6318,9 +6318,11 @@ describeSubscriptions(const char *pattern, bool verbose)
>   if (pset.sversion >= 150000)
>   appendPQExpBuffer(&buf,
>     ", subtwophasestate AS \"%s\"\n"
> -   ", subdisableonerr AS \"%s\"\n",
> +   ", subdisableonerr AS \"%s\"\n"
> +   ", sublocal AS \"%s\"\n",
>     gettext_noop("Two phase commit"),
> -   gettext_noop("Disable on error"));
> +   gettext_noop("Disable on error"),
> +   gettext_noop("Only local"));
>
> I think the column name here should be more consistent with the option
> name. e.g. it should be "Local only", not "Only local".

Modified

> ~~~
>
> 1.11 src/bin/psql/tab-complete.c - whitespace
>
> @@ -3167,7 +3167,7 @@ psql_completion(const char *text, int start, int end)
>   /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
>   else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
>   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> -   "enabled", "slot_name", "streaming",
> +   "enabled",  "slot_name", "streaming", "subscribe_local_only",
>
> The patch accidentally added a space char before the "slot_name".

Modified

> ~~~
>
> 1.12 src/include/replication/walreceiver.h - "generated"
>
> @@ -183,6 +183,7 @@ typedef struct
>   bool streaming; /* Streaming of large transactions */
>   bool twophase; /* Streaming of two-phase transactions at
>   * prepare time */
> + bool only_local; /* publish only locally generated data */
>
> This is a similar review comment as #1.5 about saying the word "generated".
> Maybe there is another way to word this?

Modified

> ~~~
>
> 1.13 src/test/regress/sql/subscription.sql - missing test case
>
> Isn't there a missing test case for ensuring that the new option is boolean?

Added test

Thanks for the comments, the attached v7 patch has the changes for the same.

Regards,
Vignesh

  62. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-08T07:40:05Z

    On Tue, Apr 5, 2022 at 8:17 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my comments for the latest patch v6-0002.
>
> PATCH v6-0002 comments
> ======================
>
> 2.1 General - should this be an independent patch?
>
> In many ways, I think most of this patch is unrelated to the other
> "local_only" patch (v6-0001).
>
> For example, IIUC even in the current HEAD, we could consider it to be
> a user error if multiple SUBSCRIPTIONS or multiple PUBLICATIONS of the
> same SUBSCRIPTION are replicating to the same TABLE on the same node
> and using "copy_data = on".
>
> So I think it would be ok to throw an ERROR if such a copy_data clash
> is detected, and then the user will have to change to use "copy_data =
> off" for some/all of them to avoid data duplication.
>
> The "local_only" option only adds some small logic to this new ERROR,
> but it's not really a prerequisite at all.
>
> e.g. this whole ERROR part of the patch can be a separate thread.

As Amit also pointed out, the patches have some dependency, I will
keep it as it is.

> ~~~
>
> 2.2 General - can we remove the "force" enum?
>
> Now, because I consider the clashing "copy_data = on" ERROR to be a
> user error, I think that is something that the user can already take
> care of themselves just using the copy_data = off.
>
> I did not really like the modifying of the "copy_data" option from
> just boolean to some kind of hybrid boolean + "force".
>
> a) It smells a bit off to me. IMO replication is supposed to end up
> with the same (replicated) data on the standby machine but this
> "force" mode seems to be just helping the user to break that concept
> and say - "I know what I'm doing, and I don't care if I get lots of
> duplicated data in the replica table - just let me do it"...
>
> b) It also somehow feels like the new "force" was introduced mostly to
> make the code ERROR handling implementation simpler, rather than to
> make the life of the end-user better. Yes, if force is removed maybe
> the copy-clash-detection-code will need to be internally quite more
> complex than it is now, but that is how it should be, instead of
> putting extra burden on the user (e.g. by complicating the PG docs and
> giving them yet more alternatives to configure). I think any clashing
> copy_data options really is a user error, but also I think the current
> boolean copy_data true/false already gives the user a way to fix it.
>
> c) Actually, your new error hint messages are similar to my
> perspective: They already say "Use CREATE/ALTER SUBSCRIPTION with
> copy_data = off or force". All I am saying is remove the "force", and
> the user can still fix the problem just by using "copy_data = off"
> appropriately.

When a user is trying to add a node to bidirectional replication
setup, the user will use the force option to copy the data from one of
the nodes and use off to skip copying the data from other nodes. This
option will be used while adding nodes to bidirectional replication,
the same has been documented with examples in the patch. I felt we
should retain this option.

> ======
>
> So (from above) I am not much in favour of the copy_data becoming a
> hybrid enum and using "force", yet that is what most of this patch is
> implementing. Anyway, the remainder of my review comments here are for
> the code in its current form. Maybe if "force" can be removed most of
> the following comments end up being redundant.
>
> ======
>
> 2.3 Commit message - wording
>
> This message is difficult to understand.
>
> I think that the long sentence "Now when user is trying..." can be
> broken into more manageable parts.

Slightly modified

> This part "and throw an error so that user can handle the initial copy
> data." also seemed a bit vague.

I have given the reference to the documentation section of the patch
for initial data copy handling

> ~~~
>
> 2.4 Commit message - more functions
>
> "This patch does couple of things:"
>
> IIUC, there seems a third thing implemented by this patch but not
> described by the comment. I think it also adds support for ALTER
> SUBSCRIPTION SET PUBLICATION WITH (subscribe_local_only)

This is part of 0001 patch, I felt this should not be part of 002 patch commit.

> ~~~
>
> 2.5 doc/src/sgml/ref/create_subscription.sgml - wording
>
> @@ -161,6 +161,13 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>            the replicated changes that was generated from other nodes. The
>            default is <literal>false</literal>.
>           </para>
> +         <para>
> +          If the tables in the publication were also subscribing to the data in
> +          the publisher from other publishers, it will affect the
> +          <command>CREATE SUBSCRIPTION</command> based on the value specified
> +          for <literal>copy_data</literal> option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>
> Is there is a simpler way to express all that?
>
> SUGGESTION
> There is some interation between the option "subscribe_local_only" and
> option "copy_data". Refer to the <xref
> linkend="sql-createsubscription-notes" /> for details.

Modified

> ~~~
>
> 2.6 doc/src/sgml/ref/create_subscription.sgml - whitespace
>
> @@ -213,18 +220,28 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>         </varlistentry>
>
>         <varlistentry>
> -        <term><literal>copy_data</literal> (<type>boolean</type>)</term>
> +        <term><literal>copy_data</literal> (<type>enum</type>)</term>
>          <listitem>
>           <para>
>            Specifies whether to copy pre-existing data in the publications
> -          that are being subscribed to when the replication starts.
> -          The default is <literal>true</literal>.
> +          that are being subscribed to when the replication starts. This
> +          parameter may be either <literal>true</literal>,
> +          <literal>false</literal> or <literal>force</literal>. The default is
> +          <literal>true</literal>.
>
> That last line has trailing whitespace.

Modified

> ~~~
>
> 2.7 doc/src/sgml/ref/create_subscription.sgml - wording
>
> +         <para>
> +          If the tables in the publication were also subscribing to the data in
> +          the publisher from other publishers, it will affect the
> +          <command>CREATE SUBSCRIPTION</command> based on the value specified
> +          for <literal>subscribe_local_only</literal> option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>
> This is similar to review comment #2.5 which I thought could be
> written in a simpler way.

Modified

> ~~~
>
> 2.8 doc/src/sgml/ref/create_subscription.sgml
>
> @@ -375,6 +392,42 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   Let's consider an existing Multi master logical replication setup between
> +   Node1 and Node2 that is created using the following steps:
> +   a) Node1 - Publication publishing employee table.
> +   b) Node2 - Subscription subscribing from publication pub1 with
> +   <literal>subscribe_local_only</literal>.
> +   c) Node2 - Publication publishing employee table.
> +   d) Node1 - Subscription subscribing from publication pub2 with
> +   <literal>subscribe_local_only</literal>.
> +   Now when user is trying to add another node Node3 to the above Multi master
> +   logical replication setup, user will have to create one subscription
> +   subscribing from Node1 and another subscription subscribing from Node2 in
> +   Node3 using <literal>subscribe_local_only</literal> option and
> +   <literal>copy_data</literal> as <literal>true</literal>, while
> +   the subscription is created, server will identify that Node2 is subscribing
> +   from Node1 and Node1 is subscribing from Node2 and throw an error like:
> +   <programlisting>
> +postgres=# CREATE SUBSCRIPTION mysub CONNECTION 'host=192.168.1.50
> port=5432 user=foo dbname=foodb'
> +           PUBLICATION mypublication, insert_only with
> (subscribe_local_only=on);
> +ERROR:  CREATE/ALTER SUBSCRIPTION with subscribe_local_only and
> copy_data as true is not allowed when the publisher might have
> replicated data, table:public.t1 might have replicated data in the
> publisher
> +HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force
> +   </programlisting>
> +   In this scenario user can solve this based on one of the 2 possibilities,
> +   a) If there are no data present in Node1 and Node2, then the user can create
> +   the subscriptions to Node1 and Node2 with
> +   <literal>subscribe_local_only</literal> as <literal>true</literal> and
> +   <literal>copy_data</literal> as <literal>false</literal>. b) If the data is
> +   present, then the user can create subscription with
> +   <literal>copy_data</literal> as <literal>force</literal> on Node1 and
> +   <literal>copy_data</literal> as <literal>false</literal> on Node2, before
> +   allowing any operations on the respective tables of Node1 and Node2, in this
> +   case <literal>copy_data</literal> is <literal>false</literal> on Node2
> +   because the data will be replicated to each other and available on both the
> +   nodes.
> +  </para>
> +
>
> That is a large slab of text in the Notes, so not very easy to digest it.
>
> I'm not sure what to suggest for this -
> - Perhaps the a,b,c,d should all be "lists" so it renders differently?
> - It almost seems like too much information to be included in this
> "Notes" section, Maybe it needs its own full page in PG Docs "Logical
> Replication" to discuss this topic.

This is moved to the new page "BiDirectional logical replication Quick
Setup" page now.

> ~~~
>
> 2.9 src/backend/commands/subscriptioncmds.c - IS_COPY_DATA_VALID
>
> @@ -69,6 +69,18 @@
>  /* check if the 'val' has 'bits' set */
>  #define IsSet(val, bits)  (((val) & (bits)) == (bits))
>
> +#define IS_COPY_DATA_VALID(copy_data) (copy_data != COPY_DATA_OFF)
>
> The macro seems misnamed because "off" is also "valid". It seems like
> it should be called something different like IS_COPY_DATA, or
> IS_COPY_DATA_ON_OR_FORCE, etc.

Modified

> ~~~
>
> 2.10 src/backend/commands/subscriptioncmds.c - DefGetCopyData
>
> + /*
> + * The set of strings accepted here should match up with
> + * the grammar's opt_boolean_or_string production.
> + */
> + if (pg_strcasecmp(sval, "true") == 0)
> + return COPY_DATA_ON;
> + if (pg_strcasecmp(sval, "false") == 0)
> + return COPY_DATA_OFF;
> + if (pg_strcasecmp(sval, "on") == 0)
> + return COPY_DATA_ON;
> + if (pg_strcasecmp(sval, "off") == 0)
> + return COPY_DATA_OFF;
> + if (pg_strcasecmp(sval, "force") == 0)
> + return COPY_DATA_FORCE;
>
> Maybe combine the return for true/on and false/off?

Modified

> ~~~
>
> 2.11 src/backend/commands/subscriptioncmds.c - fetch_table_list
>
> + appendStringInfoString(&cmd,
> +    "SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename,
> PS.srsubstate as replicated\n"
> +    "FROM pg_publication P,\n"
> +    "LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> +    "LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> +    "pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> +    "WHERE C.oid = GPT.relid AND P.pubname in (");
> +
>
> That blank line is not needed (it was not there previously) because
> the next get_publications_str is a continuation of this SQL.

Modified

> ~~~
>
> 2.12 src/backend/commands/subscriptioncmds.c - fetch_table_list comment
>
> + * It is quite possible that subscriber has not yet pulled data to
> + * the tables, but in ideal cases the table data will be subscribed.
> + * Too keep the code simple it is not checked if the subscriber table
> + * has pulled the data or not.
> + */
>
> Typo "Too" -> "To"

Modified

> ~~~
>
> 2.13 src/backend/commands/subscriptioncmds.c - force
>
> + if (copydata == COPY_DATA_ON && only_local && !slot_attisnull(slot, 3))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("CREATE/ALTER SUBSCRIPTION with subscribe_local_only and
> copy_data as true is not allowed when the publisher might have
> replicated data, table:%s.%s might have replicated data in the
> publisher",
> +    nspname, relname),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
>
> AFAIK this is the only code of this patch that is distinguishing
> between "force" and the "on". As I wrote at the beginning of this post
> I feel you can keep this ERROR (maybe the way to detect it needs to be
> more complex) and the user can fix it just by using "copy_data = off"
> appropriately. So then copy_data does not need to be changed.

When a user is trying to add a node to bidirectional replication
setup, the user will use the force option to copy the data from one of
the nodes and use off to skip copying the data from other nodes. This
option will be used while adding nodes to bidirectional replication,
the same has been documented with examples in the patch. I felt we
should retain this option.

> ~~~
>
> 2.14 src/test/regress/sql/subscription.sql - missing tests
>
> The new copy_data is not really an enum true/false/force like the PG
> docs claims.
>
> It seems more like some kind of hybrid boolean+force. So if that is
> how it is going to be then there are missing test cases to make sure
> that values like "on"/"off"/"0"/"1" still are working.

Added tests

> ~~~
>
> 2.15 src/test/subscription/t/032_circular.pl
>
> @@ -65,6 +69,25 @@ $node_A->safe_psql('postgres', "
>   PUBLICATION tap_pub_B
>   WITH (subscribe_local_only = on, copy_data = off)");
>
> +($result, $stdout, $stderr) = $node_A->psql('postgres',  "
> +        CREATE SUBSCRIPTION tap_sub_A1
> +        CONNECTION '$node_B_connstr application_name=$appname_A'
> +        PUBLICATION tap_pub_B
> +        WITH (subscribe_local_only = on, copy_data = on)");
> +like(
> +        $stderr,
> +        qr/ERROR:  CREATE\/ALTER SUBSCRIPTION with
> subscribe_local_only and copy_data as true is not allowed when the
> publisher might have replicated data/,
> +        "Create subscription with subscribe_local_only and copy_data
> having replicated table in publisher");
> +
> +$node_A->safe_psql('postgres',  "
> +        CREATE SUBSCRIPTION tap_sub_A2
> +        CONNECTION '$node_B_connstr application_name=$appname_A'
> +        PUBLICATION tap_pub_B
> +        WITH (subscribe_local_only = on, copy_data = force)");
> +
> +$node_A->safe_psql('postgres',  "
> +        DROP SUBSCRIPTION tap_sub_A2");
> +
>
> Maybe underneath it is the same, but from the outside, this looks like
> a slightly different scenario from what is mentioned everywhere else
> in the patch.
>
> I think it would be better to create a new Node_C (aka Node3) so then
> the TAP test can use the same example that you give in the commit
> message and the PG docs notes.

Modified the tests to include Node_C.

Thanks for the comments, the v7 patch attached at [1] has the fixes for the same
[1] - https://www.postgresql.org/message-id/CALDaNm1ei%3DrRwCBKWtUu8b5OsS6FFcvaxg9h0oXcjgFn8GoZnQ%40mail.gmail.com

Regards,
Vignesh

  63. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-12T04:56:12Z

    On Thu, Apr 7, 2022 at 2:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> FYI, here is a test script that is using the current patch (v6) to
> demonstrate a way to share table data between different numbers of
> nodes (up to 5 of them here).
>
> The script starts off with just 2-way sharing (nodes N1, N2),
> then expands to 3-way sharing (nodes N1, N2, N3),
> then 4-way sharing (nodes N1, N2, N3, N4),
> then 5-way sharing (nodes N1, N2, N3, N4, N5).
>
> As an extra complication, for this test, all 5 nodes have different
> initial table data, which gets replicated to the others whenever each
> new node joins the existing share group.
>
> PSA.
>


Hi Vignesh. I had some problems getting the above test script working.
It was OK up until I tried to join the 5th node (N5) to the existing 4
nodes. The ERROR was manifesting itself strangely because it appeared
that there was an index violation in the pg_subscription_rel catalog
even though AFAIK the N5 did not have any entries in it.


e.g.
2022-04-07 09:13:28.361 AEST [24237] ERROR: duplicate key value
violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
2022-04-07 09:13:28.361 AEST [24237] DETAIL: Key (srrelid,
srsubid)=(16384, 16393) already exists.
2022-04-07 09:13:28.361 AEST [24237] STATEMENT: create subscription
sub51 connection 'port=7651' publication pub1 with
(subscribe_local_only=true,copy_data=force);
2022-04-07 09:13:28.380 AEST [24237] ERROR: duplicate key value
violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
2022-04-07 09:13:28.380 AEST [24237] DETAIL: Key (srrelid,
srsubid)=(16384, 16394) already exists.
2022-04-07 09:13:28.380 AEST [24237] STATEMENT: create subscription
sub52 connection 'port=7652' publication pub2 with
(subscribe_local_only=true,copy_data=false);
2022-04-07 09:13:28.405 AEST [24237] ERROR: duplicate key value
violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
2022-04-07 09:13:28.405 AEST [24237] DETAIL: Key (srrelid,
srsubid)=(16384, 16395) already exists.
2022-04-07 09:13:28.405 AEST [24237] STATEMENT: create subscription
sub53 connection 'port=7653' publication pub3 with
(subscribe_local_only=true,copy_data=false);
2022-04-07 09:13:28.425 AEST [24237] ERROR: duplicate key value
violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
2022-04-07 09:13:28.425 AEST [24237] DETAIL: Key (srrelid,
srsubid)=(16384, 16396) already exists.
2022-04-07 09:13:28.425 AEST [24237] STATEMENT: create subscription
sub54 connection 'port=7654' publication pub4 with
(subscribe_local_only=true,copy_data=false);
2022-04-07 09:17:52.472 AEST [25852] ERROR: duplicate key value
violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
2022-04-07 09:17:52.472 AEST [25852] DETAIL: Key (srrelid,
srsubid)=(16384, 16397) already exists.
2022-04-07 09:17:52.472 AEST [25852] STATEMENT: create subscription
sub51 connection 'port=7651' publication pub1;

~~~

When I debugged this it seemed like each of the CREAT SUBSCRIPTION was
trying to make a double-entry, because the fetch_tables (your patch
v6-0002 modified SQL of this) was retuning the same table 2x.

(gdb) bt
#0 errfinish (filename=0xbc1057 "nbtinsert.c", lineno=671,
funcname=0xbc25e0 <__func__.15798> "_bt_check_unique") at elog.c:510
#1 0x0000000000526d83 in _bt_check_unique (rel=0x7f654219c2a0,
insertstate=0x7ffd9629ddd0, heapRel=0x7f65421b0e28,
checkUnique=UNIQUE_CHECK_YES, is_unique=0x7ffd9629de01,
speculativeToken=0x7ffd9629ddcc) at nbtinsert.c:664
#2 0x0000000000526157 in _bt_doinsert (rel=0x7f654219c2a0,
itup=0x19ea8e8, checkUnique=UNIQUE_CHECK_YES, indexUnchanged=false,
heapRel=0x7f65421b0e28) at nbtinsert.c:208
#3 0x000000000053450e in btinsert (rel=0x7f654219c2a0,
values=0x7ffd9629df10, isnull=0x7ffd9629def0, ht_ctid=0x19ea894,
heapRel=0x7f65421b0e28, checkUnique=UNIQUE_CHECK_YES,
indexUnchanged=false, indexInfo=0x19dea80) at nbtree.c:201
#4 0x00000000005213b6 in index_insert (indexRelation=0x7f654219c2a0,
values=0x7ffd9629df10, isnull=0x7ffd9629def0, heap_t_ctid=0x19ea894,
heapRelation=0x7f65421b0e28,  checkUnique=UNIQUE_CHECK_YES,
indexUnchanged=false, indexInfo=0x19dea80) at indexam.c:193
#5 0x00000000005c81d5 in CatalogIndexInsert (indstate=0x19de540,
heapTuple=0x19ea890) at indexing.c:158
#6 0x00000000005c8325 in CatalogTupleInsert (heapRel=0x7f65421b0e28,
tup=0x19ea890) at indexing.c:231
#7 0x00000000005f0170 in AddSubscriptionRelState (subid=16400,
relid=16384, state=105 'i', sublsn=0) at pg_subscription.c:315
#8 0x00000000006d6fa5 in CreateSubscription (pstate=0x1942dc0,
stmt=0x191f6a0, isTopLevel=true) at subscriptioncmds.c:767

~~

Aside: All this was happening when I did not have enough logical
replication workers configured. (There were WARNINGS in the logfile
that I had not noticed).
When I fix the configuration then all these other problems went away!

~~

So to summarize, I'm not sure if the fetch_tables still has some
potential problem lurking or not, but I feel that the SQL in that
function maybe needs a closer look to ensure it is always impossible
to return the same table multiple times.

------
Kind Regards,
Peter Smith.
Fujitsu Australia.

  64. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-12T06:52:22Z

    On Tue, Apr 12, 2022 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Thu, Apr 7, 2022 at 2:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > FYI, here is a test script that is using the current patch (v6) to
> > demonstrate a way to share table data between different numbers of
> > nodes (up to 5 of them here).
> >
> > The script starts off with just 2-way sharing (nodes N1, N2),
> > then expands to 3-way sharing (nodes N1, N2, N3),
> > then 4-way sharing (nodes N1, N2, N3, N4),
> > then 5-way sharing (nodes N1, N2, N3, N4, N5).
> >
> > As an extra complication, for this test, all 5 nodes have different
> > initial table data, which gets replicated to the others whenever each
> > new node joins the existing share group.
> >
> > PSA.
> >
>
>
> Hi Vignesh. I had some problems getting the above test script working.
> It was OK up until I tried to join the 5th node (N5) to the existing 4
> nodes. The ERROR was manifesting itself strangely because it appeared
> that there was an index violation in the pg_subscription_rel catalog
> even though AFAIK the N5 did not have any entries in it.
>
>
> e.g.
> 2022-04-07 09:13:28.361 AEST [24237] ERROR: duplicate key value
> violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
> 2022-04-07 09:13:28.361 AEST [24237] DETAIL: Key (srrelid,
> srsubid)=(16384, 16393) already exists.
> 2022-04-07 09:13:28.361 AEST [24237] STATEMENT: create subscription
> sub51 connection 'port=7651' publication pub1 with
> (subscribe_local_only=true,copy_data=force);
> 2022-04-07 09:13:28.380 AEST [24237] ERROR: duplicate key value
> violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
> 2022-04-07 09:13:28.380 AEST [24237] DETAIL: Key (srrelid,
> srsubid)=(16384, 16394) already exists.
> 2022-04-07 09:13:28.380 AEST [24237] STATEMENT: create subscription
> sub52 connection 'port=7652' publication pub2 with
> (subscribe_local_only=true,copy_data=false);
> 2022-04-07 09:13:28.405 AEST [24237] ERROR: duplicate key value
> violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
> 2022-04-07 09:13:28.405 AEST [24237] DETAIL: Key (srrelid,
> srsubid)=(16384, 16395) already exists.
> 2022-04-07 09:13:28.405 AEST [24237] STATEMENT: create subscription
> sub53 connection 'port=7653' publication pub3 with
> (subscribe_local_only=true,copy_data=false);
> 2022-04-07 09:13:28.425 AEST [24237] ERROR: duplicate key value
> violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
> 2022-04-07 09:13:28.425 AEST [24237] DETAIL: Key (srrelid,
> srsubid)=(16384, 16396) already exists.
> 2022-04-07 09:13:28.425 AEST [24237] STATEMENT: create subscription
> sub54 connection 'port=7654' publication pub4 with
> (subscribe_local_only=true,copy_data=false);
> 2022-04-07 09:17:52.472 AEST [25852] ERROR: duplicate key value
> violates unique constraint "pg_subscription_rel_srrelid_srsubid_index"
> 2022-04-07 09:17:52.472 AEST [25852] DETAIL: Key (srrelid,
> srsubid)=(16384, 16397) already exists.
> 2022-04-07 09:17:52.472 AEST [25852] STATEMENT: create subscription
> sub51 connection 'port=7651' publication pub1;
>
> ~~~
>
> When I debugged this it seemed like each of the CREAT SUBSCRIPTION was
> trying to make a double-entry, because the fetch_tables (your patch
> v6-0002 modified SQL of this) was retuning the same table 2x.
>
> (gdb) bt
> #0 errfinish (filename=0xbc1057 "nbtinsert.c", lineno=671,
> funcname=0xbc25e0 <__func__.15798> "_bt_check_unique") at elog.c:510
> #1 0x0000000000526d83 in _bt_check_unique (rel=0x7f654219c2a0,
> insertstate=0x7ffd9629ddd0, heapRel=0x7f65421b0e28,
> checkUnique=UNIQUE_CHECK_YES, is_unique=0x7ffd9629de01,
> speculativeToken=0x7ffd9629ddcc) at nbtinsert.c:664
> #2 0x0000000000526157 in _bt_doinsert (rel=0x7f654219c2a0,
> itup=0x19ea8e8, checkUnique=UNIQUE_CHECK_YES, indexUnchanged=false,
> heapRel=0x7f65421b0e28) at nbtinsert.c:208
> #3 0x000000000053450e in btinsert (rel=0x7f654219c2a0,
> values=0x7ffd9629df10, isnull=0x7ffd9629def0, ht_ctid=0x19ea894,
> heapRel=0x7f65421b0e28, checkUnique=UNIQUE_CHECK_YES,
> indexUnchanged=false, indexInfo=0x19dea80) at nbtree.c:201
> #4 0x00000000005213b6 in index_insert (indexRelation=0x7f654219c2a0,
> values=0x7ffd9629df10, isnull=0x7ffd9629def0, heap_t_ctid=0x19ea894,
> heapRelation=0x7f65421b0e28,  checkUnique=UNIQUE_CHECK_YES,
> indexUnchanged=false, indexInfo=0x19dea80) at indexam.c:193
> #5 0x00000000005c81d5 in CatalogIndexInsert (indstate=0x19de540,
> heapTuple=0x19ea890) at indexing.c:158
> #6 0x00000000005c8325 in CatalogTupleInsert (heapRel=0x7f65421b0e28,
> tup=0x19ea890) at indexing.c:231
> #7 0x00000000005f0170 in AddSubscriptionRelState (subid=16400,
> relid=16384, state=105 'i', sublsn=0) at pg_subscription.c:315
> #8 0x00000000006d6fa5 in CreateSubscription (pstate=0x1942dc0,
> stmt=0x191f6a0, isTopLevel=true) at subscriptioncmds.c:767
>
> ~~
>
> Aside: All this was happening when I did not have enough logical
> replication workers configured. (There were WARNINGS in the logfile
> that I had not noticed).
> When I fix the configuration then all these other problems went away!
>
> ~~
>
> So to summarize, I'm not sure if the fetch_tables still has some
> potential problem lurking or not, but I feel that the SQL in that
> function maybe needs a closer look to ensure it is always impossible
> to return the same table multiple times.

I have earlier used a distinct of srsubstate, but there is a
possibility that when multiple subscriptions are being created there
can be multiple entries because the srsubstate can be different like i
and r. I have changed srsubstate to srrelid to get the unique values.
The attached v8 patch has the changes for the same.

Regards,
Vignesh

  65. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-14T08:22:59Z

    Here are my review comments for v8-0001.

======

1. Commit message

CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999'
PUBLICATION pub1 with (local_only = true);

The spaces in the CONNECTION string are a bit strange.

~~~

2. src/backend/catalog/pg_subscription.

@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
  sub->stream = subform->substream;
  sub->twophasestate = subform->subtwophasestate;
  sub->disableonerr = subform->subdisableonerr;
+ sub->local_only = subform->sublocal;

Maybe call it subform->sublocalonly;

~~~

3. src/backend/catalog/system_views.sql

@@ -1290,8 +1290,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
 -- All columns of pg_subscription except subconninfo are publicly readable.
 REVOKE ALL ON pg_subscription FROM public;
 GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
-              subbinary, substream, subtwophasestate,
subdisableonerr, subslotname,
-              subsynccommit, subpublications)
+              subbinary, substream, subtwophasestate, subdisableonerr,
+              sublocal, subslotname, subsynccommit, subpublications)

Maybe call the column sublocalonly

~~~

4. .../libpqwalreceiver/libpqwalreceiver.c

@@ -453,6 +453,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
  PQserverVersion(conn->streamConn) >= 150000)
  appendStringInfoString(&cmd, ", two_phase 'on'");

+ if (options->proto.logical.local_only &&
+ PQserverVersion(conn->streamConn) >= 150000)
+ appendStringInfoString(&cmd, ", local_only 'on'");

Add a FIXME comment here as a reminder that this condition needs to
change for PG16.

~~~

5. src/bin/pg_dump/pg_dump.c

@@ -4361,6 +4361,7 @@ getSubscriptions(Archive *fout)
  int i_subsynccommit;
  int i_subpublications;
  int i_subbinary;
+ int i_sublocal;
  int i,
  ntups;

Maybe call this member i_sublocalonly;

~~~

6. src/bin/pg_dump/pg_dump.c
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, " s.sublocal\n");
+ else
+ appendPQExpBufferStr(query, " false AS sublocal\n");
+

6a. Add a FIXME comment as a reminder that this condition needs to
change for PG16.

6b. Change the column name to sublocalonly.

~~~

7. src/bin/pg_dump/pg_dump.h

@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
  char    *subdisableonerr;
  char    *subsynccommit;
  char    *subpublications;
+ char    *sublocal;
 } SubscriptionInfo;

Change the member name to sublocalonly

~~~

8. src/bin/psql/describe.c

@@ -6241,6 +6241,12 @@ describeSubscriptions(const char *pattern, bool verbose)
    gettext_noop("Two phase commit"),
    gettext_noop("Disable on error"));

+ /* local_only is supported only in v15 and higher */
+ if (pset.sversion >= 150000)
+ appendPQExpBuffer(&buf,
+   ", sublocal AS \"%s\"\n",
+   gettext_noop("Local only"));

7a. The comment is wrong to mention v15.

7b. Add a FIXME comment as a reminder that this condition needs to
change for PG16.

7c. Change the column name to sublocalonly.

~~~

9. src/include/catalog/pg_subscription.h

@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW

  bool substream; /* Stream in-progress transactions. */

+ bool sublocal; /* skip copying of remote origin data */

Change the member name to sublocalonly

~~~

10. src/include/replication/logicalproto.h

@@ -32,12 +32,17 @@
  *
  * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with
  * support for two-phase commit decoding (at prepare time). Introduced in PG15.
+ *
+ * LOGICALREP_PROTO_LOCALONLY_VERSION_NUM is the minimum protocol version with
+ * support for sending only locally originated data from the publisher.
+ * Introduced in PG16.

Add a FIXME comment here as a reminder that the proto version number
needs to be bumped to 4 in PG16.

~~~

11. src/test/subscription/t/032_circular.pl

Perhaps there should be another test using a third "Node_C" which
publishes some data to Node_B. Then you can ensure that by using
local_only (when Node_A is subscribing to Node_A) that nothing from
the Node_C can find its way onto Node_A. But then the test name
"circular" is a bit misleading. Maybe it should be "032_localonly"?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  66. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-16T14:18:22Z

    On Thu, Apr 14, 2022 at 1:53 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v8-0001.
>
> ======
>
> 1. Commit message
>
> CREATE SUBSCRIPTION sub1 CONNECTION 'dbname =postgres port=9999'
> PUBLICATION pub1 with (local_only = true);
>
> The spaces in the CONNECTION string are a bit strange.

Modified

> ~~~
>
> 2. src/backend/catalog/pg_subscription.
>
> @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
>   sub->stream = subform->substream;
>   sub->twophasestate = subform->subtwophasestate;
>   sub->disableonerr = subform->subdisableonerr;
> + sub->local_only = subform->sublocal;
>
> Maybe call it subform->sublocalonly;
>

Modified

> ~~~
>
> 3. src/backend/catalog/system_views.sql
>
> @@ -1290,8 +1290,8 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
>  -- All columns of pg_subscription except subconninfo are publicly readable.
>  REVOKE ALL ON pg_subscription FROM public;
>  GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
> -              subbinary, substream, subtwophasestate,
> subdisableonerr, subslotname,
> -              subsynccommit, subpublications)
> +              subbinary, substream, subtwophasestate, subdisableonerr,
> +              sublocal, subslotname, subsynccommit, subpublications)
>
> Maybe call the column sublocalonly
>

Modified

> ~~~
>
> 4. .../libpqwalreceiver/libpqwalreceiver.c
>
> @@ -453,6 +453,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
>   PQserverVersion(conn->streamConn) >= 150000)
>   appendStringInfoString(&cmd, ", two_phase 'on'");
>
> + if (options->proto.logical.local_only &&
> + PQserverVersion(conn->streamConn) >= 150000)
> + appendStringInfoString(&cmd, ", local_only 'on'");
>
> Add a FIXME comment here as a reminder that this condition needs to
> change for PG16.
>

Modified

> ~~~
>
> 5. src/bin/pg_dump/pg_dump.c
>
> @@ -4361,6 +4361,7 @@ getSubscriptions(Archive *fout)
>   int i_subsynccommit;
>   int i_subpublications;
>   int i_subbinary;
> + int i_sublocal;
>   int i,
>   ntups;
>
> Maybe call this member i_sublocalonly;
>

Modified

> ~~~
>
> 6. src/bin/pg_dump/pg_dump.c
> + if (fout->remoteVersion >= 150000)
> + appendPQExpBufferStr(query, " s.sublocal\n");
> + else
> + appendPQExpBufferStr(query, " false AS sublocal\n");
> +
>
> 6a. Add a FIXME comment as a reminder that this condition needs to
> change for PG16.
>

Modified

> 6b. Change the column name to sublocalonly.
>

Modified

> ~~~
>
> 7. src/bin/pg_dump/pg_dump.h
>
> @@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
>   char    *subdisableonerr;
>   char    *subsynccommit;
>   char    *subpublications;
> + char    *sublocal;
>  } SubscriptionInfo;
>
> Change the member name to sublocalonly
>

Modified

> ~~~
>
> 8. src/bin/psql/describe.c
>
> @@ -6241,6 +6241,12 @@ describeSubscriptions(const char *pattern, bool verbose)
>     gettext_noop("Two phase commit"),
>     gettext_noop("Disable on error"));
>
> + /* local_only is supported only in v15 and higher */
> + if (pset.sversion >= 150000)
> + appendPQExpBuffer(&buf,
> +   ", sublocal AS \"%s\"\n",
> +   gettext_noop("Local only"));
>
> 7a. The comment is wrong to mention v15.
>

I have removed this comment and added a FIXME. I will add it in once
version change is done to avoid confusion.

> 7b. Add a FIXME comment as a reminder that this condition needs to
> change for PG16.
>

Modified

> 7c. Change the column name to sublocalonly.
>

Modified

> ~~~
>
> 9. src/include/catalog/pg_subscription.h
>
> @@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>
>   bool substream; /* Stream in-progress transactions. */
>
> + bool sublocal; /* skip copying of remote origin data */
>
> Change the member name to sublocalonly
>

Modified

> ~~~
>
> 10. src/include/replication/logicalproto.h
>
> @@ -32,12 +32,17 @@
>   *
>   * LOGICALREP_PROTO_TWOPHASE_VERSION_NUM is the minimum protocol version with
>   * support for two-phase commit decoding (at prepare time). Introduced in PG15.
> + *
> + * LOGICALREP_PROTO_LOCALONLY_VERSION_NUM is the minimum protocol version with
> + * support for sending only locally originated data from the publisher.
> + * Introduced in PG16.
>
> Add a FIXME comment here as a reminder that the proto version number
> needs to be bumped to 4 in PG16.
>

Modified

> ~~~
>
> 11. src/test/subscription/t/032_circular.pl
>
> Perhaps there should be another test using a third "Node_C" which
> publishes some data to Node_B. Then you can ensure that by using
> local_only (when Node_A is subscribing to Node_A) that nothing from
> the Node_C can find its way onto Node_A. But then the test name
> "circular" is a bit misleading. Maybe it should be "032_localonly"?

Added the test and changed the file.

Thanks for the comments, attached v9 patch has the changes for the same.

Regards,
Vignesh

  67. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-19T02:59:16Z

    I checked the latest v9-0001 patch. Below are my review comments.

Other than these few trivial comments this 0001 patch looks good to me.

~~~

1. src/backend/replication/pgoutput/pgoutput.c - whitespace

@@ -1696,6 +1714,10 @@ static bool
 pgoutput_origin_filter(LogicalDecodingContext *ctx,
     RepOriginId origin_id)
 {
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+
+ if (data->local_only && origin_id != InvalidRepOriginId)
+ return true;
  return false;
 }

Suggest to add a blank line after the return true;

~~~

2. src/bin/psql/tab-complete.c - not alphabetical

@@ -1874,7 +1874,7 @@ psql_completion(const char *text, int start, int end)
  COMPLETE_WITH("(", "PUBLICATION");
  /* ALTER SUBSCRIPTION <name> SET ( */
  else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming",
"synchronous_commit", "disable_on_error");
+ COMPLETE_WITH("binary", "slot_name", "streaming", "local_only",
"synchronous_commit", "disable_on_error");

2a. AFAIK the code intended that these options be listed in
alphabetical order (I think the recent addition of disable_on_error is
also wrong here). So "local_only" should be moved.

@@ -3156,7 +3156,7 @@ psql_completion(const char *text, int start, int end)
  /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
  else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
  COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
-   "enabled", "slot_name", "streaming",
+   "enabled", "slot_name", "streaming", "local_only",
    "synchronous_commit", "two_phase", "disable_on_error");

2b. ditto

~~~

3. src/test/subscription/t/032_localonly.pl - wrong message

+$node_C->wait_for_catchup($appname_B2);
+$node_B->wait_for_catchup($appname_A);
+
+$result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full;");
+is( $result, qq(11
+12
+13),
+ 'Inserted successfully without leading to infinite recursion in
circular replication setup'
+);
+
+# check that the data published from node_C to node_B is not sent to node_A
+$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full;");
+is( $result, qq(11
+12),
+ 'Inserted successfully without leading to infinite recursion in
circular replication setup'
+);
+

The new test looked good, but the cut/paste text message ('Inserted
successfully without leading to infinite recursion in circular
replication setup') maybe needs changing because there is nothing
really "circular" about this test case.

------
Kind Regards,
Peter Smith.
Fujitsu Australia.

  68. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-20T01:56:08Z

    Below are my review comments for the v9-0002 patch (except I did not
yet look at the TAP tests).

~~~

1. General comment - describe.c

I wondered why the copy_data enum value is not displayed by the psql
\drs+ command. Should it be?

~~~

2. General comment - SUBOPT_LOCAL_ONLY

@@ -1134,7 +1204,7 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,

  case ALTER_SUBSCRIPTION_SET_PUBLICATION:
  {
- supported_opts = SUBOPT_COPY_DATA | SUBOPT_REFRESH;
+ supported_opts = SUBOPT_COPY_DATA | SUBOPT_REFRESH | SUBOPT_LOCAL_ONLY;
  parse_subscription_options(pstate, stmt->options,
     supported_opts, &opts);

@@ -1236,7 +1308,8 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
  errmsg("ALTER SUBSCRIPTION ... REFRESH is not allowed for disabled
subscriptions")));

  parse_subscription_options(pstate, stmt->options,
-    SUBOPT_COPY_DATA, &opts);
+    SUBOPT_COPY_DATA | SUBOPT_LOCAL_ONLY,
+    &opts);

I noticed there is some new code that appears to be setting the
SUBOT_LOCAL_ONLY as a supported option. Shouldn't those changes belong
in the patch 0001 instead of in this patch 0002?

~~~

3. Commit message - wording

CURRENT
a) Node1 - Publication publishing employee table.
b) Node2 - Subscription subscribing from publication pub1 with
local_only.
c) Node2 - Publication publishing employee table.
d) Node1 - Subscription subscribing from publication pub2 with
local_only.

SUGGESTION (I think below has the same meaning but is simpler)
a) Node1 - PUBLICATION pub1 for the employee table
b) Node2 - SUBSCRIPTION from pub1 with local_only=true
c) Node2 - PUBLICATION pub2 for the employee table
d) Node1 - SUBSCRIPTION from pub2 with local_only=true

~~~

4. Commit message - meaning?

CURRENT
Now when user is trying to add another node Node3 to the
above Multi master logical replication setup:
a) user will have to create one subscription subscribing from Node1 to
Node3
b) user wil have to create another subscription subscribing from
Node2 to Node3 using local_only option and copy_data as true.

While the subscription is created, server will identify that Node2 is
subscribing from Node1 and Node1 is subscribing from Node2 and throw an
error so that user can handle the initial copy data.

~

The wording above confused me. Can you clarify it?
e.g.
a) What exactly was the user hoping to achieve here?
b) Are the user steps doing something deliberately wrong just so you
can describe later that an error gets thrown?

~~~

5. doc/src/sgml/logical-replication.sgml - how to get here?

I didn’t see any easy way to get to this page. (No cross refs from anywhere?)

~~~

6. doc/src/sgml/logical-replication.sgml - section levels

I think the section levels may be a bit messed up. e.g. The HTML
rendering of sections looks a bit confused. Maybe this is same as my
review comment #13.

~~

7. doc/src/sgml/logical-replication.sgml - headings

<title>Setting Bidirection logical replication between two nodes:</title>

7a. Maybe better just to have a simpler main heading like
"Bidirectional logical replication".

7b. Don't put ":" in the title.

~~~

8. doc/src/sgml/logical-replication.sgml - missing intro

IMO this page needs some sort of introduction/blurb instead of leaping
straight into examples without any preamble text to give context.

~~~

9. doc/src/sgml/logical-replication.sgml - bullets

Suggest removing all the bullets from the example steps. (I had
something similar a while ago for the "Row Filter" page but
eventually, they all had to be removed).

~~~

10. doc/src/sgml/logical-replication.sgml - SQL too long

Many of the example commands are much too long, and for me, they are
giving scroll bars when rendered. It would be better if they can be
wrapped in appropriate places so easier to read (and no resulting
scroll bars).

~~~

11. doc/src/sgml/logical-replication.sgml - add the psql prompt

IMO it would also be easier to understand the examples if you show the
psql prompt. Then you can easily know the node context without having
to describe it in the text so often.

e.g.

+    <para>
+     Create the subscription in node2 to subscribe the changes from node1:
+<programlisting>
+CREATE SUBSCRIPTION sub_node1_node2 ...

SUGGGESTION
+    <para>
+     Create the subscription in node2 to subscribe the changes from node1
+<programlisting>
+node_2=# CREATE SUBSCRIPTION sub_node1_node2 ...

~~~

12. doc/src/sgml/logical-replication.sgml - typo

+  <para>
+   Now the BiDirectional logical replication setup is complete between node1

typo "BiDirectional"

~~~

13. doc/src/sgml/logical-replication.sgml - deep levels

The section levels are very deep, but > 3 will not appear in the table
of contents when rendered. Maybe you can rearrange to raise them all
up one level, then IMO the TOC will work better and the whole page
will be easier to read.

~~~

14. doc/src/sgml/logical-replication.sgml - unnecessarily complex?

+<programlisting>
+# Truncate the table data but do not replicate the truncate.
+ALTER PUBLICATION pub_node3 SET (publish='insert,update,delete');
+TRUNCATE t1;
+
+CREATE SUBSCRIPTION sub_node3_node1 CONNECTION 'dbname=foo host=node1
user=repuser' PUBLICATION pub_node1 WITH (copy_data = force,
local_only = on);
+
+CREATE SUBSCRIPTION sub_node3_node2 CONNECTION 'dbname=foo host=node2
user=repuser' PUBLICATION pub_node2 WITH (copy_data = off, local_only
= on);
+
+# Include truncate operations from now
+ALTER PUBLICATION pub_node3 SET (publish='insert,update,delete,truncate');
+</programlisting>

Is it really necessary for those CREATE SUBSCRIPTION to be placed
where they are? I felt it would be less confusing if you just do the
TRUNCATE between the 2x ALTER PUBLICATION... and then do those CREATE
SUBSCRIPTION separately later.

~~~

14. doc/src/sgml/logical-replication.sgml - force?

There seems no documentation anywhere that says what is the
purpose/meaning of the copy_data enum "force".

~~~

15. doc/src/sgml/ref/alter_subscription.sgml - force?

Meanings of copy_data true/false are self-evident but should something
here be explaining what is the meaning of "force"? Or a reference to
some place that explains it?

~~~

16. doc/src/sgml/ref/create_subscription.sgml - local_only missing notes?

@@ -161,6 +161,11 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
           publisher node changes regardless of their origin. The default is
           <literal>false</literal>.
          </para>
+         <para>
+          There is some interation between the "local_only" option and
+          "copy_data" option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>
         </listitem>
        </varlistentry>

16a. Typo "interation"

16b. But where in the referenced notes does it actually say anything about this?

~~~

17. doc/src/sgml/ref/create_subscription.sgml - force?

The notes about the copy_data do not say anything about what the
values mean. Specifically, there seems nothing that describes what is
the meaning of "force".

~~~

18. doc/src/sgml/ref/create_subscription.sgml - copy_data missing notes?

+
+         <para>
+          There is some interation between the "local_only" option and
+          "copy_data" option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>

18a. Typo "interation"

18b. But where in the referenced notes does it actually say anything about this?

~~~

19. doc/src/sgml/ref/create_subscription.sgml - xref to the new page

Shouldn't there be an xref on this page somewhere to your other new
"Bidirectional" docs page?

~~~

20. src/backend/commands/subscriptioncmds.c - IS_COPY_DATA_ON_OR_FORCE

@@ -69,6 +69,18 @@
 /* check if the 'val' has 'bits' set */
 #define IsSet(val, bits)  (((val) & (bits)) == (bits))

+#define IS_COPY_DATA_ON_OR_FORCE(copy_data) (copy_data != COPY_DATA_OFF)

Maybe this would be better as a static inline function?

~~~

21. src/backend/commands/subscriptioncmds.c - comment

+/*
+ * Represents whether copy_data option is specified with on, off or force.
+ */
+typedef enum CopyData
+{
+ COPY_DATA_OFF,
+ COPY_DATA_ON,
+ COPY_DATA_FORCE
+} CopyData;

I felt it might be better if the comment described these enums in the
same order they are defined.

E.g. "Represents whether copy_data option is specified as off/on, or force."

~~~

22. src/backend/commands/subscriptioncmds.c - CreateSubscription bug?

@@ -720,7 +788,8 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
  * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
  * PUBLICATION to work.
  */
- if (opts.twophase && !opts.copy_data && tables != NIL)
+ if (opts.twophase && IS_COPY_DATA_ON_OR_FORCE(opts.copy_data)
+ && tables != NIL)
  twophase_enabled = true;
Is that a bug? It does not seem to match the previous code. Should
that IS_COPY_DATA_ON_OR_FORCE be "not" ?

~~~

23. src/backend/commands/subscriptioncmds.c - long errmsg

+ if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("CREATE/ALTER SUBSCRIPTION with local_only and copy_data as
true is not allowed when the publisher might have replicated data,
table:%s.%s might have replicated data in the publisher",
+    nspname, relname),
+ errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
+

The errmsg seems way too long for the source code. Can you use string
concatenation or continuation chars to wrap the message over multiple
lines?

~~~

24. src/test/regress/sql/subscription.sql - typo

@@ -66,6 +69,15 @@ ALTER SUBSCRIPTION regress_testsub4 SET (local_only = false);
 DROP SUBSCRIPTION regress_testsub3;
 DROP SUBSCRIPTION regress_testsub4;

+-- ok - valid coy_data options

Typo "coy_data". (it looks like this typo is not caused by this patch,
but I think this patch should fix it anyhow).

~~~

25. src/test/regress/sql/subscription.sql - test order

The new tests are OK but IMO they could be re-ordered so then they
will be more consistent for the positive and negative tests.

CURRENT
"true/force/on/1" and "off/false/0"

SUGGEST
"true/on/1/force" and "false/off/0"

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  69. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-20T05:49:24Z

    Below are my review comments for the v9-0002 patch (TAP test part only).

(The order of my comments is a bit muddled because I jumped around in
the file a bit while reviewing it).

======

1. create_subscription - missing comment.

+sub create_subscription
+{
+ my ($node_subscriber, $node_publisher, $sub_name, $node_connstr,
+ $application_name, $pub_name, $copy_data_val)
+   = @_;

Maybe add a comment for this subroutine and describe the expected parameters.

~~

2. create_subscription - hides the options

IMO the "create_subscription" subroutine is hiding too many of the
details, so now it is less clear (from the caller's POV) what the test
is really doing. Perhaps the options could be passed more explicitly
to this subroutine.

e.g. Instead of

create_subscription($node_B, $node_A, $appname_B1, $node_A_connstr,
       $appname_B1, 'tap_pub_A', 'on');

perhaps explicitly set the WITH options like:

create_subscription($node_B, $node_A, $appname_B1, $node_A_connstr,
       $appname_B1, 'tap_pub_A', 'local_only = on, copy_data = on');

~~~

3. the application names are confusing

+my $appname_A1 = 'tap_sub_A_B';
+my $appname_A2 = 'tap_sub_A_C';
+my $appname_B1 = 'tap_sub_B_A';
+my $appname_B2 = 'tap_sub_B_C';
+my $appname_C1 = 'tap_sub_C_A';
+my $appname_C2 = 'tap_sub_C_B';

I found the application names using '1' and '2' to be a bit confusing.
i.e  it was unnecessarily hard to associate them (in my head) with
their relevant subscriptions. IMO it would be easier to name them
using letters like below:

SUGGESTED
my $appname_AB = 'tap_sub_A_B';
my $appname_AC = 'tap_sub_A_C';
my $appname_BA = 'tap_sub_B_A';
my $appname_BC = 'tap_sub_B_C';
my $appname_CA = 'tap_sub_C_A';
my $appname_CB = 'tap_sub_C_B';

~~~

4. create_subscription - passing the $appname 2x.

+create_subscription($node_B, $node_A, $appname_B1, $node_A_connstr,
+       $appname_B1, 'tap_pub_A', 'on');


It seemed confusing that the $app_name is passed twice.

IMO should rename all those $appname_XX vars (see previous comment) to
be $subname_XX, and just pass that create_subscription instead.

my $subname_AB = 'tap_sub_A_B';
my $subname_AC = 'tap_sub_A_C';
my $subname_BA = 'tap_sub_B_A';
my $subname_BC = 'tap_sub_B_C';
my $subname_CA = 'tap_sub_C_A';
my $subname_CB = 'tap_sub_C_B';

Then create_subscription subroutine should have one less argument.
Just add a comment saying that the appname is always assigned the same
value as the subname.

~~~

5. cleanup part - seems a bit misleading

+# cleanup
+$node_B->safe_psql(
+       'postgres', "
+        DROP SUBSCRIPTION $appname_B2");
+$node_C->safe_psql(
+     'postgres', "
+        DELETE FROM tab_full");
+$node_B->safe_psql(
+     'postgres', "
+        DELETE FROM tab_full where a = 13");

Is that comment misleading? IIUC this is not really cleaning up
everything. It is just a cleanup of the previous Node_C test part
isn't it?

~~~

6. Error case (when copy_data is true)

+# Error when creating susbcription with local_only and copy_data as true when
+# the publisher has replicated data

6a. Typo "susbcription"

6b. That comment maybe needs some more explanation - eg. say that
since Node_A is already subscribing to Node_B so when Node_B makes
another subscription to Node_A the copy doesn't really know if the
data really originated from Node_A or not...

6c. Maybe the comment needed to be more like ############## style to
denote this (and the one that follows) is a separate test case.

~~~

7. Error case (when copy_data is force)

+# Creating subscription with local_only and copy_data as force should be
+# successful when the publisher has replicated data

7a. Same typo "subscription"

~~~

8. Add 3rd node when the existing node has some data

8a. Maybe that comment needs to be expanded more. This isn't just
"adding a node" - you are joining it to the others as another
bi-directional participant in a 3-way group. And the comment should
clarify exactly which nodes have the initial data. Both the existing 2
you are joining to? The new one only? All of them?

8b. IMO is would be much clearer to do SELECT from all the nodes at
the start of this test just to re-confirm what is all the initial data
on these nodes before joining the 3rd node.

NOTE - These same review comments apply to the other test combinations too
- Add 3rd node when the existing node has no data
- Add 3rd node when the new node has some data

~~~

9. Inserted data

+# insert some data in all the nodes
+$node_A->safe_psql('postgres', "INSERT INTO tab_full VALUES (13);");
+$node_B->safe_psql('postgres', "INSERT INTO tab_full VALUES (21);");
+$node_C->safe_psql('postgres', "INSERT INTO tab_full VALUES (31);");

They seemed strange values (13, 21, 31) to add. I couldn't work out
the "meaning" of them.

Wouldn't values like 13, 23, 33 make more sense (e.g pattern is 10 x
node# + something)

~~~

10. verify_data($node_A, $node_B, $node_C);

All the expected values are too buried in this subroutine which makes
it hard to read. I think it would be easier to read (from the caller's
POV) if you can pass the *expected* values as another arg into the
verify_data subroutine.

e.g. is something like this possible?
verify_data($node_A, $node_B, $node_C, [11,12,13])

~~~

11. ALTER for ignoring  'truncate'

+$node_C->safe_psql('postgres',
+       "ALTER PUBLICATION tap_pub_C SET (publish='insert,update,delete');");
+
+$node_C->safe_psql('postgres', "TRUNCATE tab_full");
+
+create_subscription($node_C, $node_A, $appname_C1, $node_A_connstr,
+       $appname_C1, 'tap_pub_A', 'force');
+create_subscription($node_C, $node_B, $appname_C2, $node_B_connstr,
+       $appname_C2, 'tap_pub_B', 'off');
+
+#include truncates now
+$node_C->safe_psql('postgres',
+       "ALTER PUBLICATION tap_pub_C SET
(publish='insert,update,delete,truncate');"
+);

Do those create_subscription calls need to be encapsulated by the
ALTER like that? I did not think so. It looks a bit more complex if
done this way.

~~~

12. clean_subscriber_contents - misnamed?

This subroutine feels a bit misnamed. It seems to be doing lots of
things like detaching the Node_C and deleting all table data from all
nodes. That all seems quite different from just "clean subscriber
contents".

~~~

13. table initial data?

+clean_subscriber_contents($node_A, $node_B, $node_C);
+
+##########################################################################
+# Add 3rd node when the new node has some data
+##########################################################################

But does this test case *really* have some data? I am not so sure.
Doesn't the preceding "clean_subscriber_contents" call remove all the
data that might have been there? That is why I think all the tests out
to have SELECT (previous comment #8) so they can re-confirm what data
is really in those tables before doing each test part.


------
Kind Regards,
Peter Smith.
Fujitsu Australia

  70. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-22T16:15:25Z

    On Tue, Apr 19, 2022 at 8:29 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> I checked the latest v9-0001 patch. Below are my review comments.
>
> Other than these few trivial comments this 0001 patch looks good to me.
>
> ~~~
>
> 1. src/backend/replication/pgoutput/pgoutput.c - whitespace
>
> @@ -1696,6 +1714,10 @@ static bool
>  pgoutput_origin_filter(LogicalDecodingContext *ctx,
>      RepOriginId origin_id)
>  {
> + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
> +
> + if (data->local_only && origin_id != InvalidRepOriginId)
> + return true;
>   return false;
>  }
>
> Suggest to add a blank line after the return true;

Modified

> ~~~
>
> 2. src/bin/psql/tab-complete.c - not alphabetical
>
> @@ -1874,7 +1874,7 @@ psql_completion(const char *text, int start, int end)
>   COMPLETE_WITH("(", "PUBLICATION");
>   /* ALTER SUBSCRIPTION <name> SET ( */
>   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> TailMatches("SET", "("))
> - COMPLETE_WITH("binary", "slot_name", "streaming",
> "synchronous_commit", "disable_on_error");
> + COMPLETE_WITH("binary", "slot_name", "streaming", "local_only",
> "synchronous_commit", "disable_on_error");
>
> 2a. AFAIK the code intended that these options be listed in
> alphabetical order (I think the recent addition of disable_on_error is
> also wrong here). So "local_only" should be moved.

Modified

> @@ -3156,7 +3156,7 @@ psql_completion(const char *text, int start, int end)
>   /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
>   else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
>   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> -   "enabled", "slot_name", "streaming",
> +   "enabled", "slot_name", "streaming", "local_only",
>     "synchronous_commit", "two_phase", "disable_on_error");
>
> 2b. ditto

Modified

> ~~~
>
> 3. src/test/subscription/t/032_localonly.pl - wrong message
>
> +$node_C->wait_for_catchup($appname_B2);
> +$node_B->wait_for_catchup($appname_A);
> +
> +$result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full;");
> +is( $result, qq(11
> +12
> +13),
> + 'Inserted successfully without leading to infinite recursion in
> circular replication setup'
> +);
> +
> +# check that the data published from node_C to node_B is not sent to node_A
> +$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full;");
> +is( $result, qq(11
> +12),
> + 'Inserted successfully without leading to infinite recursion in
> circular replication setup'
> +);
> +
>
> The new test looked good, but the cut/paste text message ('Inserted
> successfully without leading to infinite recursion in circular
> replication setup') maybe needs changing because there is nothing
> really "circular" about this test case.

Modified

Attached v10 patch has the changes for the same.

Regards,
Vignesh

  71. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-22T16:23:48Z

    On Wed, Apr 20, 2022 at 7:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Below are my review comments for the v9-0002 patch (except I did not
> yet look at the TAP tests).
>
> ~~~
>
> 1. General comment - describe.c
>
> I wondered why the copy_data enum value is not displayed by the psql
> \drs+ command. Should it be?
>

I noticed that we generally don't display the values for the options.
I did not add it to keep it consistent with other options.

> ~~~
>
> 2. General comment - SUBOPT_LOCAL_ONLY
>
> @@ -1134,7 +1204,7 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>
>   case ALTER_SUBSCRIPTION_SET_PUBLICATION:
>   {
> - supported_opts = SUBOPT_COPY_DATA | SUBOPT_REFRESH;
> + supported_opts = SUBOPT_COPY_DATA | SUBOPT_REFRESH | SUBOPT_LOCAL_ONLY;
>   parse_subscription_options(pstate, stmt->options,
>      supported_opts, &opts);
>
> @@ -1236,7 +1308,8 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>   errmsg("ALTER SUBSCRIPTION ... REFRESH is not allowed for disabled
> subscriptions")));
>
>   parse_subscription_options(pstate, stmt->options,
> -    SUBOPT_COPY_DATA, &opts);
> +    SUBOPT_COPY_DATA | SUBOPT_LOCAL_ONLY,
> +    &opts);
>
> I noticed there is some new code that appears to be setting the
> SUBOT_LOCAL_ONLY as a supported option. Shouldn't those changes belong
> in the patch 0001 instead of in this patch 0002?

Modified

> ~~~
>
> 3. Commit message - wording
>
> CURRENT
> a) Node1 - Publication publishing employee table.
> b) Node2 - Subscription subscribing from publication pub1 with
> local_only.
> c) Node2 - Publication publishing employee table.
> d) Node1 - Subscription subscribing from publication pub2 with
> local_only.
>
> SUGGESTION (I think below has the same meaning but is simpler)
> a) Node1 - PUBLICATION pub1 for the employee table
> b) Node2 - SUBSCRIPTION from pub1 with local_only=true
> c) Node2 - PUBLICATION pub2 for the employee table
> d) Node1 - SUBSCRIPTION from pub2 with local_only=true

Modified

> ~~~
>
> 4. Commit message - meaning?
>
> CURRENT
> Now when user is trying to add another node Node3 to the
> above Multi master logical replication setup:
> a) user will have to create one subscription subscribing from Node1 to
> Node3
> b) user wil have to create another subscription subscribing from
> Node2 to Node3 using local_only option and copy_data as true.
>
> While the subscription is created, server will identify that Node2 is
> subscribing from Node1 and Node1 is subscribing from Node2 and throw an
> error so that user can handle the initial copy data.
>
> ~
>
> The wording above confused me. Can you clarify it?
> e.g.
> a) What exactly was the user hoping to achieve here?
> b) Are the user steps doing something deliberately wrong just so you
> can describe later that an error gets thrown?

Modified

> ~~~
>
> 5. doc/src/sgml/logical-replication.sgml - how to get here?
>
> I didn’t see any easy way to get to this page. (No cross refs from anywhere?)

I have added a link from create subscription page

> ~~~
>
> 6. doc/src/sgml/logical-replication.sgml - section levels
>
> I think the section levels may be a bit messed up. e.g. The HTML
> rendering of sections looks a bit confused. Maybe this is same as my
> review comment #13.

Modified

> ~~
>
> 7. doc/src/sgml/logical-replication.sgml - headings
>
> <title>Setting Bidirection logical replication between two nodes:</title>
>
> 7a. Maybe better just to have a simpler main heading like
> "Bidirectional logical replication".

Modified

> 7b. Don't put ":" in the title.

Modified

> ~~~
>
> 8. doc/src/sgml/logical-replication.sgml - missing intro
>
> IMO this page needs some sort of introduction/blurb instead of leaping
> straight into examples without any preamble text to give context.

Modified

> ~~~
>
> 9. doc/src/sgml/logical-replication.sgml - bullets
>
> Suggest removing all the bullets from the example steps. (I had
> something similar a while ago for the "Row Filter" page but
> eventually, they all had to be removed).

Modified

> ~~~
>
> 10. doc/src/sgml/logical-replication.sgml - SQL too long
>
> Many of the example commands are much too long, and for me, they are
> giving scroll bars when rendered. It would be better if they can be
> wrapped in appropriate places so easier to read (and no resulting
> scroll bars).

Modified

> ~~~
>
> 11. doc/src/sgml/logical-replication.sgml - add the psql prompt
>
> IMO it would also be easier to understand the examples if you show the
> psql prompt. Then you can easily know the node context without having
> to describe it in the text so often.
>
> e.g.
>
> +    <para>
> +     Create the subscription in node2 to subscribe the changes from node1:
> +<programlisting>
> +CREATE SUBSCRIPTION sub_node1_node2 ...
>
> SUGGGESTION
> +    <para>
> +     Create the subscription in node2 to subscribe the changes from node1
> +<programlisting>
> +node_2=# CREATE SUBSCRIPTION sub_node1_node2 ...

Modified

> ~~~
>
> 12. doc/src/sgml/logical-replication.sgml - typo
>
> +  <para>
> +   Now the BiDirectional logical replication setup is complete between node1
>
> typo "BiDirectional"

Modified to bidirectional

> ~~~
>
> 13. doc/src/sgml/logical-replication.sgml - deep levels
>
> The section levels are very deep, but > 3 will not appear in the table
> of contents when rendered. Maybe you can rearrange to raise them all
> up one level, then IMO the TOC will work better and the whole page
> will be easier to read.

Modified

> ~~~
>
> 14. doc/src/sgml/logical-replication.sgml - unnecessarily complex?
>
> +<programlisting>
> +# Truncate the table data but do not replicate the truncate.
> +ALTER PUBLICATION pub_node3 SET (publish='insert,update,delete');
> +TRUNCATE t1;
> +
> +CREATE SUBSCRIPTION sub_node3_node1 CONNECTION 'dbname=foo host=node1
> user=repuser' PUBLICATION pub_node1 WITH (copy_data = force,
> local_only = on);
> +
> +CREATE SUBSCRIPTION sub_node3_node2 CONNECTION 'dbname=foo host=node2
> user=repuser' PUBLICATION pub_node2 WITH (copy_data = off, local_only
> = on);
> +
> +# Include truncate operations from now
> +ALTER PUBLICATION pub_node3 SET (publish='insert,update,delete,truncate');
> +</programlisting>
>
> Is it really necessary for those CREATE SUBSCRIPTION to be placed
> where they are? I felt it would be less confusing if you just do the
> TRUNCATE between the 2x ALTER PUBLICATION... and then do those CREATE
> SUBSCRIPTION separately later.

Modified

> ~~~
>
> 14. doc/src/sgml/logical-replication.sgml - force?
>
> There seems no documentation anywhere that says what is the
> purpose/meaning of the copy_data enum "force".

This is added in create subscription notes

> ~~~
>
> 15. doc/src/sgml/ref/alter_subscription.sgml - force?
>
> Meanings of copy_data true/false are self-evident but should something
> here be explaining what is the meaning of "force"? Or a reference to
> some place that explains it?

This is mentioned in create subscription notes, added a reference to
create subscription notes.

> ~~~
>
> 16. doc/src/sgml/ref/create_subscription.sgml - local_only missing notes?
>
> @@ -161,6 +161,11 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>            publisher node changes regardless of their origin. The default is
>            <literal>false</literal>.
>           </para>
> +         <para>
> +          There is some interation between the "local_only" option and
> +          "copy_data" option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>          </listitem>
>         </varlistentry>
>
> 16a. Typo "interation"

Modified

> 16b. But where in the referenced notes does it actually say anything about this?

Modified

> ~~~
>
> 17. doc/src/sgml/ref/create_subscription.sgml - force?
>
> The notes about the copy_data do not say anything about what the
> values mean. Specifically, there seems nothing that describes what is
> the meaning of "force".

I have mentioned this in the notes section.

> ~~~
>
> 18. doc/src/sgml/ref/create_subscription.sgml - copy_data missing notes?
>
> +
> +         <para>
> +          There is some interation between the "local_only" option and
> +          "copy_data" option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>
> 18a. Typo "interation"

Modified

> 18b. But where in the referenced notes does it actually say anything about this?

Added

> ~~~
>
> 19. doc/src/sgml/ref/create_subscription.sgml - xref to the new page
>
> Shouldn't there be an xref on this page somewhere to your other new
> "Bidirectional" docs page?

Added reference

> ~~~
>
> 20. src/backend/commands/subscriptioncmds.c - IS_COPY_DATA_ON_OR_FORCE
>
> @@ -69,6 +69,18 @@
>  /* check if the 'val' has 'bits' set */
>  #define IsSet(val, bits)  (((val) & (bits)) == (bits))
>
> +#define IS_COPY_DATA_ON_OR_FORCE(copy_data) (copy_data != COPY_DATA_OFF)
>
> Maybe this would be better as a static inline function?

 What is the advantage of doing this change? I have not changed this
as the macro usage is fine.
Thoughts?

> ~~~
>
> 21. src/backend/commands/subscriptioncmds.c - comment
>
> +/*
> + * Represents whether copy_data option is specified with on, off or force.
> + */
> +typedef enum CopyData
> +{
> + COPY_DATA_OFF,
> + COPY_DATA_ON,
> + COPY_DATA_FORCE
> +} CopyData;
>
> I felt it might be better if the comment described these enums in the
> same order they are defined.
>
> E.g. "Represents whether copy_data option is specified as off/on, or force."

Modified

> ~~~
>
> 22. src/backend/commands/subscriptioncmds.c - CreateSubscription bug?
>
> @@ -720,7 +788,8 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>   * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
>   * PUBLICATION to work.
>   */
> - if (opts.twophase && !opts.copy_data && tables != NIL)
> + if (opts.twophase && IS_COPY_DATA_ON_OR_FORCE(opts.copy_data)
> + && tables != NIL)
>   twophase_enabled = true;
> Is that a bug? It does not seem to match the previous code. Should
> that IS_COPY_DATA_ON_OR_FORCE be "not" ?

Modified

> ~~~
>
> 23. src/backend/commands/subscriptioncmds.c - long errmsg
>
> + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("CREATE/ALTER SUBSCRIPTION with local_only and copy_data as
> true is not allowed when the publisher might have replicated data,
> table:%s.%s might have replicated data in the publisher",
> +    nspname, relname),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
> +
>
> The errmsg seems way too long for the source code. Can you use string
> concatenation or continuation chars to wrap the message over multiple
> lines?

I had seen that the long error message elsewhere also is in a single
line. I think we should keep it as it is to maintain the coding
standard.
Thoughts?

> ~~~
>
> 24. src/test/regress/sql/subscription.sql - typo
>
> @@ -66,6 +69,15 @@ ALTER SUBSCRIPTION regress_testsub4 SET (local_only = false);
>  DROP SUBSCRIPTION regress_testsub3;
>  DROP SUBSCRIPTION regress_testsub4;
>
> +-- ok - valid coy_data options
>
> Typo "coy_data". (it looks like this typo is not caused by this patch,
> but I think this patch should fix it anyhow).

Modified

> ~~~
>
> 25. src/test/regress/sql/subscription.sql - test order
>
> The new tests are OK but IMO they could be re-ordered so then they
> will be more consistent for the positive and negative tests.
>
> CURRENT
> "true/force/on/1" and "off/false/0"
>
> SUGGEST
> "true/on/1/force" and "false/off/0"

Modified

Thanks for the comments, the v10 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0PmOz71O6ofhZkB0rts5Ak2HUhMuuMQoViH_LAXTBeBw%40mail.gmail.com

Regards,
Vignesh

  72. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-22T16:28:22Z

    On Wed, Apr 20, 2022 at 11:19 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Below are my review comments for the v9-0002 patch (TAP test part only).
>
> (The order of my comments is a bit muddled because I jumped around in
> the file a bit while reviewing it).
>
> ======
>
> 1. create_subscription - missing comment.
>
> +sub create_subscription
> +{
> + my ($node_subscriber, $node_publisher, $sub_name, $node_connstr,
> + $application_name, $pub_name, $copy_data_val)
> +   = @_;
>
> Maybe add a comment for this subroutine and describe the expected parameters.

Added

> ~~
>
> 2. create_subscription - hides the options
>
> IMO the "create_subscription" subroutine is hiding too many of the
> details, so now it is less clear (from the caller's POV) what the test
> is really doing. Perhaps the options could be passed more explicitly
> to this subroutine.
>
> e.g. Instead of
>
> create_subscription($node_B, $node_A, $appname_B1, $node_A_connstr,
>        $appname_B1, 'tap_pub_A', 'on');
>
> perhaps explicitly set the WITH options like:
>
> create_subscription($node_B, $node_A, $appname_B1, $node_A_connstr,
>        $appname_B1, 'tap_pub_A', 'local_only = on, copy_data = on');

Modified

> ~~~
>
> 3. the application names are confusing
>
> +my $appname_A1 = 'tap_sub_A_B';
> +my $appname_A2 = 'tap_sub_A_C';
> +my $appname_B1 = 'tap_sub_B_A';
> +my $appname_B2 = 'tap_sub_B_C';
> +my $appname_C1 = 'tap_sub_C_A';
> +my $appname_C2 = 'tap_sub_C_B';
>
> I found the application names using '1' and '2' to be a bit confusing.
> i.e  it was unnecessarily hard to associate them (in my head) with
> their relevant subscriptions. IMO it would be easier to name them
> using letters like below:
>
> SUGGESTED
> my $appname_AB = 'tap_sub_A_B';
> my $appname_AC = 'tap_sub_A_C';
> my $appname_BA = 'tap_sub_B_A';
> my $appname_BC = 'tap_sub_B_C';
> my $appname_CA = 'tap_sub_C_A';
> my $appname_CB = 'tap_sub_C_B';

Removed appname and used subscription names for application name

> ~~~
>
> 4. create_subscription - passing the $appname 2x.
>
> +create_subscription($node_B, $node_A, $appname_B1, $node_A_connstr,
> +       $appname_B1, 'tap_pub_A', 'on');
>
>
> It seemed confusing that the $app_name is passed twice.
>
> IMO should rename all those $appname_XX vars (see previous comment) to
> be $subname_XX, and just pass that create_subscription instead.
>
> my $subname_AB = 'tap_sub_A_B';
> my $subname_AC = 'tap_sub_A_C';
> my $subname_BA = 'tap_sub_B_A';
> my $subname_BC = 'tap_sub_B_C';
> my $subname_CA = 'tap_sub_C_A';
> my $subname_CB = 'tap_sub_C_B';
>
> Then create_subscription subroutine should have one less argument.
> Just add a comment saying that the appname is always assigned the same
> value as the subname.

Modifeid

> ~~~
>
> 5. cleanup part - seems a bit misleading
>
> +# cleanup
> +$node_B->safe_psql(
> +       'postgres', "
> +        DROP SUBSCRIPTION $appname_B2");
> +$node_C->safe_psql(
> +     'postgres', "
> +        DELETE FROM tab_full");
> +$node_B->safe_psql(
> +     'postgres', "
> +        DELETE FROM tab_full where a = 13");
>
> Is that comment misleading? IIUC this is not really cleaning up
> everything. It is just a cleanup of the previous Node_C test part
> isn't it?

Modified to clear the operations done by this test

> ~~~
>
> 6. Error case (when copy_data is true)
>
> +# Error when creating susbcription with local_only and copy_data as true when
> +# the publisher has replicated data
>
> 6a. Typo "susbcription"

Modified

> 6b. That comment maybe needs some more explanation - eg. say that
> since Node_A is already subscribing to Node_B so when Node_B makes
> another subscription to Node_A the copy doesn't really know if the
> data really originated from Node_A or not...

Slightly reworded and modified

> 6c. Maybe the comment needed to be more like ############## style to
> denote this (and the one that follows) is a separate test case.

Modified

> ~~~
>
> 7. Error case (when copy_data is force)
>
> +# Creating subscription with local_only and copy_data as force should be
> +# successful when the publisher has replicated data
>
> 7a. Same typo "subscription"

I felt subscription is correct in this case, made no change for this

> ~~~
>
> 8. Add 3rd node when the existing node has some data
>
> 8a. Maybe that comment needs to be expanded more. This isn't just
> "adding a node" - you are joining it to the others as another
> bi-directional participant in a 3-way group. And the comment should
> clarify exactly which nodes have the initial data. Both the existing 2
> you are joining to? The new one only? All of them?

Modified

> 8b. IMO is would be much clearer to do SELECT from all the nodes at
> the start of this test just to re-confirm what is all the initial data
> on these nodes before joining the 3rd node.

Modified

> NOTE - These same review comments apply to the other test combinations too
> - Add 3rd node when the existing node has no data
> - Add 3rd node when the new node has some data

Modified

> ~~~
>
> 9. Inserted data
>
> +# insert some data in all the nodes
> +$node_A->safe_psql('postgres', "INSERT INTO tab_full VALUES (13);");
> +$node_B->safe_psql('postgres', "INSERT INTO tab_full VALUES (21);");
> +$node_C->safe_psql('postgres', "INSERT INTO tab_full VALUES (31);");
>
> They seemed strange values (13, 21, 31) to add. I couldn't work out
> the "meaning" of them.
>
> Wouldn't values like 13, 23, 33 make more sense (e.g pattern is 10 x
> node# + something)

Modified

> ~~~
>
> 10. verify_data($node_A, $node_B, $node_C);
>
> All the expected values are too buried in this subroutine which makes
> it hard to read. I think it would be easier to read (from the caller's
> POV) if you can pass the *expected* values as another arg into the
> verify_data subroutine.
>
> e.g. is something like this possible?
> verify_data($node_A, $node_B, $node_C, [11,12,13])

Modified

> ~~~
>
> 11. ALTER for ignoring  'truncate'
>
> +$node_C->safe_psql('postgres',
> +       "ALTER PUBLICATION tap_pub_C SET (publish='insert,update,delete');");
> +
> +$node_C->safe_psql('postgres', "TRUNCATE tab_full");
> +
> +create_subscription($node_C, $node_A, $appname_C1, $node_A_connstr,
> +       $appname_C1, 'tap_pub_A', 'force');
> +create_subscription($node_C, $node_B, $appname_C2, $node_B_connstr,
> +       $appname_C2, 'tap_pub_B', 'off');
> +
> +#include truncates now
> +$node_C->safe_psql('postgres',
> +       "ALTER PUBLICATION tap_pub_C SET
> (publish='insert,update,delete,truncate');"
> +);
>
> Do those create_subscription calls need to be encapsulated by the
> ALTER like that? I did not think so. It looks a bit more complex if
> done this way.

Modified

> ~~~
>
> 12. clean_subscriber_contents - misnamed?
>
> This subroutine feels a bit misnamed. It seems to be doing lots of
> things like detaching the Node_C and deleting all table data from all
> nodes. That all seems quite different from just "clean subscriber
> contents".

changed it to detach_node_clean_table_data

> ~~~
>
> 13. table initial data?
>
> +clean_subscriber_contents($node_A, $node_B, $node_C);
> +
> +##########################################################################
> +# Add 3rd node when the new node has some data
> +##########################################################################
>
> But does this test case *really* have some data? I am not so sure.
> Doesn't the preceding "clean_subscriber_contents" call remove all the
> data that might have been there? That is why I think all the tests out
> to have SELECT (previous comment #8) so they can re-confirm what data
> is really in those tables before doing each test part.

Modified

Thanks for the comments, the v10 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0PmOz71O6ofhZkB0rts5Ak2HUhMuuMQoViH_LAXTBeBw%40mail.gmail.com

Regards,
Vignesh

  73. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-27T05:45:28Z

    Here are my review comments for v10-0001 and v10-0002.

(I did not yet look at the v10-0002 TAP tests. I will check those
separately later).

=================
v10-0001 comments
=================

1.1 src/include/catalog/pg_subscription.h

@@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW

  bool substream; /* Stream in-progress transactions. */

+ bool sublocalonly; /* skip copying of remote origin data */
+
  char subtwophasestate; /* Stream two-phase transactions */

  bool subdisableonerr; /* True if a worker error should cause the

1.1.a Comment should start uppercase.

1.1.b Perhaps it is better to say "Skip replication of" instead of
"Skip copying" ?

~~~

1.2 src/include/catalog/pg_subscription.h

@@ -110,6 +112,7 @@ typedef struct Subscription
  bool binary; /* Indicates if the subscription wants data in
  * binary format */
  bool stream; /* Allow streaming in-progress transactions. */
+ bool local_only; /* Skip copying of remote origin data */
  char twophasestate; /* Allow streaming two-phase transactions */

Same comment as #1.1

=================
v10-0002 comments
=================

2.1 Commit message

b) user wil have to create another subscription subscribing from
Node2 to Node3 using local_only option and copy_data as true.

Typo: "wil"

~~~

2.2 Commit message

Here when user has specified local_only 'on' which indicates that the
publisher should only replicate the changes that are generated locally, but in
this case since the publisher node is also subscribing data from other nodes,
the publisher node can have remotely originated data, so throw an error in this
case to prevent remotely generated data being replicated to the subscriber. If
user still intends to continue with the operation user can specify copy_data
as 'force' and proceed.

SUGGESTED (minor rewording)
In the scenario above the user has specified local_only 'on' (which
indicates that the publisher should only replicate the changes that
are generated locally), but in this case, the publisher node is also
subscribing data from other nodes, so the publisher node may have
remotely originated data. We throw an error, in this case, to draw
attention to there being possible remote data. If the user still
wishes to continue with the operation user can specify copy_data as
'force' and proceed.

~~~

2.3 doc/src/sgml/logical-replication.sgml

+   <para>
+     Bidirectional replication is useful in creating multi master database
+     which helps in performing read/write operations from any of the nodes.
+     Setting up bidirectional logical replication between two nodes requires
+     creation of the publication in all the nodes, creating subscription in
+     each of the nodes that subcribes to data from all the nodes. The steps
+     to create two node bidirectional replication is listed below:
+   </para>

2.3.a Typo: "subcribes"

2.3.b Wording: "creating subscription” -> "and creating subscriptions..."

2.3.c Wording: "The steps to create two node bidirectional replication
is listed below:" -> "The steps to create a two-node bidirectional
replication are given below:"

~~~

2.4 doc/src/sgml/logical-replication.sgml

+<programlisting>
+node2=# CREATE SUBSCRIPTION sub_node1_node2
+node2=# CONNECTION 'dbname=foo host=node1 user=repuser'
+node2=# PUBLICATION pub_node1
+node2=# WITH (copy_data = off, local_only = on);
+CREATE SUBSCRIPTION
+</programlisting>

I am not sure if those psql continuation prompts are right. Shouldn't
they be "node2-#" instead of "node2=#"?

~~~

2.5 doc/src/sgml/logical-replication.sgml

+<programlisting>
+node2=# CREATE SUBSCRIPTION sub_node1_node2
+node2=# CONNECTION 'dbname=foo host=node1 user=repuser'
+node2=# PUBLICATION pub_node1
+node2=# WITH (copy_data = off, local_only = on);
+CREATE SUBSCRIPTION
+</programlisting>
+   </para>

IIUC the closing </para> should be on the same line as the
</programlisting>. I recall there was some recent github push about
this sometime in the last month - maybe you can check to confirm it.

~~~

2.6 doc/src/sgml/logical-replication.sgml

+   <para>
+     Create the subscription in node2 to subscribe the changes from node1:
+<programlisting>
+node2=# CREATE SUBSCRIPTION sub_node1_node2

IMO the naming convention here is backwards/confusing. E.g. The
"pub_node1" is the publisher at node1. So similarly, I think the
subscriber at node2 should be called "sub_node2_node1" (not
"sub_node1_node2")

~~~

2.7 doc/src/sgml/logical-replication.sgml

+   <para>
+     Adding a new node node3 to the existing node1 and node2 requires setting
+     up subscription in node1 and node2 to replicate the data from node3 and
+     setting up subscription in node3 to replicate data from node1 and node2.
+     The steps for the same is listed below:
+   </para>

There are several sections that say "The steps for the same is listed
below:". The sentence for all of them seemed redundant to me.

~~~

2.8 doc/src/sgml/logical-replication.sgml

+    <para>
+     Adding a new node node3 to the existing node1 and node2 when data is
+     present in existing nodes node1 and node2 needs similar steps, only change
+     required here is that node3 should create subscription with copy_data as
+     force to one of the existing nodes to receive the existing data during
+     initial data synchronization. The steps for the same is listed below:
+   </para>

I thought it should be 2 sentences. So "needs similar steps, only
change required here" -> "needs similar steps. The only change
required here..."

~~~

2.9 doc/src/sgml/logical-replication.sgml

I think every time you say option names or values like "copy_data" in
the text they should be using <literal> font.

~~~

2.10 doc/src/sgml/logical-replication.sgml

+   <para>
+     Adding a new node node3 to the existing node1 and node2 when data is
+     present in the new node node3 needs similar steps, few changes are
+     required here to get the existing data from node3 to node1 and node2 and
+     later cleaning up of data in node3 before synchronization of all the data
+     from the existing nodes. The steps for the same is listed below:
+   </para>

I thought it should be 2 sentences. So "needs similar steps, few
changes are required here" -> "needs similar steps. A few changes are
required here..."

~~~

2.11 doc/src/sgml/logical-replication.sgml

All the text that says "create subscription" and "create publication"
maybe should be change to "create a subscription" and "create a
publication" etc.

~~~

2.12 doc/src/sgml/ref/alter_subscription.sgml - copy_data

+         <para>
+          There is some interaction between the "local_only" option and
+          "copy_data" option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for interaction
+          details and usage of force for copy_data option.
          </para>

2.12.a Maybe options should not be in quotes like that. I think they
should be <literal> font.

2.12.b It is a bit misleading because there is no "Notes" section here
on this page. Maybe it should say refer to the CREATE SUBSCRIPTION
Notes.

2.12.c The last copy_data should also be <literal> font.

~~~

2.13 doc/src/sgml/ref/create_subscription.sgml - local_only

          </para>
+         <para>
+          There is some interaction between the "local_only" option and
+          "copy_data" option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>
         </listitem>

2.13.a Maybe options should not be in quotes like that. I think they
should be <literal> font.

2.13.b The last copy_data should also be <literal> font.

~~~

2.14 doc/src/sgml/ref/create_subscription.sgml - copy_data


          <para>
           Specifies whether to copy pre-existing data in the publications
-          that are being subscribed to when the replication starts.
-          The default is <literal>true</literal>.
+          that are being subscribed to when the replication starts. This
+          parameter may be either <literal>true</literal>,
+          <literal>false</literal> or <literal>force</literal>. The default is
+          <literal>true</literal>.
          </para>

2.14.a Maybe options should not be in quotes like that. I think they
should be <literal> font.

2.14.b The last copy_data should also be <literal> font.

~~~

2.15 doc/src/sgml/ref/create_subscription.sgml

@@ -374,6 +388,16 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   If subscription is created with local_only as 'on' and copy_data as 'on', it
+   will check if the publisher tables are being subscribed to any other
+   publisher and throw an error to prevent inconsistent data in the
+   subscription. User can continue with the copy operation without throwing any
+   error in this case by specifying copy_data as 'force'. Refer to the
+   <xref linkend="bidirectional-logical-replication"/> on how
+   copy_data and local_only can be used in bidirectional replication.
+  </para>

2.15.a I think all those mentions of copy_data and local_only and
their values should be in <literal> font instead of in quotes.

2.15.b Wording: "User can" -> "The user can"

~~~

2.16 src/backend/commands/subscriptioncmds.c - macro

> 20. src/backend/commands/subscriptioncmds.c - IS_COPY_DATA_ON_OR_FORCE
>
> @@ -69,6 +69,18 @@
>  /* check if the 'val' has 'bits' set */
>  #define IsSet(val, bits)  (((val) & (bits)) == (bits))
>
> +#define IS_COPY_DATA_ON_OR_FORCE(copy_data) (copy_data != COPY_DATA_OFF)
>
> Maybe this would be better as a static inline function?

Vignesh: What is the advantage of doing this change? I have not changed this
as the macro usage is fine. Thoughts?

Originally I was going to suggest the macro should use extra parens
like ((copy_data) != COPY_DATA_OFF), but then I thought if it was a
function then it would have enum type-checking which would be better.
If you want to keep the macro then please make the parens change.

~~~

2.17 src/backend/commands/subscriptioncmds.c - DefGetCopyData

+ /*
+ * The set of strings accepted here should match up with
+ * the grammar's opt_boolean_or_string production.
+ */
+ if (pg_strcasecmp(sval, "true") == 0 ||
+ pg_strcasecmp(sval, "on") == 0)
+ return COPY_DATA_ON;
+ if (pg_strcasecmp(sval, "false") == 0 ||
+ pg_strcasecmp(sval, "off") == 0)
+ return COPY_DATA_OFF;
+ if (pg_strcasecmp(sval, "force") == 0)
+ return COPY_DATA_FORCE;

I think you can change the order of these to be off/on/force, so then
the order is consistent with the T_Integer case.

~~~

2.18 src/backend/commands/subscriptioncmds.c - DefGetCopyData

+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires a boolean or \"force\"", def->defname));
+ return COPY_DATA_OFF;                                           /*
keep compiler quiet */

Excessive comment indent? Has pg_indent been run?

~~~

2.19 src/backend/commands/subscriptioncmds.c - AlterSubscription

@@ -1236,7 +1306,8 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
  errmsg("ALTER SUBSCRIPTION ... REFRESH is not allowed for disabled
subscriptions")));

  parse_subscription_options(pstate, stmt->options,
-    SUBOPT_COPY_DATA, &opts);
+    SUBOPT_COPY_DATA,
+    &opts);

This is a formatting change only. Maybe it does not belong in this
patch unless it is the result of pg_indent.

~~~

2.20 src/backend/commands/subscriptioncmds.c - fetch_table_list

> 23. src/backend/commands/subscriptioncmds.c - long errmsg
>
> + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("CREATE/ALTER SUBSCRIPTION with local_only and copy_data as
> true is not allowed when the publisher might have replicated data,
> table:%s.%s might have replicated data in the publisher",
> +    nspname, relname),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
> +
>
> The errmsg seems way too long for the source code. Can you use string
> concatenation or continuation chars to wrap the message over multiple
> lines?

Vignesh: I had seen that the long error message elsewhere also is in a
single line. I think we should keep it as it is to maintain the coding
standard. Thoughts?

OK, if you say it is already common practice then it's fine by me to
leave it as-is.

~~~

2.21 src/test/regress/expected/subscription.out - make check

make check fails.
 1 of 214 tests failed.

2.21.a It looks like maybe you did not update the expected ordering of
some of the tests, after some minor adjustments in subscriprion.sql in
v10. So the expected output needs to be fixed in the patch.

2.21.b. Suggest adding this patch to CF so that the cfbot can pick up
such test problems earlier.


------

Kind Regards,
Peter Smith.
Fujitsu Australia

  74. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-28T02:27:15Z

    Here are my review comments for v10-0002 (TAP tests part only)

FIle: src/test/subscription/t/032_localonly.pl

======

1.

+# Detach node C and clean the table contents.
+sub detach_node_clean_table_data
+{

1a. Maybe say "Detach node C from the node-group of (A, B, C) and
clean the table contents from all nodes"

1b. Actually wondered do you need to TRUNCATE from both A and B (maybe
only truncate 1 is OK since those nodes are still using MMC). OTOH
maybe your explicit way makes the test simpler.

~~~

2.

+# Subroutine for verify the data is replicated successfully.
+sub verify_data
+{

2a. Typo: "for verify"  -> "to verify"

2b. The messages in this function maybe are not very appropriate. They
say 'Inserted successfully without leading to infinite recursion in
circular replication setup', but really the function is only testing
all the data is the same as 'expected'. So it could be the result of
any operation - not just Insert.

~~~

3. SELECT ORDER BY?

$result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full;");
is($result, qq(11
12
13),
   'Node_C data replicated to Node_B'
);

I am not sure are these OK like this or if *every* SELECT use ORDER BY
to make sure the data is in the same qq expected order? There are
multiple cases like this.

(BTW, I think this comment needs to be applied for the v10-0001 patch,
maybe not v10-0002).

~~~

4.

+###############################################################################
+# Specifying local_only 'on' which indicates that the publisher should only
+# replicate the changes that are generated locally from node_B, but in
+# this case since the node_B is also subscribing data from node_A, node_B can
+# have data originated from node_A, so throw an error in this case to prevent
+# node_A data being replicated to the node_C.
+###############################################################################

There is something wrong with the description because there is no
"node_C" in this test. You are just creating a 2nd subscription on
node A.

~~

5.

+($result, $stdout, $stderr) = $node_A->psql(
+       'postgres', "
+        CREATE SUBSCRIPTION tap_sub_A3
+        CONNECTION '$node_B_connstr application_name=$subname_AB'
+        PUBLICATION tap_pub_B
+        WITH (local_only = on, copy_data = on)");
+like(

It seemed strange to call this 2nd subscription "tap_sub_A3". Wouldn't
it be better to call it "tap_sub_A2"?

~~~

6.

+###############################################################################
+# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B) bidirectional
+# replication setup when the existing nodes (node_A & node_B) has no data and
+# the new node (node_C) some pre-existing data.
+###############################################################################
+$node_C->safe_psql('postgres', "INSERT INTO tab_full VALUES (31);");
+
+$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full order by 1;");
+is( $result, qq(), 'Check existing data');
+
+$result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full order by 1;");
+is( $result, qq(), 'Check existing data');
+
+$result = $node_C->safe_psql('postgres', "SELECT * FROM tab_full order by 1;");
+is($result, qq(31), 'Check existing data');
+
+create_subscription($node_A, $node_C, $subname_AC, $node_C_connstr,
+       'tap_pub_C', 'copy_data = force, local_only = on');
+create_subscription($node_B, $node_C, $subname_BC, $node_C_connstr,
+       'tap_pub_C', 'copy_data = force, local_only = on');
+

Because the Node_C does not yet have any subscriptions aren't these
cases where you didn't really need to use "force"?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  75. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-28T17:04:12Z

    On Wed, Apr 27, 2022 at 11:15 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v10-0001 and v10-0002.
>
> (I did not yet look at the v10-0002 TAP tests. I will check those
> separately later).
>
> =================
> v10-0001 comments
> =================
>
> 1.1 src/include/catalog/pg_subscription.h
>
> @@ -70,6 +70,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>
>   bool substream; /* Stream in-progress transactions. */
>
> + bool sublocalonly; /* skip copying of remote origin data */
> +
>   char subtwophasestate; /* Stream two-phase transactions */
>
>   bool subdisableonerr; /* True if a worker error should cause the
>
> 1.1.a Comment should start uppercase.

Modified

> 1.1.b Perhaps it is better to say "Skip replication of" instead of
> "Skip copying" ?

Modified

> ~~~
>
> 1.2 src/include/catalog/pg_subscription.h
>
> @@ -110,6 +112,7 @@ typedef struct Subscription
>   bool binary; /* Indicates if the subscription wants data in
>   * binary format */
>   bool stream; /* Allow streaming in-progress transactions. */
> + bool local_only; /* Skip copying of remote origin data */
>   char twophasestate; /* Allow streaming two-phase transactions */
>
> Same comment as #1.1

Modified

> =================
> v10-0002 comments
> =================
>
> 2.1 Commit message
>
> b) user wil have to create another subscription subscribing from
> Node2 to Node3 using local_only option and copy_data as true.
>
> Typo: "wil"

Modified

> ~~~
>
> 2.2 Commit message
>
> Here when user has specified local_only 'on' which indicates that the
> publisher should only replicate the changes that are generated locally, but in
> this case since the publisher node is also subscribing data from other nodes,
> the publisher node can have remotely originated data, so throw an error in this
> case to prevent remotely generated data being replicated to the subscriber. If
> user still intends to continue with the operation user can specify copy_data
> as 'force' and proceed.
>
> SUGGESTED (minor rewording)
> In the scenario above the user has specified local_only 'on' (which
> indicates that the publisher should only replicate the changes that
> are generated locally), but in this case, the publisher node is also
> subscribing data from other nodes, so the publisher node may have
> remotely originated data. We throw an error, in this case, to draw
> attention to there being possible remote data. If the user still
> wishes to continue with the operation user can specify copy_data as
> 'force' and proceed.

Modified

> ~~~
>
> 2.3 doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +     Bidirectional replication is useful in creating multi master database
> +     which helps in performing read/write operations from any of the nodes.
> +     Setting up bidirectional logical replication between two nodes requires
> +     creation of the publication in all the nodes, creating subscription in
> +     each of the nodes that subcribes to data from all the nodes. The steps
> +     to create two node bidirectional replication is listed below:
> +   </para>
>
> 2.3.a Typo: "subcribes"

Modified

> 2.3.b Wording: "creating subscription” -> "and creating subscriptions..."

Modified

> 2.3.c Wording: "The steps to create two node bidirectional replication
> is listed below:" -> "The steps to create a two-node bidirectional
> replication are given below:"

Modified

> ~~~
>
> 2.4 doc/src/sgml/logical-replication.sgml
>
> +<programlisting>
> +node2=# CREATE SUBSCRIPTION sub_node1_node2
> +node2=# CONNECTION 'dbname=foo host=node1 user=repuser'
> +node2=# PUBLICATION pub_node1
> +node2=# WITH (copy_data = off, local_only = on);
> +CREATE SUBSCRIPTION
> +</programlisting>
>
> I am not sure if those psql continuation prompts are right. Shouldn't
> they be "node2-#" instead of "node2=#"?

Modified

> ~~~
>
> 2.5 doc/src/sgml/logical-replication.sgml
>
> +<programlisting>
> +node2=# CREATE SUBSCRIPTION sub_node1_node2
> +node2=# CONNECTION 'dbname=foo host=node1 user=repuser'
> +node2=# PUBLICATION pub_node1
> +node2=# WITH (copy_data = off, local_only = on);
> +CREATE SUBSCRIPTION
> +</programlisting>
> +   </para>
>
> IIUC the closing </para> should be on the same line as the
> </programlisting>. I recall there was some recent github push about
> this sometime in the last month - maybe you can check to confirm it.

 I have seen the programlisting across multiple places and noticed
that there is no such guideline being followed. I have not made any
change for this comment.

> ~~~
>
> 2.6 doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +     Create the subscription in node2 to subscribe the changes from node1:
> +<programlisting>
> +node2=# CREATE SUBSCRIPTION sub_node1_node2
>
> IMO the naming convention here is backwards/confusing. E.g. The
> "pub_node1" is the publisher at node1. So similarly, I think the
> subscriber at node2 should be called "sub_node2_node1" (not
> "sub_node1_node2")

Modified

> ~~~
>
> 2.7 doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +     Adding a new node node3 to the existing node1 and node2 requires setting
> +     up subscription in node1 and node2 to replicate the data from node3 and
> +     setting up subscription in node3 to replicate data from node1 and node2.
> +     The steps for the same is listed below:
> +   </para>
>
> There are several sections that say "The steps for the same is listed
> below:". The sentence for all of them seemed redundant to me.

Removed it.

> ~~~
>
> 2.8 doc/src/sgml/logical-replication.sgml
>
> +    <para>
> +     Adding a new node node3 to the existing node1 and node2 when data is
> +     present in existing nodes node1 and node2 needs similar steps, only change
> +     required here is that node3 should create subscription with copy_data as
> +     force to one of the existing nodes to receive the existing data during
> +     initial data synchronization. The steps for the same is listed below:
> +   </para>
>
> I thought it should be 2 sentences. So "needs similar steps, only
> change required here" -> "needs similar steps. The only change
> required here..."

Modified

> ~~~
>
> 2.9 doc/src/sgml/logical-replication.sgml
>
> I think every time you say option names or values like "copy_data" in
> the text they should be using <literal> font.

Modified

> ~~~
>
> 2.10 doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +     Adding a new node node3 to the existing node1 and node2 when data is
> +     present in the new node node3 needs similar steps, few changes are
> +     required here to get the existing data from node3 to node1 and node2 and
> +     later cleaning up of data in node3 before synchronization of all the data
> +     from the existing nodes. The steps for the same is listed below:
> +   </para>
>
> I thought it should be 2 sentences. So "needs similar steps, few
> changes are required here" -> "needs similar steps. A few changes are
> required here..."

Modified

> ~~~
>
> 2.11 doc/src/sgml/logical-replication.sgml
>
> All the text that says "create subscription" and "create publication"
> maybe should be change to "create a subscription" and "create a
> publication" etc.

Modified

> ~~~
>
> 2.12 doc/src/sgml/ref/alter_subscription.sgml - copy_data
>
> +         <para>
> +          There is some interaction between the "local_only" option and
> +          "copy_data" option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for interaction
> +          details and usage of force for copy_data option.
>           </para>
>
> 2.12.a Maybe options should not be in quotes like that. I think they
> should be <literal> font.

Modified

> 2.12.b It is a bit misleading because there is no "Notes" section here
> on this page. Maybe it should say refer to the CREATE SUBSCRIPTION
> Notes.

Modified

> 2.12.c The last copy_data should also be <literal> font.

Modified

> ~~~
>
> 2.13 doc/src/sgml/ref/create_subscription.sgml - local_only
>
>           </para>
> +         <para>
> +          There is some interaction between the "local_only" option and
> +          "copy_data" option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>          </listitem>
>
> 2.13.a Maybe options should not be in quotes like that. I think they
> should be <literal> font.

Modified

> 2.13.b The last copy_data should also be <literal> font.

Modified

> ~~~
>
> 2.14 doc/src/sgml/ref/create_subscription.sgml - copy_data
>
>
>           <para>
>            Specifies whether to copy pre-existing data in the publications
> -          that are being subscribed to when the replication starts.
> -          The default is <literal>true</literal>.
> +          that are being subscribed to when the replication starts. This
> +          parameter may be either <literal>true</literal>,
> +          <literal>false</literal> or <literal>force</literal>. The default is
> +          <literal>true</literal>.
>           </para>
>
> 2.14.a Maybe options should not be in quotes like that. I think they
> should be <literal> font.

Modified

> 2.14.b The last copy_data should also be <literal> font.

Modified

> ~~~
>
> 2.15 doc/src/sgml/ref/create_subscription.sgml
>
> @@ -374,6 +388,16 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   If subscription is created with local_only as 'on' and copy_data as 'on', it
> +   will check if the publisher tables are being subscribed to any other
> +   publisher and throw an error to prevent inconsistent data in the
> +   subscription. User can continue with the copy operation without throwing any
> +   error in this case by specifying copy_data as 'force'. Refer to the
> +   <xref linkend="bidirectional-logical-replication"/> on how
> +   copy_data and local_only can be used in bidirectional replication.
> +  </para>
>
> 2.15.a I think all those mentions of copy_data and local_only and
> their values should be in <literal> font instead of in quotes.

Modified

> 2.15.b Wording: "User can" -> "The user can"

Modified

> ~~~
>
> 2.16 src/backend/commands/subscriptioncmds.c - macro
>
> > 20. src/backend/commands/subscriptioncmds.c - IS_COPY_DATA_ON_OR_FORCE
> >
> > @@ -69,6 +69,18 @@
> >  /* check if the 'val' has 'bits' set */
> >  #define IsSet(val, bits)  (((val) & (bits)) == (bits))
> >
> > +#define IS_COPY_DATA_ON_OR_FORCE(copy_data) (copy_data != COPY_DATA_OFF)
> >
> > Maybe this would be better as a static inline function?
>
> Vignesh: What is the advantage of doing this change? I have not changed this
> as the macro usage is fine. Thoughts?
>
> Originally I was going to suggest the macro should use extra parens
> like ((copy_data) != COPY_DATA_OFF), but then I thought if it was a
> function then it would have enum type-checking which would be better.
> If you want to keep the macro then please make the parens change.

Modified to include parenthesis.

> ~~~
>
> 2.17 src/backend/commands/subscriptioncmds.c - DefGetCopyData
>
> + /*
> + * The set of strings accepted here should match up with
> + * the grammar's opt_boolean_or_string production.
> + */
> + if (pg_strcasecmp(sval, "true") == 0 ||
> + pg_strcasecmp(sval, "on") == 0)
> + return COPY_DATA_ON;
> + if (pg_strcasecmp(sval, "false") == 0 ||
> + pg_strcasecmp(sval, "off") == 0)
> + return COPY_DATA_OFF;
> + if (pg_strcasecmp(sval, "force") == 0)
> + return COPY_DATA_FORCE;
>
> I think you can change the order of these to be off/on/force, so then
> the order is consistent with the T_Integer case.

Modified

> ~~~
>
> 2.18 src/backend/commands/subscriptioncmds.c - DefGetCopyData
>
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("%s requires a boolean or \"force\"", def->defname));
> + return COPY_DATA_OFF;                                           /*
> keep compiler quiet */
>
> Excessive comment indent? Has pg_indent been run?

Modified

> ~~~
>
> 2.19 src/backend/commands/subscriptioncmds.c - AlterSubscription
>
> @@ -1236,7 +1306,8 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>   errmsg("ALTER SUBSCRIPTION ... REFRESH is not allowed for disabled
> subscriptions")));
>
>   parse_subscription_options(pstate, stmt->options,
> -    SUBOPT_COPY_DATA, &opts);
> +    SUBOPT_COPY_DATA,
> +    &opts);
>
> This is a formatting change only. Maybe it does not belong in this
> patch unless it is the result of pg_indent.

Modified

> ~~~
>
> 2.20 src/backend/commands/subscriptioncmds.c - fetch_table_list
>
> > 23. src/backend/commands/subscriptioncmds.c - long errmsg
> >
> > + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
> > + ereport(ERROR,
> > + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> > + errmsg("CREATE/ALTER SUBSCRIPTION with local_only and copy_data as
> > true is not allowed when the publisher might have replicated data,
> > table:%s.%s might have replicated data in the publisher",
> > +    nspname, relname),
> > + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
> > +
> >
> > The errmsg seems way too long for the source code. Can you use string
> > concatenation or continuation chars to wrap the message over multiple
> > lines?
>
> Vignesh: I had seen that the long error message elsewhere also is in a
> single line. I think we should keep it as it is to maintain the coding
> standard. Thoughts?
>
> OK, if you say it is already common practice then it's fine by me to
> leave it as-is.

Ok, no change done.

> ~~~
>
> 2.21 src/test/regress/expected/subscription.out - make check
>
> make check fails.
>  1 of 214 tests failed.
>
> 2.21.a It looks like maybe you did not update the expected ordering of
> some of the tests, after some minor adjustments in subscriprion.sql in
> v10. So the expected output needs to be fixed in the patch.

Modified

> 2.21.b. Suggest adding this patch to CF so that the cfbot can pick up
> such test problems earlier.

CF entry added

Thanks for the comments, the attached v11 patch has the changes for the same.

Regards,
Vignesh

  76. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-28T17:07:22Z

    On Thu, Apr 28, 2022 at 7:57 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v10-0002 (TAP tests part only)
>
> FIle: src/test/subscription/t/032_localonly.pl
>
> ======
>
> 1.
>
> +# Detach node C and clean the table contents.
> +sub detach_node_clean_table_data
> +{
>
> 1a. Maybe say "Detach node C from the node-group of (A, B, C) and
> clean the table contents from all nodes"

Modified

> 1b. Actually wondered do you need to TRUNCATE from both A and B (maybe
> only truncate 1 is OK since those nodes are still using MMC). OTOH
> maybe your explicit way makes the test simpler.

cleaning the data in all nodes to keep the test simpler

> ~~~
>
> 2.
>
> +# Subroutine for verify the data is replicated successfully.
> +sub verify_data
> +{
>
> 2a. Typo: "for verify"  -> "to verify"

Modified

> 2b. The messages in this function maybe are not very appropriate. They
> say 'Inserted successfully without leading to infinite recursion in
> circular replication setup', but really the function is only testing
> all the data is the same as 'expected'. So it could be the result of
> any operation - not just Insert.

Modified

> ~~~
>
> 3. SELECT ORDER BY?
>
> $result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full;");
> is($result, qq(11
> 12
> 13),
>    'Node_C data replicated to Node_B'
> );
>
> I am not sure are these OK like this or if *every* SELECT use ORDER BY
> to make sure the data is in the same qq expected order? There are
> multiple cases like this.
>
> (BTW, I think this comment needs to be applied for the v10-0001 patch,
> maybe not v10-0002).

Modified

> ~~~
>
> 4.
>
> +###############################################################################
> +# Specifying local_only 'on' which indicates that the publisher should only
> +# replicate the changes that are generated locally from node_B, but in
> +# this case since the node_B is also subscribing data from node_A, node_B can
> +# have data originated from node_A, so throw an error in this case to prevent
> +# node_A data being replicated to the node_C.
> +###############################################################################
>
> There is something wrong with the description because there is no
> "node_C" in this test. You are just creating a 2nd subscription on
> node A.

Modified

> ~~
>
> 5.
>
> +($result, $stdout, $stderr) = $node_A->psql(
> +       'postgres', "
> +        CREATE SUBSCRIPTION tap_sub_A3
> +        CONNECTION '$node_B_connstr application_name=$subname_AB'
> +        PUBLICATION tap_pub_B
> +        WITH (local_only = on, copy_data = on)");
> +like(
>
> It seemed strange to call this 2nd subscription "tap_sub_A3". Wouldn't
> it be better to call it "tap_sub_A2"?

Modified

> ~~~
>
> 6.
>
> +###############################################################################
> +# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B) bidirectional
> +# replication setup when the existing nodes (node_A & node_B) has no data and
> +# the new node (node_C) some pre-existing data.
> +###############################################################################
> +$node_C->safe_psql('postgres', "INSERT INTO tab_full VALUES (31);");
> +
> +$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full order by 1;");
> +is( $result, qq(), 'Check existing data');
> +
> +$result = $node_B->safe_psql('postgres', "SELECT * FROM tab_full order by 1;");
> +is( $result, qq(), 'Check existing data');
> +
> +$result = $node_C->safe_psql('postgres', "SELECT * FROM tab_full order by 1;");
> +is($result, qq(31), 'Check existing data');
> +
> +create_subscription($node_A, $node_C, $subname_AC, $node_C_connstr,
> +       'tap_pub_C', 'copy_data = force, local_only = on');
> +create_subscription($node_B, $node_C, $subname_BC, $node_C_connstr,
> +       'tap_pub_C', 'copy_data = force, local_only = on');
> +
>
> Because the Node_C does not yet have any subscriptions aren't these
> cases where you didn't really need to use "force"?

Modified

The v11 patch attached at [1] has the fixes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2hRyXc4DKPLnBgem_LHBmy%3DWXFEsm9-xhckye1YXcpPA%40mail.gmail.com

Regards,
Vignesh

  77. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-04-29T02:37:46Z

    Here are my review comments for the v11* patches.

======

v11-0001 - no more comments. LGTM

======

V11-0002

1. doc/src/sgml/logical-replication.sgml

+   <para>
+     Bidirectional replication is useful in creating multi master database
+     which helps in performing read/write operations from any of the nodes.
+     Setting up bidirectional logical replication between two nodes requires
+     creation of the publication in all the nodes, creating subscriptions in
+     each of the nodes that subscribes to data from all the nodes. The steps
+     to create a two-node bidirectional replication are given below:
+   </para>

Wording: "creating multi master database" -> "creating a multi-master database"
Wording: "creation of the publication in all the nodes" -> "creation
of a publication in all the nodes".

~~~

2. doc/src/sgml/logical-replication.sgml

> +<programlisting>
> +node2=# CREATE SUBSCRIPTION sub_node1_node2
> +node2=# CONNECTION 'dbname=foo host=node1 user=repuser'
> +node2=# PUBLICATION pub_node1
> +node2=# WITH (copy_data = off, local_only = on);
> +CREATE SUBSCRIPTION
> +</programlisting>
> + </para>
>
> IIUC the closing </para> should be on the same line as the
> </programlisting>. I recall there was some recent github push about
> this sometime in the last month - maybe you can check to confirm it.

Vignesh: I have seen the programlisting across multiple places and noticed
that there is no such guideline being followed. I have not made any
change for this comment.

FYI – I found the push [1] by PeterE that I was referring to so. I
thought we perhaps should follow this, even if not all other code is
doing so. But you can choose.

~~~

3. doc/src/sgml/logical-replication.sgml

+   <para>
+    Create a subscription in <literal>node3</literal> to subscribe the changes
+    from <literal>node1</literal> and <literal>node2</literal>, here
+    <literal>copy_data</literal> is specified as <literal>force</literal> when
+    creating a subscription to <literal>node1</literal> so that the existing
+    table data is copied during initial sync:
+<programlisting>
+node3=# CREATE SUBSCRIPTION
+node3-# sub_node3_node1 CONNECTION 'dbname=foo host=node1 user=repuser'
+node3-# PUBLICATION pub_node1
+node3-# WITH (copy_data = force, local_only = on);
+CREATE SUBSCRIPTION
+node3=# CREATE SUBSCRIPTION
+node3-# sub_node3_node2 CONNECTION 'dbname=foo host=node2 user=repuser'
+node3-# PUBLICATION pub_node2
+node3-# WITH (copy_data = off, local_only = on);
+CREATE SUBSCRIPTION
+</programlisting>

I think this part should be split into 2 separate program listings
each for the different subscriptions. Then those descriptions can be
described separately (e.g. why one is force and the other is not).
Then it will also be more consistent with how you split/explained
something similar in the previous section on this page.

------
[1] https://github.com/postgres/postgres/commit/d7ab2a9a3c0a2800ab36bb48d1cc97370067777e

Kind Regards,
Peter Smith.
Fujitsu Australia

  78. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-04-29T11:01:41Z

    On Fri, Apr 29, 2022 at 8:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the v11* patches.
>
> ======
>
> v11-0001 - no more comments. LGTM
>
> ======
>
> V11-0002
>
> 1. doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +     Bidirectional replication is useful in creating multi master database
> +     which helps in performing read/write operations from any of the nodes.
> +     Setting up bidirectional logical replication between two nodes requires
> +     creation of the publication in all the nodes, creating subscriptions in
> +     each of the nodes that subscribes to data from all the nodes. The steps
> +     to create a two-node bidirectional replication are given below:
> +   </para>
>
> Wording: "creating multi master database" -> "creating a multi-master database"
> Wording: "creation of the publication in all the nodes" -> "creation
> of a publication in all the nodes".

Modified

> ~~~
>
> 2. doc/src/sgml/logical-replication.sgml
>
> > +<programlisting>
> > +node2=# CREATE SUBSCRIPTION sub_node1_node2
> > +node2=# CONNECTION 'dbname=foo host=node1 user=repuser'
> > +node2=# PUBLICATION pub_node1
> > +node2=# WITH (copy_data = off, local_only = on);
> > +CREATE SUBSCRIPTION
> > +</programlisting>
> > + </para>
> >
> > IIUC the closing </para> should be on the same line as the
> > </programlisting>. I recall there was some recent github push about
> > this sometime in the last month - maybe you can check to confirm it.
>
> Vignesh: I have seen the programlisting across multiple places and noticed
> that there is no such guideline being followed. I have not made any
> change for this comment.
>
> FYI – I found the push [1] by PeterE that I was referring to so. I
> thought we perhaps should follow this, even if not all other code is
> doing so. But you can choose.

Modified

> ~~~
>
> 3. doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +    Create a subscription in <literal>node3</literal> to subscribe the changes
> +    from <literal>node1</literal> and <literal>node2</literal>, here
> +    <literal>copy_data</literal> is specified as <literal>force</literal> when
> +    creating a subscription to <literal>node1</literal> so that the existing
> +    table data is copied during initial sync:
> +<programlisting>
> +node3=# CREATE SUBSCRIPTION
> +node3-# sub_node3_node1 CONNECTION 'dbname=foo host=node1 user=repuser'
> +node3-# PUBLICATION pub_node1
> +node3-# WITH (copy_data = force, local_only = on);
> +CREATE SUBSCRIPTION
> +node3=# CREATE SUBSCRIPTION
> +node3-# sub_node3_node2 CONNECTION 'dbname=foo host=node2 user=repuser'
> +node3-# PUBLICATION pub_node2
> +node3-# WITH (copy_data = off, local_only = on);
> +CREATE SUBSCRIPTION
> +</programlisting>
>
> I think this part should be split into 2 separate program listings
> each for the different subscriptions. Then those descriptions can be
> described separately (e.g. why one is force and the other is not).
> Then it will also be more consistent with how you split/explained
> something similar in the previous section on this page.

Modified
The attached v12 patch has the changes for the same.

Regards,
Vignesh

  79. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-02T00:18:59Z

    Thanks for updating the patches for all my prior feedback.

For v12* I have only minor feedback for the docs.

======
V12-0001

no comments

======
V12-0002

1. Commit message

In another thread using the terminology "multi-master" seems to be
causing some problems. Maybe you should choose different wording in
this commit message to avoid having the same problems?

~~~

2. doc/src/sgml/logical-replication.sgml

These are all similar minor wording suggestions to split the sentences.
Wording: ", here copy_data is specified as XXX ..." -> ". Use
copy_data specified as XXX ..."

Also:
Wording: "... to subscribe the changes from nodeXXX..."
-> "... to subscribe to the changes from nodeXXX... " (add "to")
-> "... to subscribe to nodeXXX."  (or I preferred just remove the
whole "changes" part)

2a.
Create a subscription in node3 to subscribe the changes from node1,
here copy_data is specified as force so that the existing table data
is copied during initial sync:

SUGGESTION
Create a subscription in node3 to subscribe to node1. Use copy_data
specified as force so that the existing table data is copied during
initial sync:

2b.
Create a subscription in node1 to subscribe the changes fromnode3,
here copy_data is specified as force so that the existing table data
is copied during initial sync:

SUGGESTION
Create a subscription in node1 to subscribe to node3. Use copy_data
specified as force so that the existing table data is copied during
initial sync:

2c.
Create a subscription in node2 to subscribe the changes from node3,
here copy_data is specified as force so that the existing table data
is copied during initial sync:

SUGGESTION
Create a subscription in node2 to subscribe to node3. Use copy_data
specified as force so that the existing table data is copied during
initial sync:

2d.
Create a subscription in node3 to subscribe the changes from node1,
here copy_data is specified as force when creating a subscription to
node1 so that the existing table data is copied during initial sync:

SUGGESTION
Create a subscription in node3 to subscribe to node1. Use copy_data
specified as force when creating a subscription to node1 so that the
existing table data is copied during initial sync:

2e.
Create a subscription in node3 to subscribe the changes from node2,
here copy_data is specified as off as the initial table data would
have been copied in the earlier step:

SUGGESTION (this one has a bit more re-wording than the others)
Create a subscription in node3 to subscribe to node2. Use copy_data
specified as off because the initial table data would have been
already copied in the previous step:

~~~

3. doc/src/sgml/logical-replication.sgml

31.11.2. Adding new node when there is no data in any of the nodes
31.11.3. Adding new node when data is present in the existing nodes
31.11.4. Adding new node when data is present in the new node

Minor change to the above heading?

Wording: "Adding new node ..." -> "Adding a new node ..."

------
[1] https://www.postgresql.org/message-id/flat/CAHut%2BPuwRAoWY9pz%3DEubps3ooQCOBFiYPU9Yi%3DVB-U%2ByORU7OA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  80. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-04T06:47:00Z

    On Mon, May 2, 2022 at 5:49 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Thanks for updating the patches for all my prior feedback.
>
> For v12* I have only minor feedback for the docs.
>
> ======
> V12-0001
>
> no comments
>
> ======
> V12-0002
>
> 1. Commit message
>
> In another thread using the terminology "multi-master" seems to be
> causing some problems. Maybe you should choose different wording in
> this commit message to avoid having the same problems?

I felt, we can leave this to committer

> ~~~
>
> 2. doc/src/sgml/logical-replication.sgml
>
> These are all similar minor wording suggestions to split the sentences.
> Wording: ", here copy_data is specified as XXX ..." -> ". Use
> copy_data specified as XXX ..."
>
> Also:
> Wording: "... to subscribe the changes from nodeXXX..."
> -> "... to subscribe to the changes from nodeXXX... " (add "to")
> -> "... to subscribe to nodeXXX."  (or I preferred just remove the
> whole "changes" part)
>
> 2a.
> Create a subscription in node3 to subscribe the changes from node1,
> here copy_data is specified as force so that the existing table data
> is copied during initial sync:
>
> SUGGESTION
> Create a subscription in node3 to subscribe to node1. Use copy_data
> specified as force so that the existing table data is copied during
> initial sync:

Modified

> 2b.
> Create a subscription in node1 to subscribe the changes fromnode3,
> here copy_data is specified as force so that the existing table data
> is copied during initial sync:
>
> SUGGESTION
> Create a subscription in node1 to subscribe to node3. Use copy_data
> specified as force so that the existing table data is copied during
> initial sync:

Modified

> 2c.
> Create a subscription in node2 to subscribe the changes from node3,
> here copy_data is specified as force so that the existing table data
> is copied during initial sync:
>
> SUGGESTION
> Create a subscription in node2 to subscribe to node3. Use copy_data
> specified as force so that the existing table data is copied during
> initial sync:

Modified

> 2d.
> Create a subscription in node3 to subscribe the changes from node1,
> here copy_data is specified as force when creating a subscription to
> node1 so that the existing table data is copied during initial sync:
>
> SUGGESTION
> Create a subscription in node3 to subscribe to node1. Use copy_data
> specified as force when creating a subscription to node1 so that the
> existing table data is copied during initial sync:

Modified

> 2e.
> Create a subscription in node3 to subscribe the changes from node2,
> here copy_data is specified as off as the initial table data would
> have been copied in the earlier step:
>
> SUGGESTION (this one has a bit more re-wording than the others)
> Create a subscription in node3 to subscribe to node2. Use copy_data
> specified as off because the initial table data would have been
> already copied in the previous step:

Modified

> ~~~
>
> 3. doc/src/sgml/logical-replication.sgml
>
> 31.11.2. Adding new node when there is no data in any of the nodes
> 31.11.3. Adding new node when data is present in the existing nodes
> 31.11.4. Adding new node when data is present in the new node
>
> Minor change to the above heading?
>
> Wording: "Adding new node ..." -> "Adding a new node ..."

Modified

Thanks for the comments, the attached v13 patch has the changes for the same.

Regards,
Vignesh

  81. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-05T07:22:47Z

    Thanks for making changes from all my past reviews.

1. I checked the cfbot result is OK

2. Both patches apply cleanly in my VM

3. Patch changes
V13-0001 - No diffs; it is identical to v12-0001
V13-0002 - only changed for pg docs. So I rebuilt v13 docs and checked
the HTML rendering. Looked OK.

So I have no more review comments. LGTM.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  82. Re: Handle infinite recursion in logical replication setup

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-05-05T09:11:05Z

    On 2022-May-04, vignesh C wrote:

> On Mon, May 2, 2022 at 5:49 AM Peter Smith <smithpb2250@gmail.com> wrote:

> > 1. Commit message
> >
> > In another thread using the terminology "multi-master" seems to be
> > causing some problems. Maybe you should choose different wording in
> > this commit message to avoid having the same problems?
> 
> I felt, we can leave this to committer

Please don't.  See
https://postgr.es/m/20200615182235.x7lch5n6kcjq4aue@alap3.anarazel.de

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/

  83. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-05T11:14:14Z

    On Thu, May 5, 2022 at 2:41 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2022-May-04, vignesh C wrote:
>
> > On Mon, May 2, 2022 at 5:49 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> > > 1. Commit message
> > >
> > > In another thread using the terminology "multi-master" seems to be
> > > causing some problems. Maybe you should choose different wording in
> > > this commit message to avoid having the same problems?
> >
> > I felt, we can leave this to committer
>
> Please don't.  See
> https://postgr.es/m/20200615182235.x7lch5n6kcjq4aue@alap3.anarazel.de

In this thread changing multimaster to multi-source did not get agreed
and the changes did not get committed. The code still uses multimaster
as in [1]. I think we should keep it as "multimaster" as that is
already being used in [1].

[1] - https://www.postgresql.org/docs/current/different-replication-solutions.html

Regards,
Vignesh

  84. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-18T04:59:30Z

    On Wed, May 4, 2022 at 12:17 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Thanks for the comments, the attached v13 patch has the changes for the same.
>

Few comments on v13-0001
======================
1.
+ *
+ * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
+ * PG16.
...
@@ -477,6 +489,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
OutputPluginOptions *opt,
  else
  ctx->twophase_opt_given = true;

+ if (data->local_only && data->protocol_version <
LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support local_only, need
%d or higher",
+ data->protocol_version, LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)));

What is the need to change the protocol version for this parameter? As
per my understanding, we only need a different protocol version when
we are sending some new message or some additional information in an
existing message as we have done for the streaming/two_phase options
which doesn't seem to be the case here.

2.
@@ -29,6 +29,7 @@ typedef struct PGOutputData
  bool streaming;
  bool messages;
  bool two_phase;
+ bool local_only;
 } PGOutputData;

It seems we already have a similar option named 'only_local' in
TestDecodingData which has the exactly same functionality. So, isn't
it better to name this as 'only_local' instead of 'local_only'? Also,
let's add a test case for 'only_local' option of test_decoding.

-- 
With Regards,
Amit Kapila.

  85. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-05-18T08:10:42Z

    On Wed, May 4, 2022 2:47 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached v13 patch has the changes for the same.
> 

Thanks for your patch. Here are some comments on v13-0001 patch.

1) 
@@ -152,6 +152,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
         </listitem>
        </varlistentry>
 
+       <varlistentry>
+        <term><literal>local_only</literal> (<type>boolean</type>)</term>
+        <listitem>
+         <para>
+          Specifies whether the subscription will request the publisher to send
+          locally originated changes at the publisher node, or send any
+          publisher node changes regardless of their origin. The default is
+          <literal>false</literal>.
+         </para>
+        </listitem>
+       </varlistentry>
+
        <varlistentry>
         <term><literal>slot_name</literal> (<type>string</type>)</term>
         <listitem>

I think this change should be put after "The following parameters control the
subscription's replication behavior after it has been created", thoughts?

2)
A new column "sublocalonly" is added to pg_subscription, so maybe we need add it
to pg_subscription document, too. (in doc/src/sgml/catalogs.sgml)

3)
/*
 * Currently we always forward.
 */
static bool
pgoutput_origin_filter(LogicalDecodingContext *ctx,
					   RepOriginId origin_id)

Should we modify the comment of pgoutput_origin_filter()? It doesn't match the
code.

Regards,
Shi yu

  86. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-18T10:52:22Z

    On Wed, May 18, 2022 at 10:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, May 4, 2022 at 12:17 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v13 patch has the changes for the same.
> >
>
> Few comments on v13-0001
> ======================
>

Few comments on v13-0002
===========================
1.
The steps
+     to create a two-node bidirectional replication are given below:
+   </para>

The steps given after this will be successful only when there is no
data in any of the nodes and that is not clear by reading docs.

2.
+  <sect2 id="add-new-node-data-in-existing-node">
+   <title>Adding a new node when data is present in the existing nodes</title>
+    <para>
+     Adding a new node <literal>node3</literal> to the existing
+     <literal>node1</literal> and <literal>node2</literal> when data is present
+     in existing nodes <literal>node1</literal> and <literal>node2</literal>
+     needs similar steps. The only change required here is that
+     <literal>node3</literal> should create a subscription with
+     <literal>copy_data = force</literal> to one of the existing nodes to
+     receive the existing data during initial data synchronization.
+   </para>

I think the steps for these require the user to lock the required
tables/database (or in some other way hold the operations on required
tables) for node-2 till the time setup is complete, otherwise, node-3
might miss some data. It seems the same is also missing in the
section: "Adding a new node when data is present in the new node".

3.
+
+  <sect2 id="add-node-data-present-in-new-node">
+   <title>Adding a new node when data is present in the new node</title>
...
...

+   <para>
+    Create a subscription in <literal>node1</literal> to subscribe to
+    <literal>node3</literal>. Use <literal>copy_data</literal> specified as
+    <literal>force</literal> so that the existing table data is copied during
+    initial sync:
+<programlisting>
+node1=# CREATE SUBSCRIPTION sub_node1_node3
+node1-# CONNECTION 'dbname=foo host=node3 user=repuser'
+node1-# PUBLICATION pub_node3
+node1-# WITH (copy_data = force, local_only = on);
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+   <para>
+    Create a subscription in <literal>node2</literal> to subscribe to
+    <literal>node3</literal>. Use <literal>copy_data</literal> specified as
+    <literal>force</literal> so that the existing table data is copied during
+    initial sync:
+<programlisting>
+node2=# CREATE SUBSCRIPTION sub_node2_node3
+node2-# CONNECTION 'dbname=foo host=node3 user=repuser'
+node2-# PUBLICATION pub_node3
+node2-# WITH (copy_data = force, local_only = on);
+CREATE SUBSCRIPTION
+</programlisting></para>

Why do we need to use "copy_data = force" here? AFAIU, unless, we
create any subscription on node-3, we don't need the 'force' option.

4. We should have a generic section to explain how users can add a new
node using the new options to the existing set of nodes in all cases.
For example, say when the existing set of nodes has some data and the
new node also has some pre-existing data. I think the basic steps are
something like: a. create a required publication(s) on the new node.
(b) create subscriptions on existing nodes pointing to publication on
the new node with the local_only option as true and copy_data = on.
(c) wait for data to be copied from the new node to existing nodes.
(d) Truncate the data on the new node. (e) create subscriptions
corresponding to each of the existing publisher nodes on the new node
with local_only as true and copy_data = force for one of the nodes.
(f) One needs to ensure that there is no new activity on required
tables/database (either by locking or in some other way) in all the
nodes for which copy_data option is kept as false while creating
subscriptions in the previous step to avoid any data loss.

We also need to mention in Notes that as all operations are not
transactional, user is advised to take backup of existing data to
avoid any inconsistency.

5.
 * It is quite possible that subscriber has not yet pulled data to
+ * the tables, but in ideal cases the table data will be subscribed.
+ * To keep the code simple it is not checked if the subscriber table
+ * has pulled the data or not.
+ */
+ if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))

Sorry, but I don't understand what you intend to say by the above
comment. Can you please explain?

6. I feel we can explain the case with only two nodes in the commit
message, why do we need to use a three-node case?

-- 
With Regards,
Amit Kapila.

  87. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-05-19T08:41:54Z

    On Wed, May 4, 2022 2:47 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached v13 patch has the changes for the same.
>

Here are some comments on v13-0002 patch.

1)
+		 * Throw an error so that the user can take care of the initial data
+		 * copying and then create subscription with copy_data off.

Should "with copy_data off" be changed to "with copy_data off or force"?

2)
		case ALTER_SUBSCRIPTION_ADD_PUBLICATION:
		case ALTER_SUBSCRIPTION_DROP_PUBLICATION:
...
					/*
					 * See ALTER_SUBSCRIPTION_REFRESH for details why this is
					 * not allowed.
					 */
					if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)

I think we need some changes here, too. Should it be modified to:
if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))

Regards,
Shi yu

  88. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-20T09:37:47Z

    On Wed, May 18, 2022 at 10:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, May 4, 2022 at 12:17 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v13 patch has the changes for the same.
> >
>
> Few comments on v13-0001
> ======================
> 1.
> + *
> + * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
> + * PG16.
> ...
> @@ -477,6 +489,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
> OutputPluginOptions *opt,
>   else
>   ctx->twophase_opt_given = true;
>
> + if (data->local_only && data->protocol_version <
> LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)
> + ereport(ERROR,
> + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> + errmsg("requested proto_version=%d does not support local_only, need
> %d or higher",
> + data->protocol_version, LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)));
>
> What is the need to change the protocol version for this parameter? As
> per my understanding, we only need a different protocol version when
> we are sending some new message or some additional information in an
> existing message as we have done for the streaming/two_phase options
> which doesn't seem to be the case here.

Modified

> 2.
> @@ -29,6 +29,7 @@ typedef struct PGOutputData
>   bool streaming;
>   bool messages;
>   bool two_phase;
> + bool local_only;
>  } PGOutputData;
>
> It seems we already have a similar option named 'only_local' in
> TestDecodingData which has the exactly same functionality. So, isn't
> it better to name this as 'only_local' instead of 'local_only'? Also,
> let's add a test case for 'only_local' option of test_decoding.

Modified and added test for only_local option of test_decoding.

Thanks for the comments, the attached v14 patch has the changes for the same.

Regards,
Vignesh

  89. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-20T09:45:42Z

    On Wed, May 18, 2022 at 1:40 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Wed, May 4, 2022 2:47 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v13 patch has the changes for the same.
> >
>
> Thanks for your patch. Here are some comments on v13-0001 patch.
>
> 1)
> @@ -152,6 +152,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
>          </listitem>
>         </varlistentry>
>
> +       <varlistentry>
> +        <term><literal>local_only</literal> (<type>boolean</type>)</term>
> +        <listitem>
> +         <para>
> +          Specifies whether the subscription will request the publisher to send
> +          locally originated changes at the publisher node, or send any
> +          publisher node changes regardless of their origin. The default is
> +          <literal>false</literal>.
> +         </para>
> +        </listitem>
> +       </varlistentry>
> +
>         <varlistentry>
>          <term><literal>slot_name</literal> (<type>string</type>)</term>
>          <listitem>
>
> I think this change should be put after "The following parameters control the
> subscription's replication behavior after it has been created", thoughts?

Modified

> 2)
> A new column "sublocalonly" is added to pg_subscription, so maybe we need add it
> to pg_subscription document, too. (in doc/src/sgml/catalogs.sgml)

Modified

> 3)
> /*
>  * Currently we always forward.
>  */
> static bool
> pgoutput_origin_filter(LogicalDecodingContext *ctx,
>                                            RepOriginId origin_id)
>
> Should we modify the comment of pgoutput_origin_filter()? It doesn't match the
> code.

Modified

Thanks for the comments, the v14 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0xuYy35vOudVHBjov3fQ%3DjBRHJHKUUN9VarqO%3DYqtaxg%40mail.gmail.com

Regards,
Vignesh

  90. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-20T10:00:54Z

    On Wed, May 18, 2022 at 4:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, May 18, 2022 at 10:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, May 4, 2022 at 12:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Thanks for the comments, the attached v13 patch has the changes for the same.
> > >
> >
> > Few comments on v13-0001
> > ======================
> >
>
> Few comments on v13-0002
> ===========================
> 1.
> The steps
> +     to create a two-node bidirectional replication are given below:
> +   </para>
>
> The steps given after this will be successful only when there is no
> data in any of the nodes and that is not clear by reading docs.

Modified

> 2.
> +  <sect2 id="add-new-node-data-in-existing-node">
> +   <title>Adding a new node when data is present in the existing nodes</title>
> +    <para>
> +     Adding a new node <literal>node3</literal> to the existing
> +     <literal>node1</literal> and <literal>node2</literal> when data is present
> +     in existing nodes <literal>node1</literal> and <literal>node2</literal>
> +     needs similar steps. The only change required here is that
> +     <literal>node3</literal> should create a subscription with
> +     <literal>copy_data = force</literal> to one of the existing nodes to
> +     receive the existing data during initial data synchronization.
> +   </para>
>
> I think the steps for these require the user to lock the required
> tables/database (or in some other way hold the operations on required
> tables) for node-2 till the time setup is complete, otherwise, node-3
> might miss some data. It seems the same is also missing in the
> section: "Adding a new node when data is present in the new node".

Modified. Mentioned that lock is required on new node-3 and node-2. I
have mentioned node-3 also should be locked so that no operation is
happened in node-3, else there can be some dml operation after
truncate which is sent to node-1 and the same data is synced when
create subscription in node-3 subscribing to the publisher in node-1.

> 3.
> +
> +  <sect2 id="add-node-data-present-in-new-node">
> +   <title>Adding a new node when data is present in the new node</title>
> ...
> ...
>
> +   <para>
> +    Create a subscription in <literal>node1</literal> to subscribe to
> +    <literal>node3</literal>. Use <literal>copy_data</literal> specified as
> +    <literal>force</literal> so that the existing table data is copied during
> +    initial sync:
> +<programlisting>
> +node1=# CREATE SUBSCRIPTION sub_node1_node3
> +node1-# CONNECTION 'dbname=foo host=node3 user=repuser'
> +node1-# PUBLICATION pub_node3
> +node1-# WITH (copy_data = force, local_only = on);
> +CREATE SUBSCRIPTION
> +</programlisting></para>
> +
> +   <para>
> +    Create a subscription in <literal>node2</literal> to subscribe to
> +    <literal>node3</literal>. Use <literal>copy_data</literal> specified as
> +    <literal>force</literal> so that the existing table data is copied during
> +    initial sync:
> +<programlisting>
> +node2=# CREATE SUBSCRIPTION sub_node2_node3
> +node2-# CONNECTION 'dbname=foo host=node3 user=repuser'
> +node2-# PUBLICATION pub_node3
> +node2-# WITH (copy_data = force, local_only = on);
> +CREATE SUBSCRIPTION
> +</programlisting></para>
>
> Why do we need to use "copy_data = force" here? AFAIU, unless, we
> create any subscription on node-3, we don't need the 'force' option.

Modified to on.

> 4. We should have a generic section to explain how users can add a new
> node using the new options to the existing set of nodes in all cases.
> For example, say when the existing set of nodes has some data and the
> new node also has some pre-existing data. I think the basic steps are
> something like: a. create a required publication(s) on the new node.
> (b) create subscriptions on existing nodes pointing to publication on
> the new node with the local_only option as true and copy_data = on.
> (c) wait for data to be copied from the new node to existing nodes.
> (d) Truncate the data on the new node. (e) create subscriptions
> corresponding to each of the existing publisher nodes on the new node
> with local_only as true and copy_data = force for one of the nodes.
> (f) One needs to ensure that there is no new activity on required
> tables/database (either by locking or in some other way) in all the
> nodes for which copy_data option is kept as false while creating
> subscriptions in the previous step to avoid any data loss.

Added

> We also need to mention in Notes that as all operations are not
> transactional, user is advised to take backup of existing data to
> avoid any inconsistency.

Modified

> 5.
>  * It is quite possible that subscriber has not yet pulled data to
> + * the tables, but in ideal cases the table data will be subscribed.
> + * To keep the code simple it is not checked if the subscriber table
> + * has pulled the data or not.
> + */
> + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
>
> Sorry, but I don't understand what you intend to say by the above
> comment. Can you please explain?

When the user specifies copy_data as on, we should check if the
publisher has the publication tables being subscribed from a remote
publisher. If so throw an error as it remote origin data present.
Ex:
Node1 - pub1 for table t1 -- no data
Node2 - Sub1 subscribing to data from pub1
Node2 - pub2 for table t1 -- no data
Node3 - create subscription to Node2 with copy_data = ON

In this case even though the table does not have any remote origin
data,  as Node2 is subscribing to data from Node1, throw an error.
We throw an error irrespective of data present in Node1 or not to keep
the code simple.

> 6. I feel we can explain the case with only two nodes in the commit
> message, why do we need to use a three-node case?

Modified

Thanks for the comments, the v14 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0xuYy35vOudVHBjov3fQ%3DjBRHJHKUUN9VarqO%3DYqtaxg%40mail.gmail.com

Regards,
Vignesh

  91. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-20T10:02:11Z

    On Thu, May 19, 2022 at 2:12 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Wed, May 4, 2022 2:47 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v13 patch has the changes for the same.
> >
>
> Here are some comments on v13-0002 patch.
>
> 1)
> +                * Throw an error so that the user can take care of the initial data
> +                * copying and then create subscription with copy_data off.
>
> Should "with copy_data off" be changed to "with copy_data off or force"?

Modified

> 2)
>                 case ALTER_SUBSCRIPTION_ADD_PUBLICATION:
>                 case ALTER_SUBSCRIPTION_DROP_PUBLICATION:
> ...
>                                         /*
>                                          * See ALTER_SUBSCRIPTION_REFRESH for details why this is
>                                          * not allowed.
>                                          */
>                                         if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
>
> I think we need some changes here, too. Should it be modified to:
> if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))

Modified

Thanks for the comments, the v14 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0xuYy35vOudVHBjov3fQ%3DjBRHJHKUUN9VarqO%3DYqtaxg%40mail.gmail.com

Regards,
Vignesh

  92. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-23T04:31:12Z

    On Fri, May 20, 2022 at 3:08 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, May 18, 2022 at 10:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Few comments on v13-0001
> > ======================
> > 1.
> > + *
> > + * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
> > + * PG16.
> > ...
> > @@ -477,6 +489,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
> > OutputPluginOptions *opt,
> >   else
> >   ctx->twophase_opt_given = true;
> >
> > + if (data->local_only && data->protocol_version <
> > LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)
> > + ereport(ERROR,
> > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > + errmsg("requested proto_version=%d does not support local_only, need
> > %d or higher",
> > + data->protocol_version, LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)));
> >
> > What is the need to change the protocol version for this parameter? As
> > per my understanding, we only need a different protocol version when
> > we are sending some new message or some additional information in an
> > existing message as we have done for the streaming/two_phase options
> > which doesn't seem to be the case here.
>
> Modified
>

It seems you forgot to remove the comments after removing the code
corresponding to the above. See below.
+ *
+ * LOGICALREP_PROTO_LOCALONLY_VERSION_NUM is the minimum protocol version with
+ * support for sending only locally originated data from the publisher.
+ * Introduced in PG16.
+ *
+ * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
+ * PG16.
  */

Few other comments on 0001
========================
1.
+ * Return true if data has originated remotely when only_local option is
+ * enabled, false otherwise.

Can we slightly change the comment to:"Return true if the data source
(origin) is remote and user has requested only local data, false
otherwise."

2.
+SELECT pg_replication_origin_session_reset();
+SELECT pg_drop_replication_slot('regression_slot_only_local');
+SELECT pg_replication_origin_drop('regress_test_decoding:
regression_slot_only_local');
\ No newline at end of file

At the end of the file, there should be a new line.

3.
+       <structfield>subonlylocal</structfield> <type>bool</type>
+      </para>
+      <para>
+       If true, subscription will request the publisher to send locally
+       originated changes at the publisher node, or send any publisher node
+       changes regardless of their origin

The case where this option is not set is not clear from the
description. Can we change the description to: "If true, the
subscription will request that the publisher send locally originated
changes. False indicates that the publisher sends any changes
regardless of their origin."

4. This new option 'subonlylocal' is placed before 'substream' in docs
(catalogs.sgml) and after it in structures in pg_subscription.h. I
suggest adding it after 'subdisableonerr' in docs and
pg_subscription.h. Also, adjust other places in subscriber-side code
to place it consistently.

5.
@@ -4516,6 +4524,8 @@ getSubscriptions(Archive *fout)
  pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
  subinfo[i].subdisableonerr =
  pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
+ subinfo[i].subonlylocal =
+ pg_strdup(PQgetvalue(res, i, i_subonlylocal));

  /* Decide whether we want to dump it */
  selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -4589,6 +4599,9 @@ dumpSubscription(Archive *fout, const
SubscriptionInfo *subinfo)
  if (strcmp(subinfo->subdisableonerr, "t") == 0)
  appendPQExpBufferStr(query, ", disable_on_error = true");

+ if (strcmp(subinfo->subonlylocal, "t") == 0)
+ appendPQExpBufferStr(query, ", only_local = true");
+
  if (strcmp(subinfo->subsynccommit, "off") != 0)
  appendPQExpBuffer(query, ", synchronous_commit = %s",
fmtId(subinfo->subsynccommit));

diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1d21c2906f..ddb855fd16 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
  char    *subdisableonerr;
  char    *subsynccommit;
  char    *subpublications;
+ char    *subonlylocal;
 } SubscriptionInfo;

To keep this part of the code consistent, I think it is better to
place 'subonlylocal' after 'subdisableonerr' in SubscriptionInfo.

-- 
With Regards,
Amit Kapila.

  93. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-23T17:34:03Z

    On Mon, May 23, 2022 at 10:01 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 20, 2022 at 3:08 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, May 18, 2022 at 10:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Few comments on v13-0001
> > > ======================
> > > 1.
> > > + *
> > > + * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
> > > + * PG16.
> > > ...
> > > @@ -477,6 +489,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
> > > OutputPluginOptions *opt,
> > >   else
> > >   ctx->twophase_opt_given = true;
> > >
> > > + if (data->local_only && data->protocol_version <
> > > LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)
> > > + ereport(ERROR,
> > > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > > + errmsg("requested proto_version=%d does not support local_only, need
> > > %d or higher",
> > > + data->protocol_version, LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)));
> > >
> > > What is the need to change the protocol version for this parameter? As
> > > per my understanding, we only need a different protocol version when
> > > we are sending some new message or some additional information in an
> > > existing message as we have done for the streaming/two_phase options
> > > which doesn't seem to be the case here.
> >
> > Modified
> >
>
> It seems you forgot to remove the comments after removing the code
> corresponding to the above. See below.
> + *
> + * LOGICALREP_PROTO_LOCALONLY_VERSION_NUM is the minimum protocol version with
> + * support for sending only locally originated data from the publisher.
> + * Introduced in PG16.
> + *
> + * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
> + * PG16.
>   */

Removed it.

> Few other comments on 0001
> ========================
> 1.
> + * Return true if data has originated remotely when only_local option is
> + * enabled, false otherwise.
>
> Can we slightly change the comment to:"Return true if the data source
> (origin) is remote and user has requested only local data, false
> otherwise."

Modified

> 2.
> +SELECT pg_replication_origin_session_reset();
> +SELECT pg_drop_replication_slot('regression_slot_only_local');
> +SELECT pg_replication_origin_drop('regress_test_decoding:
> regression_slot_only_local');
> \ No newline at end of file
>
> At the end of the file, there should be a new line.

I checked the other regression sql files and found that a newline is
not added at the end of the file. I did not make any change for this.
Thoughts?

> 3.
> +       <structfield>subonlylocal</structfield> <type>bool</type>
> +      </para>
> +      <para>
> +       If true, subscription will request the publisher to send locally
> +       originated changes at the publisher node, or send any publisher node
> +       changes regardless of their origin
>
> The case where this option is not set is not clear from the
> description. Can we change the description to: "If true, the
> subscription will request that the publisher send locally originated
> changes. False indicates that the publisher sends any changes
> regardless of their origin."

Modified

> 4. This new option 'subonlylocal' is placed before 'substream' in docs
> (catalogs.sgml) and after it in structures in pg_subscription.h. I
> suggest adding it after 'subdisableonerr' in docs and
> pg_subscription.h. Also, adjust other places in subscriber-side code
> to place it consistently.

Modified

> 5.
> @@ -4516,6 +4524,8 @@ getSubscriptions(Archive *fout)
>   pg_strdup(PQgetvalue(res, i, i_subtwophasestate));
>   subinfo[i].subdisableonerr =
>   pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
> + subinfo[i].subonlylocal =
> + pg_strdup(PQgetvalue(res, i, i_subonlylocal));
>
>   /* Decide whether we want to dump it */
>   selectDumpableObject(&(subinfo[i].dobj), fout);
> @@ -4589,6 +4599,9 @@ dumpSubscription(Archive *fout, const
> SubscriptionInfo *subinfo)
>   if (strcmp(subinfo->subdisableonerr, "t") == 0)
>   appendPQExpBufferStr(query, ", disable_on_error = true");
>
> + if (strcmp(subinfo->subonlylocal, "t") == 0)
> + appendPQExpBufferStr(query, ", only_local = true");
> +
>   if (strcmp(subinfo->subsynccommit, "off") != 0)
>   appendPQExpBuffer(query, ", synchronous_commit = %s",
> fmtId(subinfo->subsynccommit));
>
> diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
> index 1d21c2906f..ddb855fd16 100644
> --- a/src/bin/pg_dump/pg_dump.h
> +++ b/src/bin/pg_dump/pg_dump.h
> @@ -661,6 +661,7 @@ typedef struct _SubscriptionInfo
>   char    *subdisableonerr;
>   char    *subsynccommit;
>   char    *subpublications;
> + char    *subonlylocal;
>  } SubscriptionInfo;
>
> To keep this part of the code consistent, I think it is better to
> place 'subonlylocal' after 'subdisableonerr' in SubscriptionInfo.

Modified

Thanks for the comments, the attached v15 patch has the fixes for the same.

Regards,
Vignesh

  94. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-05-24T02:36:48Z

    On Tue, May 24, 2022 1:34 AM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached v15 patch has the fixes for the same.
> 

Thanks for updating the patch. I have a comment on the document in 0002 patch.

@@ -300,6 +310,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
           replication from the publisher. The default is
           <literal>false</literal>.
          </para>
+         <para>
+          There is some interaction between the <literal>only_local</literal>
+          option and <literal>copy_data</literal> option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>
         </listitem>
        </varlistentry>

This change is related to "only_local" option and "copy_data" option, so I think
it should be put together with "only_local", instead of "disable_on_error".

Besides, I tested some cross version cases, pg_dump and describe command, the
results are all as expected.

Regards,
Shi yu

  95. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-25T11:55:02Z

    On Mon, May 23, 2022 at 10:01 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 20, 2022 at 3:08 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, May 18, 2022 at 10:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Few comments on v13-0001
> > > ======================
> > > 1.
> > > + *
> > > + * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
> > > + * PG16.
> > > ...
> > > @@ -477,6 +489,12 @@ pgoutput_startup(LogicalDecodingContext *ctx,
> > > OutputPluginOptions *opt,
> > >   else
> > >   ctx->twophase_opt_given = true;
> > >
> > > + if (data->local_only && data->protocol_version <
> > > LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)
> > > + ereport(ERROR,
> > > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > > + errmsg("requested proto_version=%d does not support local_only, need
> > > %d or higher",
> > > + data->protocol_version, LOGICALREP_PROTO_LOCALONLY_VERSION_NUM)));
> > >
> > > What is the need to change the protocol version for this parameter? As
> > > per my understanding, we only need a different protocol version when
> > > we are sending some new message or some additional information in an
> > > existing message as we have done for the streaming/two_phase options
> > > which doesn't seem to be the case here.
> >
> > Modified
> >
>
> It seems you forgot to remove the comments after removing the code
> corresponding to the above. See below.
> + *
> + * LOGICALREP_PROTO_LOCALONLY_VERSION_NUM is the minimum protocol version with
> + * support for sending only locally originated data from the publisher.
> + * Introduced in PG16.
> + *
> + * FIXME: LOGICALREP_PROTO_LOCALONLY_VERSION_NUM needs to be bumped to 4 in
> + * PG16.
>   */
>
> Few other comments on 0001
> ========================
> 1.
> + * Return true if data has originated remotely when only_local option is
> + * enabled, false otherwise.
>
> Can we slightly change the comment to:"Return true if the data source
> (origin) is remote and user has requested only local data, false
> otherwise."
>
> 2.
> +SELECT pg_replication_origin_session_reset();
> +SELECT pg_drop_replication_slot('regression_slot_only_local');
> +SELECT pg_replication_origin_drop('regress_test_decoding:
> regression_slot_only_local');
> \ No newline at end of file
>
> At the end of the file, there should be a new line.
>

Thanks for pointing this out, for some reason vim does not show these
new lines at the end of the file, VS Code shows the last new line. I
have added it.

The attached v16 patch has the changes for the same.

Regards,
Vignesh

  96. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-05-25T11:56:32Z

    On Tue, May 24, 2022 at 8:06 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Tue, May 24, 2022 1:34 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v15 patch has the fixes for the same.
> >
>
> Thanks for updating the patch. I have a comment on the document in 0002 patch.
>
> @@ -300,6 +310,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
>            replication from the publisher. The default is
>            <literal>false</literal>.
>           </para>
> +         <para>
> +          There is some interaction between the <literal>only_local</literal>
> +          option and <literal>copy_data</literal> option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>          </listitem>
>         </varlistentry>
>
> This change is related to "only_local" option and "copy_data" option, so I think
> it should be put together with "only_local", instead of "disable_on_error".

Modified.

The v16 patch attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2_Ytder-6C68ia%3Dm39cmknAhxf2KGkeNAtxt84MxMT3w%40mail.gmail.com

Regards,
Vignesh

  97. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-26T01:36:29Z

    Here are my review comments for patch v15-0001.

======

1. Commit message

Should this also say new test cases were added for the test_decoding plugin?

~~~

2. contrib/test_decoding/sql/replorigin.sql

@@ -119,3 +119,18 @@ SELECT data FROM
pg_logical_slot_get_changes('regression_slot_no_lsn', NULL, NUL
 SELECT pg_replication_origin_session_reset();
 SELECT pg_drop_replication_slot('regression_slot_no_lsn');
 SELECT pg_replication_origin_drop('regress_test_decoding:
regression_slot_no_lsn');
+
+-- Verify that remote origin data is not returned with only-local option
+SELECT 'init' FROM
pg_create_logical_replication_slot('regression_slot_only_local',
'test_decoding');
+SELECT pg_replication_origin_create('regress_test_decoding:
regression_slot_only_local');
+SELECT pg_replication_origin_session_setup('regress_test_decoding:
regression_slot_only_local');
+INSERT INTO origin_tbl(data) VALUES ('only_local, commit1');
+-- remote origin data returned when only-local option is not set
+SELECT data FROM
pg_logical_slot_get_changes('regression_slot_only_local', NULL, NULL,
'skip-empty-xacts', '1', 'include-xids', '0', 'only-local', '0');
+INSERT INTO origin_tbl(data) VALUES ('only_local, commit2');
+-- remote origin data not returned when only-local option is set
+SELECT data FROM
pg_logical_slot_get_changes('regression_slot_only_local', NULL, NULL,
'skip-empty-xacts', '1', 'include-xids', '0', 'only-local', '1');
+-- Clean up
+SELECT pg_replication_origin_session_reset();
+SELECT pg_drop_replication_slot('regression_slot_only_local');
+SELECT pg_replication_origin_drop('regress_test_decoding:
regression_slot_only_local');

All the new comments should consistently start with upper case.

~~~

3. doc/src/sgml/catalogs.sgml

+      <para>
+       If true, the subscription will request that the publisher send locally
+       originated changes. False indicates that the publisher sends any changes
+       regardless of their origin.
+      </para></entry>

SUGGESTION
If true, the subscription will request the publisher to only send
changes that originated locally.

~~~

4. doc/src/sgml/ref/create_subscription.sgml

+         <para>
+          Specifies whether the subscription will request the publisher to send
+          locally originated changes at the publisher node, or send any
+          publisher node changes regardless of their origin. The default is
+          <literal>false</literal>.
+         </para>

This wording should be more similar to the same information in catalogs.sgml

SUGGESTION
Specifies whether the subscription will request the publisher to only
send changes that originated locally, or to send any changes
regardless of origin.

~~~

5. src/include/replication/walreceiver.h

@@ -183,6 +183,7 @@ typedef struct
  bool streaming; /* Streaming of large transactions */
  bool twophase; /* Streaming of two-phase transactions at
  * prepare time */
+ bool only_local; /* publish only locally originated data */
  } logical;
  } proto;
 } WalRcvStreamOptions;

SUGGESTION
/* Publish only local origin data */

~~~

6. src/test/subscription/t/032_onlylocal.pl - cosmetic changes

6a.
+# Setup a bidirectional logical replication between Node_A & Node_B

SUGGESTION
"... node_A and node_B"

6b.
+is(1, 1, "Circular replication setup is complete");

SUGGESTION (or maybe saying "circular" is also OK - I wasn't sure)
"Bidirectional replication setup is complete"

6c.
+# check that bidirectional logical replication setup...

Start comment sentence with upper case.

6d.
+###############################################################################
+# check that remote data that is originated from node_C to node_B is not
+# published to node_A
+###############################################################################

SUGGESTION
Check that remote data of node_B (that originated from node_C) is not
published to node_A

6e.
+is($result, qq(11
+12
+13), 'Node_C data replicated to Node_B'
+);

SUGGESTION for message
'node_C data replicated to node_B'

6f.
+is($result, qq(11
+12), 'Remote data originated from other node is not replicated when
only_local option is ON'
+);

SUGGESTION for message
'Remote data originating from another node (not the publisher) is not
replicated when only_local option is ON'

6g.
"Circular replication setup is complete"
'Inserted successfully without leading to infinite recursion in
bidirectional replication setup'
'Inserted successfully without leading to infinite recursion in
bidirectional replication setup'
'Node_C data replicated to Node_B'
'Remote data originated from other node is not replicated when
only_local option is ON'

Why do some of the "is" messages have single quotes and others have
double quotes? Should be consistent.

~~~

7. src/test/subscription/t/032_onlylocal.pl

+my $appname_B2 = 'tap_sub_B2';
+$node_B->safe_psql(
+ 'postgres', "
+ CREATE SUBSCRIPTION tap_sub_B2
+ CONNECTION '$node_C_connstr application_name=$appname_B2'
+ PUBLICATION tap_pub_C
+ WITH (only_local = on)");
+

AFAIK the "WITH (only_local = on)" is unnecessary here. We don't care
where the node_C data came from for this test case.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  98. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-26T04:20:13Z

    On Thu, May 26, 2022 at 7:06 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for patch v15-0001.
>
> ======
>
> 1. Commit message
>
> Should this also say new test cases were added for the test_decoding plugin?
>
> ~~~

Shall we take this out as a separate test-case patch? I think it is a
good idea to do a missing test case for existing functionality.

-- 
With Regards,
Amit Kapila.

  99. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-26T04:59:41Z

    On Thu, May 26, 2022 at 2:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, May 26, 2022 at 7:06 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my review comments for patch v15-0001.
> >
> > ======
> >
> > 1. Commit message
> >
> > Should this also say new test cases were added for the test_decoding plugin?
> >
> > ~~~
>
> Shall we take this out as a separate test-case patch? I think it is a
> good idea to do a missing test case for existing functionality.
>

+1. I also think this new test could be posted separately.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  100. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-26T05:08:42Z

    On Thu, May 26, 2022 at 7:06 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> 3. doc/src/sgml/catalogs.sgml
>
> +      <para>
> +       If true, the subscription will request that the publisher send locally
> +       originated changes. False indicates that the publisher sends any changes
> +       regardless of their origin.
> +      </para></entry>
>
> SUGGESTION
> If true, the subscription will request the publisher to only send
> changes that originated locally.
>
> ~~~
>

I think it is a good idea to keep the description related to the
'False' value as without that it is slightly unclear.

> 4. doc/src/sgml/ref/create_subscription.sgml
>
> +         <para>
> +          Specifies whether the subscription will request the publisher to send
> +          locally originated changes at the publisher node, or send any
> +          publisher node changes regardless of their origin. The default is
> +          <literal>false</literal>.
> +         </para>
>
> This wording should be more similar to the same information in catalogs.sgml
>
> SUGGESTION
> Specifies whether the subscription will request the publisher to only
> send changes that originated locally, or to send any changes
> regardless of origin.
>
> ~~~
>

Here, also, let's keep the wording related to the default value.

-- 
With Regards,
Amit Kapila.

  101. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-26T05:38:20Z

    On Thu, May 26, 2022 at 3:08 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, May 26, 2022 at 7:06 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> >
> > 3. doc/src/sgml/catalogs.sgml
> >
> > +      <para>
> > +       If true, the subscription will request that the publisher send locally
> > +       originated changes. False indicates that the publisher sends any changes
> > +       regardless of their origin.
> > +      </para></entry>
> >
> > SUGGESTION
> > If true, the subscription will request the publisher to only send
> > changes that originated locally.
> >
> > ~~~
> >
>
> I think it is a good idea to keep the description related to the
> 'False' value as without that it is slightly unclear.
>
> > 4. doc/src/sgml/ref/create_subscription.sgml
> >
> > +         <para>
> > +          Specifies whether the subscription will request the publisher to send
> > +          locally originated changes at the publisher node, or send any
> > +          publisher node changes regardless of their origin. The default is
> > +          <literal>false</literal>.
> > +         </para>
> >
> > This wording should be more similar to the same information in catalogs.sgml
> >
> > SUGGESTION
> > Specifies whether the subscription will request the publisher to only
> > send changes that originated locally, or to send any changes
> > regardless of origin.
> >
> > ~~~
> >
>
> Here, also, let's keep the wording related to the default value.
>
> --

I agree. Probably I was not being clear enough with those review comments.

My suggestions were only meant as substitutes for the FIRST sentences
of those patch fragments, not the entire paragraph.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  102. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-26T08:36:41Z

    This post completes my review of patch v15*.

======

1. A general comment affecting both patches

I realised the things defined in the subscription WITH are referred to
in the PG DOCS as "parameters" (not "options"). See CREATE
SUBSCRIPTION [1]. This is already reflected in some of my review
comments for the v15-0002 below., but I'd already posted my v15-0001
review comments before noticing this.

Please also go through the v15-0001 patch to make all the necessary
terminology changes (including in the v15-0001 commit message).

=====

Please find below review comments for patch v15-0002.

2. Commit message

This patch does couple of things:
change 1) Added force option for copy_data.
change 2) Check and throw an error if the publication tables were also
subscribing data in the publisher from other publishers.

--

2a.
"couple" -> "a couple"
"Added force" -> "Adds force ..."
"Check and throw" -> "Checks and throws ..."

2b.
Isn't "Change 2)" only applicable when copy_data and only_local are
on? It should say so here.

~~~

3. Commit message

node1: Table t1 (c1 int) has data 1, 2, 3, 4
node2: Table t1 (c1 int) has data 5, 6, 7, 8

--

I feel these kinds of replication examples are always much easier to
visualize when the data somehow indicates where it came from. E.g.
consider using different data in your example like:

node1: Table t1 (c1 int) has data 11, 12, 13, 14
node2: Table t1 (c1 int) has data 21, 22, 23, 24

~~~

4. Commit message

Examples in this message exceed 80 chars. Please wrap the SQL appropriately

~~~

5. Commit message

After this the data will be something like this:
node1:
1, 2, 3, 4, 5, 6, 7, 8

node2:
1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8

So, you can see that data on node2 (5, 6, 7, 8) is duplicated.

--

Yeah, but is that entirely correct though? AFAIK the scenario
described by this example is going to recurse forever.

~~~

6. Commit message

This problem can be solved by using only_local and copy_data option as
given below

--

"option" -> "parameters"

~~~

7. Commit message

There are other minor review comments but it's too messy to report in
detail here. I suggest checking all the ":" (some were missing) and
the letter case of/within the sentences. I also suggest
cutting/pasting all this text into MSWord or some grammar checker to
correct anything else reported.

~~~

8. Commit message

CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '<node1 details>'
PUBLICATION pub_node1 WITH (copy_data = force, only_local = on);
ERROR:  CREATE/ALTER SUBSCRIPTION with only_local and copy_data as
true is not allowed when the publisher might have replicated data

--

Looks like a typo: e.g. If copy_data is "force" then how is the error happening?

~~~

9. Commit message

The examples seem confused/muddled in some way IMO.

IIUC the new ERROR is like a new integrity check for when the
subscription is using the only_local = on, along with copy_data = on.

And so then the copy_data = force was added as a way to override that
previous error being reported.

So unless you know about the possible error then the "force" option
makes no sense. So I think you need to change the order of the
explanations in the commit message to describe the new error first.

======

10. doc/src/sgml/logical-replication.sgml

+
+ <sect1 id="bidirectional-logical-replication">
+  <title>Bidirectional logical replication</title>
+
+  <sect2 id="setting-bidirectional-replication-two-nodes">
+   <title>Setting bidirectional replication between two nodes</title>
+   <para>
+    Bidirectional replication is useful in creating a multi-master database
+    which helps in performing read/write operations from any of the nodes.
+    Setting up bidirectional logical replication between two nodes requires
+    creation of a publication in all the nodes, creating subscriptions in
+    each of the nodes that subscribes to data from all the nodes. The steps
+    to create a two-node bidirectional replication when there is no data in
+    both the nodes are given below:
+   </para>

This description is confusing. It seems to be trying to say something
about a multi-master system (e.g. you say "all the nodes") yet this is
all written inside the section about "... two nodes", so saying "all
the nodes" makes no sense here.

I think you need to split this information and put some kind of blurb
text at the top level (section 31.11), and then this section (for "two
nodes") should ONLY be talking about the case when there are just TWO
nodes.

~~~

11. doc/src/sgml/logical-replication.sgml

+   <para>
+    Lock the required tables in <literal>node1</literal> and
+    <literal>node2</literal> till the setup is completed.
+   </para>

"till" -> "until" (make this same change in multiple places)

~~~

12. doc/src/sgml/logical-replication.sgml

+  <sect2 id="add-new-node-data-in-existing-node">
+   <title>Adding a new node when data is present in the existing nodes</title>
+    <para>
+     Adding a new node <literal>node3</literal> to the existing
+     <literal>node1</literal> and <literal>node2</literal> when data is present
+     in existing nodes <literal>node1</literal> and <literal>node2</literal>
+     needs similar steps. The only change required here is that
+     <literal>node3</literal> should create a subscription with
+     <literal>copy_data = force</literal> to one of the existing nodes to
+     receive the existing data during initial data synchronization.
+   </para>

I think perhaps this should clarify that there is NO data in node3 for
this example.

~~~

13. doc/src/sgml/logical-replication.sgml - section 31.11.3

+   <para>
+    Lock the required tables in <literal>node2</literal> and
+    <literal>node3</literal> till the setup is completed.
+   </para>

Should you describe why you say no lock is required for tables on
node1? (Similar information is in section 31.11.4 also)

~~~

14. doc/src/sgml/logical-replication.sgml - section 31.11.3

+   <para>
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node2</literal>:
+<programlisting>
+node3=# CREATE SUBSCRIPTION sub_node3_node2
+node3-# CONNECTION 'dbname=foo host=node2 user=repuser'
+node3-# PUBLICATION pub_node2
+node3-# WITH (copy_data = off, only_local = on);
+CREATE SUBSCRIPTION
+</programlisting></para>

Using a similar description as in section 31.11.4 this should probably
also say something like:
"Use copy_data specified as off because the initial table data would
have been already copied in the previous step".

~~~

15. doc/src/sgml/logical-replication.sgml - section 31.11.4

+  <sect2 id="add-node-data-present-in-new-node">
+   <title>Adding a new node when data is present in the new node</title>
+   <para>
+     Adding a new node <literal>node3</literal> to the existing
+     <literal>node1</literal> and <literal>node2</literal> when data is present
+     in the new node <literal>node3</literal> needs similar steps. A
few changes
+     are required here to get the existing data from <literal>node3</literal>
+     to <literal>node1</literal> and <literal>node2</literal> and later
+     cleaning up of data in <literal>node3</literal> before synchronization of
+     all the data from the existing nodes.
+   </para>

I think perhaps this should clarify that there is NO data in node1 or
node2 for this example.

~~~

16. doc/src/sgml/logical-replication.sgml

+  <sect2 id="generic-steps-add-new-node">
+   <title>Generic steps to add a new node to the existing set of nodes</title>

I think all the steps in this section should be numbered because the
order is important.

~~~

17. doc/src/sgml/logical-replication.sgml

+  <sect2 id="generic-steps-add-new-node">
+   <title>Generic steps to add a new node to the existing set of nodes</title>

IIUC the word "parameter" should replace the word "option" in all the
text of this section.

~~~

18. doc/src/sgml/logical-replication.sgml - Notes

+  <sect2>
+   <title>Notes</title>
+   <para>
+    Setting up bidirectional logical replication across nodes requires multiple
+    steps to be performed on various nodes, as all operations are not
+    transactional, user is advised to take backup of existing data to avoid any
+    inconsistency.
+   </para>
+  </sect2>

18a.
I thought this "Notes" section should be rendered more like an sgml
note or a warning. E.g. like the warning on this page [2].

18b.
SUGGESTION (split the para into multiple sentences). e.g.
Setting up bidirectional logical replication across nodes requires
multiple steps to be performed on various nodes. Because not all
operations are transactional, the user is advised to take backups.

======

19. doc/src/sgml/ref/alter_subscription.sgml

+         <para>
+          There is some interaction between the <literal>only_local</literal>
+          option and <literal>copy_data</literal> option. Refer to the
+          <command>CREATE SUBSCRIPTION</command>
+          <xref linkend="sql-createsubscription-notes" /> for interaction
+          details and usage of <literal>force</literal> for
+          <literal>copy_data</literal> option.
          </para>

"option" -> "parameter" (3x)

======

20. doc/src/sgml/ref/create_subscription.sgml

+         <para>
+          There is some interaction between the <literal>only_local</literal>
+          option and <literal>copy_data</literal> option. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for interaction
+          details and usage of <literal>force</literal> for
+          <literal>copy_data</literal> option.
+         </para>

"option" -> "parameter" (3x)

======

21. src/backend/commands/subscriptioncmds.c

@@ -69,6 +69,18 @@
 /* check if the 'val' has 'bits' set */
 #define IsSet(val, bits)  (((val) & (bits)) == (bits))

+#define IS_COPY_DATA_ON_OR_FORCE(copy_data) ((copy_data) != COPY_DATA_OFF)
+
+/*
+ * Represents whether copy_data option is specified with off, on or force.
+ */
+typedef enum CopyData
+{
+ COPY_DATA_OFF,
+ COPY_DATA_ON,
+ COPY_DATA_FORCE
+} CopyData;
+

21a.
"option" -> "parameter"

21b.
I think the new #define is redundant and can be removed (see the next
review comment)

21c.
I think you set the OFF enum value like:
typedef enum CopyData
{
 COPY_DATA_OFF = 0,
 COPY_DATA_ON,
 COPY_DATA_FORCE
} CopyData;

then it will greatly simplify things; many changes in this file will
be unnecessary (see subsequent review comments)

~~~

22. src/backend/commands/subscriptioncmds.c

+/*
+ * Validate the value specified for copy_data option.
+ */
+static CopyData
+DefGetCopyData(DefElem *def)

"option" -> "parameter"

~~~

23. src/backend/commands/subscriptioncmds.c

+ /*
+ * Allow 0, 1, "true", "false", "on", "off" or "force".
+ */
+ switch (nodeTag(def->arg))
+ {
+ case T_Integer:
+ switch (intVal(def->arg))
+ {
+ case 0:
+ return COPY_DATA_OFF;
+ case 1:
+ return COPY_DATA_ON;
+ default:
+ /* otherwise, error out below */
+ break;
+ }
+ break;

What about also allowing copy_data = 2, and making it equivalent to "force"?

~~~

24. src/backend/commands/subscriptioncmds.c

@@ -333,17 +400,17 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
  errmsg("%s and %s are mutually exclusive options",
  "connect = false", "create_slot = true")));

- if (opts->copy_data &&
+ if (IS_COPY_DATA_ON_OR_FORCE(opts->copy_data) &&
  IsSet(opts->specified_opts, SUBOPT_COPY_DATA))

I think this change would not be needed if you set the OFF enum = 0.

~~~
25. src/backend/commands/subscriptioncmds.c

@@ -671,13 +738,14 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
  * Set sync state based on if we were asked to do data copy or
  * not.
  */
- table_state = opts.copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY;
+ table_state = IS_COPY_DATA_ON_OR_FORCE(opts.copy_data) ?
SUBREL_STATE_INIT : SUBREL_STATE_READY;

I think this change would not be needed if you set the OFF enum = 0.

~~~

26. src/backend/commands/subscriptioncmds.c

@@ -720,7 +788,8 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
  * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
  * PUBLICATION to work.
  */
- if (opts.twophase && !opts.copy_data && tables != NIL)
+ if (opts.twophase && opts.copy_data == COPY_DATA_OFF &&
+ tables != NIL)
  twophase_enabled = true;

I think this change would not be needed if you set the OFF enum = 0.

~~~

27. src/backend/commands/subscriptioncmds.c

@@ -851,7 +921,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
  list_length(subrel_states), sizeof(Oid), oid_cmp))
  {
  AddSubscriptionRelState(sub->oid, relid,
- copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
+ IS_COPY_DATA_ON_OR_FORCE(copy_data) ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
  InvalidXLogRecPtr);

I think this change would not be needed if you set the OFF enum = 0.


~~~
28. src/backend/commands/subscriptioncmds.c

@@ -1157,7 +1227,7 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
  * See ALTER_SUBSCRIPTION_REFRESH for details why this is
  * not allowed.
  */
- if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))
  ereport(ERROR,
  (errcode(ERRCODE_SYNTAX_ERROR),
  errmsg("ALTER SUBSCRIPTION with refresh and copy_data is not allowed
when two_phase is enabled"),

I think this change would not be needed if you set the OFF enum = 0.

~~~

29. src/backend/commands/subscriptioncmds.c

@@ -1209,7 +1279,7 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
  * See ALTER_SUBSCRIPTION_REFRESH for details why this is
  * not allowed.
  */
- if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))
  ereport(ERROR,
  (errcode(ERRCODE_SYNTAX_ERROR),
  errmsg("ALTER SUBSCRIPTION with refresh and copy_data is not allowed
when two_phase is enabled"),

I think this change would not be needed if you set the OFF enum = 0.

~~~

30. src/backend/commands/subscriptioncmds.c

@@ -1255,7 +1325,8 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
  *
  * For more details see comments atop worker.c.
  */
- if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
+ IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))
  ereport(ERROR,
  (errcode(ERRCODE_SYNTAX_ERROR),
  errmsg("ALTER SUBSCRIPTION ... REFRESH with copy_data is not allowed
when two_phase is enabled"),

I think this change would not be needed if you set the OFF enum = 0.

~~~

31. src/backend/commands/subscriptioncmds.c

+ appendStringInfoString(&cmd,
+    "SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename,
PS.srrelid as replicated\n"
+    "FROM pg_publication P,\n"
+    "LATERAL pg_get_publication_tables(P.pubname) GPT\n"
+    "LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
+    "pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
+    "WHERE C.oid = GPT.relid AND P.pubname in (");

use upper case "in" -> "IN"

======

32.  src/test/regress/sql/subscription.sql

@@ -40,6 +40,9 @@ SET SESSION AUTHORIZATION 'regress_subscription_user';

 -- fail - invalid option combinations
 CREATE SUBSCRIPTION regress_testsub2 CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
false, copy_data = true);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
false, copy_data = on);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
false, copy_data = 1);
+CREATE SUBSCRIPTION regress_testsub2 CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
false, copy_data = force);

32a.
The test for "copy_data = 1" should confirm it is the same as "on".

32b.
Should also test copy_data = 2 (and confirm it is the same as "force").

======

33. src/test/subscription/t/032_localonly.pl - cosmetic changes

33a.
+# Detach node C from the node-group of (A, B, C) and clean the table contents
+# from all nodes.

SUGGESTION
Detach node_C from the node-group of (node_A, node_B, node_C) and ...

33b.
+# Subroutine to create subscription and wait till the initial sync is
completed.

"till" -> "until"

33c.
+# Subroutine to create subscription and wait till the initial sync is
completed.
+# Subroutine expects subscriber node, publisher node, subscription name,
+# destination connection string, publication name and the subscription with
+# options to be passed as input parameters.
+sub create_subscription
+{
+ my ($node_subscriber, $node_publisher, $sub_name, $node_connstr,
+ $pub_name, $with_options)
+   = @_;

"subscription with options" => "subscription parameters"
"$with_options" -> "$sub_params"

33d.
+# Specifying only_local 'on' which indicates that the publisher should only

"Specifying" => "Specify"

------
[1] https://www.postgresql.org/docs/15/sql-createsubscription.html
[2] https://www.postgresql.org/docs/current/bgworker.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  103. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-26T11:10:22Z

    On Fri, May 20, 2022 at 3:31 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, May 18, 2022 at 4:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 5.
> >  * It is quite possible that subscriber has not yet pulled data to
> > + * the tables, but in ideal cases the table data will be subscribed.
> > + * To keep the code simple it is not checked if the subscriber table
> > + * has pulled the data or not.
> > + */
> > + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
> >
> > Sorry, but I don't understand what you intend to say by the above
> > comment. Can you please explain?
>
> When the user specifies copy_data as on, we should check if the
> publisher has the publication tables being subscribed from a remote
> publisher. If so throw an error as it remote origin data present.
> Ex:
> Node1 - pub1 for table t1 -- no data
> Node2 - Sub1 subscribing to data from pub1
> Node2 - pub2 for table t1 -- no data
> Node3 - create subscription to Node2 with copy_data = ON
>
> In this case even though the table does not have any remote origin
> data,  as Node2 is subscribing to data from Node1, throw an error.
> We throw an error irrespective of data present in Node1 or not to keep
> the code simple.
>

I think we can change the contents of comments something like: "Throw
an error if the publisher has subscribed to the same table from some
other publisher. We cannot differentiate between the local and
non-local data that is present in the HEAP during the initial sync.
Identification of local data can be done only from the WAL by using
the origin id.  XXX: For simplicity, we don't check whether the table
has any data or not. If the table doesn't have any data then we don't
need to distinguish between local and non-local data so we can avoid
throwing error in that case."

Few more comments:
==================
Patch 0002
======
1.
+ if (copydata == COPY_DATA_ON && only_local && !slot_attisnull(slot, 3))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("CREATE/ALTER SUBSCRIPTION with only_local and copy_data as
true is not allowed when the publisher might have replicated data,
table:%s.%s might have replicated data in the publisher",
+    nspname, relname),
+ errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));

Can we split the error message? errmsg: table:%s.%s has replicated
data in the publisher; errdetail:CREATE/ALTER SUBSCRIPTION with
only_local and copy_data as true is not allowed when the publisher has
replicated data

2.
+   <para>
+    Lock the required tables in the new node until the setup is complete.
+   </para>
+   <para>
+    Create subscriptions on existing nodes pointing to publication on
+    the new node with <literal>only_local</literal> option specified as
+    <literal>on</literal> and <literal>copy_data</literal> specified as
+    <literal>on</literal>.
+   </para>
+   <para>
+    Wait for data to be copied from the new node to existing nodes.
+   </para>
+   <para>
+    Alter the publication in new node so that the truncate operation is not
+    replicated to the subscribers.
+   </para>

Here and at other places, we should specify that the lock mode should
to acquire the lock on table should be EXCLUSIVE so that no concurrent
DML is allowed on it. Also, it is better if somewhere we explain why
and which nodes need locks?

Patch 0001:
==========
1.
+$node_A->append_conf(
+ 'postgresql.conf', qq(
+max_prepared_transactions = 10
+logical_decoding_work_mem = 64kB
+));

I don't see why you need to set these parameters. There is no test
case that needs these parameters. Please remove these from here and
all other similar places in 032_onlylocal.pl.

-- 
With Regards,
Amit Kapila.

  104. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-05-27T02:24:02Z

    On Wed, Apr 6, 2022 at 9:36 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Apr 5, 2022 at 7:06 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Below are some other name ideas. Maybe they are not improvements, but
> > > it might help other people to come up with something better.
> > >
> > > subscribe_publocal_only = true/false
> > > origin = pub_only/any
> > > adjacent_data_only = true/false
> > > direct_subscriptions_only = true/false
> > > ...
> >
> > FWIW, The subscriber wants "changes originated on publisher". From
> > that angle origin = publisher/any looks attractive. It also leaves
> > open the possibility that the subscriber may ask changes from a set of
> > origins or even non-publisher origins.
> >
>
> So, how are you imagining extending it for multiple origins? I think
> we can have multiple origins data on a node when there are multiple
> subscriptions pointing to the different or same node. The origin names
> are internally generated names but are present in
> pg_replication_origin. So, are we going to ask the user to check that
> table and specify it as an option? But, note, that the user may not be
> able to immediately recognize which origin data is useful for her.
>

I still don't have a very clear answer for the usability aspect but it
seems this was discussed in PGCon-Unconference [1] (Section: Origin
filter) and there also it was suggested to allow users to specify
multiple origin names. So, probably Ashutosh's idea makes sense and we
should use "origin = publisher/any or origin=local/any". Among these,
I would prefer later.

Note: I don't have email-id of Julian Markwort who has raised this
topic in Unconference. Euler, if you know, it might be better to add
him here, so that he is aware of what is going on and share his
opinion if he wishes to.

[1] - https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference

-- 
With Regards,
Amit Kapila.

  105. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-27T05:25:36Z

    Here are some more review comments for v15-0002 I found while working
through the documented examples. These are all for
doc/src/sgml/logical-replication.sgml. The section numbers (e.g.
31.11.1) are the assigned numbers after HTML rendering.

The general comment is that the sections appeared at first to be
independent of each other, but they are not really - sometimes
existing data and existing pub/sub are assumed to remain. Therefore, I
think some more text needs to be added to clarify the initial state
for each of these sections.

======

1. Section id.

<sect1 id="bidirectional-logical-replication">

IMO this section ID should be renamed
"logical-replication-bidirectional" then it will be more consistent
filenames with all the other logical replication section names.

~~~

2. Section 31.11.2. Adding a new node when there is no data in any of the nodes

2a.
This seems like a continuation of 31.11.1 because pub/sub of
node1,node2 is assumed to exist still.

2b.
The bottom of this section should say after these steps any subsequent
incremental data changes on any node will be replicated to all nodes
(node1, node2, node3)

~~~

3. Section 31.11.3. Adding a new node when data is present in the existing nodes

3a.
This seems to be a continuation of 31.11.1 because pub/sub (and
initial data) of node1/node2 is assumed to exist.

3b.

The bottom of this section should say after these steps that any
initial data of node1/node2 will be seen in node3, and any subsequent
incremental data changes on any node will be replicated to all nodes
(node1, node2, node3)

~~~

4. Section 31.11.4. Adding a new node when data is present in the new node

4a.
This seems to be a continuation of 31.11.1 because pub/sub (and
initial data) of node1/node2 is assumed to exist.

4b.
It is not made very clear up-front if the tables on node1 and node2
are empty or not. Apparently, they are considered NOT empty because
later in the example you are using "force" when you create the
subscription on node3.

4c.
The SQL wrapping here is strangely different from others (put the
subscription name on 1st line)
node3=# CREATE SUBSCRIPTION
node3-# sub_node3_node1 CONNECTION 'dbname=foo host=node1 user=repuser'
node3-# PUBLICATION pub_node1
node3-# WITH (copy_data = force, only_local = on);
CREATE SUBSCRIPTION

4d.
The SQL wrapping here is strangely different from others (put the
subscription name on 1st line)
node3=# CREATE SUBSCRIPTION
node3-# sub_node3_node2 CONNECTION 'dbname=foo host=node2 user=repuser'
node3-# PUBLICATION pub_node2
node3-# WITH (copy_data = off, only_local = on);
CREATE SUBSCRIPTION

4e.
The bottom of this section should say after this that any initial data
of node1/node2 will be seen in node3, and any subsequent incremental
data changes on any node will be replicated to all nodes (node1,
node2, node3)

~~~

5. Section 31.11.5. Generic steps to add a new node to the existing set of nodes

5a
Create subscriptions on the new node pointing to publication on the
first node with only_local option specified as on and copy_data option
specified as "force".

-> that should say "Create a subscription" (singular)

5b.
Create subscriptions on the new node pointing to publications on the
remaining node with only_local option specified as on and copy_data
option specified as off.

-> that should say "on the remaining node" (plural)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  106. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-05-27T07:04:18Z

    On Wed, May 25, 2022 7:55 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> The attached v16 patch has the changes for the same.
> 

Thanks for updating the patch.

Some comments for the document in 0002 patch.

1.
+   <para>
+    Lock the required tables in <literal>node1</literal> and
+    <literal>node2</literal> till the setup is completed.
+   </para>
+
+   <para>
+    Create a publication in <literal>node1</literal>:
+<programlisting>
+node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1;
+CREATE PUBLICATION
+</programlisting></para>

If the table is locked in the very beginning, we will not be able to create the
publication (because the locks have conflict). Maybe we should switch the order
of creating publication and locking tables here.

2.
In the case of "Adding a new node when data is present in the new node", we need
to truncate table t1 in node3, but the truncate operation would be blocked
because the table has be locked before. Maybe we need some changes for it.

Regards,
Shi yu

  107. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-05-27T08:38:25Z

    On Fri, May 27, 2022 at 5:04 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Wed, May 25, 2022 7:55 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v16 patch has the changes for the same.
> >
>
> Thanks for updating the patch.
>
> Some comments for the document in 0002 patch.
>
> 1.
> +   <para>
> +    Lock the required tables in <literal>node1</literal> and
> +    <literal>node2</literal> till the setup is completed.
> +   </para>
> +
> +   <para>
> +    Create a publication in <literal>node1</literal>:
> +<programlisting>
> +node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1;
> +CREATE PUBLICATION
> +</programlisting></para>
>
> If the table is locked in the very beginning, we will not be able to create the
> publication (because the locks have conflict). Maybe we should switch the order
> of creating publication and locking tables here.
>

I agree. It seems some of the locking instructions in the earlier
sections 31.11.1 - 31.11.4 are contradictory to the later "generic"
steps given in "31.11.5. Generic steps to add a new node to the
existing set of nodes". I'm assuming the generic steps are the
"correct" steps

e.g. generic steps say get the lock on new node tables AFTER the
publication of new node.
e.g. generic steps say do NOT get a table lock on the node one you are
(first) joining to.

~~~

Furthermore, the generic steps are describing attaching a new N+1th
node to some (1 ... N) other nodes.

So I think in the TWO-node case (section 31.11.1) node2 should be
treated as the "new" node that you are attaching to the "first" node1.
IMO the section 31.11.1 example should be reversed so that it obeys
the "generic" pattern.
e.g. It should be doing the CREATE PUBLICATION pub_node2 first (since
that is the "new" node)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  108. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:38:40Z

    On Thu, May 26, 2022 at 7:06 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for patch v15-0001.
>
> ======
>
> 1. Commit message
>
> Should this also say new test cases were added for the test_decoding plugin?

Separated the test as a 0001 patch.

> ~~~
>
> 2. contrib/test_decoding/sql/replorigin.sql
>
> @@ -119,3 +119,18 @@ SELECT data FROM
> pg_logical_slot_get_changes('regression_slot_no_lsn', NULL, NUL
>  SELECT pg_replication_origin_session_reset();
>  SELECT pg_drop_replication_slot('regression_slot_no_lsn');
>  SELECT pg_replication_origin_drop('regress_test_decoding:
> regression_slot_no_lsn');
> +
> +-- Verify that remote origin data is not returned with only-local option
> +SELECT 'init' FROM
> pg_create_logical_replication_slot('regression_slot_only_local',
> 'test_decoding');
> +SELECT pg_replication_origin_create('regress_test_decoding:
> regression_slot_only_local');
> +SELECT pg_replication_origin_session_setup('regress_test_decoding:
> regression_slot_only_local');
> +INSERT INTO origin_tbl(data) VALUES ('only_local, commit1');
> +-- remote origin data returned when only-local option is not set
> +SELECT data FROM
> pg_logical_slot_get_changes('regression_slot_only_local', NULL, NULL,
> 'skip-empty-xacts', '1', 'include-xids', '0', 'only-local', '0');
> +INSERT INTO origin_tbl(data) VALUES ('only_local, commit2');
> +-- remote origin data not returned when only-local option is set
> +SELECT data FROM
> pg_logical_slot_get_changes('regression_slot_only_local', NULL, NULL,
> 'skip-empty-xacts', '1', 'include-xids', '0', 'only-local', '1');
> +-- Clean up
> +SELECT pg_replication_origin_session_reset();
> +SELECT pg_drop_replication_slot('regression_slot_only_local');
> +SELECT pg_replication_origin_drop('regress_test_decoding:
> regression_slot_only_local');
>
> All the new comments should consistently start with upper case.

Modified

> ~~~
>
> 3. doc/src/sgml/catalogs.sgml
>
> +      <para>
> +       If true, the subscription will request that the publisher send locally
> +       originated changes. False indicates that the publisher sends any changes
> +       regardless of their origin.
> +      </para></entry>
>
> SUGGESTION
> If true, the subscription will request the publisher to only send
> changes that originated locally.

Modified

> ~~~
>
> 4. doc/src/sgml/ref/create_subscription.sgml
>
> +         <para>
> +          Specifies whether the subscription will request the publisher to send
> +          locally originated changes at the publisher node, or send any
> +          publisher node changes regardless of their origin. The default is
> +          <literal>false</literal>.
> +         </para>
>
> This wording should be more similar to the same information in catalogs.sgml
>
> SUGGESTION
> Specifies whether the subscription will request the publisher to only
> send changes that originated locally, or to send any changes
> regardless of origin.

Modified

> ~~~
>
> 5. src/include/replication/walreceiver.h
>
> @@ -183,6 +183,7 @@ typedef struct
>   bool streaming; /* Streaming of large transactions */
>   bool twophase; /* Streaming of two-phase transactions at
>   * prepare time */
> + bool only_local; /* publish only locally originated data */
>   } logical;
>   } proto;
>  } WalRcvStreamOptions;
>
> SUGGESTION
> /* Publish only local origin data */

Modified

> ~~~
>
> 6. src/test/subscription/t/032_onlylocal.pl - cosmetic changes
>
> 6a.
> +# Setup a bidirectional logical replication between Node_A & Node_B
>
> SUGGESTION
> "... node_A and node_B"

Modified

> 6b.
> +is(1, 1, "Circular replication setup is complete");
>
> SUGGESTION (or maybe saying "circular" is also OK - I wasn't sure)
> "Bidirectional replication setup is complete"

Modified

> 6c.
> +# check that bidirectional logical replication setup...
>
> Start comment sentence with upper case.

Modified

> 6d.
> +###############################################################################
> +# check that remote data that is originated from node_C to node_B is not
> +# published to node_A
> +###############################################################################
>
> SUGGESTION
> Check that remote data of node_B (that originated from node_C) is not
> published to node_A

Modified

> 6e.
> +is($result, qq(11
> +12
> +13), 'Node_C data replicated to Node_B'
> +);
>
> SUGGESTION for message
> 'node_C data replicated to node_B'

Modified

> 6f.
> +is($result, qq(11
> +12), 'Remote data originated from other node is not replicated when
> only_local option is ON'
> +);
>
> SUGGESTION for message
> 'Remote data originating from another node (not the publisher) is not
> replicated when only_local option is ON'

Modified

> 6g.
> "Circular replication setup is complete"
> 'Inserted successfully without leading to infinite recursion in
> bidirectional replication setup'
> 'Inserted successfully without leading to infinite recursion in
> bidirectional replication setup'
> 'Node_C data replicated to Node_B'
> 'Remote data originated from other node is not replicated when
> only_local option is ON'
>
> Why do some of the "is" messages have single quotes and others have
> double quotes? Should be consistent.

Modified

> ~~~
>
> 7. src/test/subscription/t/032_onlylocal.pl
>
> +my $appname_B2 = 'tap_sub_B2';
> +$node_B->safe_psql(
> + 'postgres', "
> + CREATE SUBSCRIPTION tap_sub_B2
> + CONNECTION '$node_C_connstr application_name=$appname_B2'
> + PUBLICATION tap_pub_C
> + WITH (only_local = on)");
> +
>
> AFAIK the "WITH (only_local = on)" is unnecessary here. We don't care
> where the node_C data came from for this test case.

This test was added to handle the comment as in [1]

Thanks for the comments, the attached v17 patch has the fixes for the same.
I have also separated the bidirectional logical replication steps,
since the steps are slightly complex and keeping it separate will help
in easier review and modifying.
[1] - https://www.postgresql.org/message-id/CAHut%2BPvwqwzuoQio-hJniarDUOTyxFrwrS9hucd47gEW-9wu-g%40mail.gmail.com

Regards,
Vignesh

  109. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:46:16Z

    On Thu, May 26, 2022 at 2:06 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> This post completes my review of patch v15*.
>
> ======
>
> 1. A general comment affecting both patches
>
> I realised the things defined in the subscription WITH are referred to
> in the PG DOCS as "parameters" (not "options"). See CREATE
> SUBSCRIPTION [1]. This is already reflected in some of my review
> comments for the v15-0002 below., but I'd already posted my v15-0001
> review comments before noticing this.
>
> Please also go through the v15-0001 patch to make all the necessary
> terminology changes (including in the v15-0001 commit message).

Modified

> =====
>
> Please find below review comments for patch v15-0002.
>
> 2. Commit message
>
> This patch does couple of things:
> change 1) Added force option for copy_data.
> change 2) Check and throw an error if the publication tables were also
> subscribing data in the publisher from other publishers.
>
> --
>
> 2a.
> "couple" -> "a couple"
> "Added force" -> "Adds force ..."
> "Check and throw" -> "Checks and throws ..."

Modified

> 2b.
> Isn't "Change 2)" only applicable when copy_data and only_local are
> on? It should say so here.

Modified

> ~~~
>
> 3. Commit message
>
> node1: Table t1 (c1 int) has data 1, 2, 3, 4
> node2: Table t1 (c1 int) has data 5, 6, 7, 8
>
> --
>
> I feel these kinds of replication examples are always much easier to
> visualize when the data somehow indicates where it came from. E.g.
> consider using different data in your example like:
>
> node1: Table t1 (c1 int) has data 11, 12, 13, 14
> node2: Table t1 (c1 int) has data 21, 22, 23, 24

Modified

> ~~~
>
> 4. Commit message
>
> Examples in this message exceed 80 chars. Please wrap the SQL appropriately

Modified

> ~~~
>
> 5. Commit message
>
> After this the data will be something like this:
> node1:
> 1, 2, 3, 4, 5, 6, 7, 8
>
> node2:
> 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8
>
> So, you can see that data on node2 (5, 6, 7, 8) is duplicated.
>
> --
>
> Yeah, but is that entirely correct though? AFAIK the scenario
> described by this example is going to recurse forever.

Modified

> ~~~
>
> 6. Commit message
>
> This problem can be solved by using only_local and copy_data option as
> given below
>
> --
>
> "option" -> "parameters"

Modified

> ~~~
>
> 7. Commit message
>
> There are other minor review comments but it's too messy to report in
> detail here. I suggest checking all the ":" (some were missing) and
> the letter case of/within the sentences. I also suggest
> cutting/pasting all this text into MSWord or some grammar checker to
> correct anything else reported.

Modified

> ~~~
>
> 8. Commit message
>
> CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '<node1 details>'
> PUBLICATION pub_node1 WITH (copy_data = force, only_local = on);
> ERROR:  CREATE/ALTER SUBSCRIPTION with only_local and copy_data as
> true is not allowed when the publisher might have replicated data
>
> --
>
> Looks like a typo: e.g. If copy_data is "force" then how is the error happening?

Modified

> ~~~
>
> 9. Commit message
>
> The examples seem confused/muddled in some way IMO.
>
> IIUC the new ERROR is like a new integrity check for when the
> subscription is using the only_local = on, along with copy_data = on.
>
> And so then the copy_data = force was added as a way to override that
> previous error being reported.
>
> So unless you know about the possible error then the "force" option
> makes no sense. So I think you need to change the order of the
> explanations in the commit message to describe the new error first.

Modified

> ======
>
> 10. doc/src/sgml/logical-replication.sgml
>
> +
> + <sect1 id="bidirectional-logical-replication">
> +  <title>Bidirectional logical replication</title>
> +
> +  <sect2 id="setting-bidirectional-replication-two-nodes">
> +   <title>Setting bidirectional replication between two nodes</title>
> +   <para>
> +    Bidirectional replication is useful in creating a multi-master database
> +    which helps in performing read/write operations from any of the nodes.
> +    Setting up bidirectional logical replication between two nodes requires
> +    creation of a publication in all the nodes, creating subscriptions in
> +    each of the nodes that subscribes to data from all the nodes. The steps
> +    to create a two-node bidirectional replication when there is no data in
> +    both the nodes are given below:
> +   </para>
>
> This description is confusing. It seems to be trying to say something
> about a multi-master system (e.g. you say "all the nodes") yet this is
> all written inside the section about "... two nodes", so saying "all
> the nodes" makes no sense here.
>
> I think you need to split this information and put some kind of blurb
> text at the top level (section 31.11), and then this section (for "two
> nodes") should ONLY be talking about the case when there are just TWO
> nodes.

Modified

> ~~~
>
> 11. doc/src/sgml/logical-replication.sgml
>
> +   <para>
> +    Lock the required tables in <literal>node1</literal> and
> +    <literal>node2</literal> till the setup is completed.
> +   </para>
>
> "till" -> "until" (make this same change in multiple places)

Modified

> ~~~
>
> 12. doc/src/sgml/logical-replication.sgml
>
> +  <sect2 id="add-new-node-data-in-existing-node">
> +   <title>Adding a new node when data is present in the existing nodes</title>
> +    <para>
> +     Adding a new node <literal>node3</literal> to the existing
> +     <literal>node1</literal> and <literal>node2</literal> when data is present
> +     in existing nodes <literal>node1</literal> and <literal>node2</literal>
> +     needs similar steps. The only change required here is that
> +     <literal>node3</literal> should create a subscription with
> +     <literal>copy_data = force</literal> to one of the existing nodes to
> +     receive the existing data during initial data synchronization.
> +   </para>
>
> I think perhaps this should clarify that there is NO data in node3 for
> this example.

Modified

> ~~~
>
> 13. doc/src/sgml/logical-replication.sgml - section 31.11.3
>
> +   <para>
> +    Lock the required tables in <literal>node2</literal> and
> +    <literal>node3</literal> till the setup is completed.
> +   </para>
>
> Should you describe why you say no lock is required for tables on
> node1? (Similar information is in section 31.11.4 also)

Modified

> ~~~
>
> 14. doc/src/sgml/logical-replication.sgml - section 31.11.3
>
> +   <para>
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node2</literal>:
> +<programlisting>
> +node3=# CREATE SUBSCRIPTION sub_node3_node2
> +node3-# CONNECTION 'dbname=foo host=node2 user=repuser'
> +node3-# PUBLICATION pub_node2
> +node3-# WITH (copy_data = off, only_local = on);
> +CREATE SUBSCRIPTION
> +</programlisting></para>
>
> Using a similar description as in section 31.11.4 this should probably
> also say something like:
> "Use copy_data specified as off because the initial table data would
> have been already copied in the previous step".

Modified

> ~~~
>
> 15. doc/src/sgml/logical-replication.sgml - section 31.11.4
>
> +  <sect2 id="add-node-data-present-in-new-node">
> +   <title>Adding a new node when data is present in the new node</title>
> +   <para>
> +     Adding a new node <literal>node3</literal> to the existing
> +     <literal>node1</literal> and <literal>node2</literal> when data is present
> +     in the new node <literal>node3</literal> needs similar steps. A
> few changes
> +     are required here to get the existing data from <literal>node3</literal>
> +     to <literal>node1</literal> and <literal>node2</literal> and later
> +     cleaning up of data in <literal>node3</literal> before synchronization of
> +     all the data from the existing nodes.
> +   </para>
>
> I think perhaps this should clarify that there is NO data in node1 or
> node2 for this example.

This section contents have been removed because this is not feasible currently.

> ~~~
>
> 16. doc/src/sgml/logical-replication.sgml
>
> +  <sect2 id="generic-steps-add-new-node">
> +   <title>Generic steps to add a new node to the existing set of nodes</title>
>
> I think all the steps in this section should be numbered because the
> order is important.

Modified

> ~~~
>
> 17. doc/src/sgml/logical-replication.sgml
>
> +  <sect2 id="generic-steps-add-new-node">
> +   <title>Generic steps to add a new node to the existing set of nodes</title>
>
> IIUC the word "parameter" should replace the word "option" in all the
> text of this section.

Modified

> ~~~
>
> 18. doc/src/sgml/logical-replication.sgml - Notes
>
> +  <sect2>
> +   <title>Notes</title>
> +   <para>
> +    Setting up bidirectional logical replication across nodes requires multiple
> +    steps to be performed on various nodes, as all operations are not
> +    transactional, user is advised to take backup of existing data to avoid any
> +    inconsistency.
> +   </para>
> +  </sect2>
>
> 18a.
> I thought this "Notes" section should be rendered more like an sgml
> note or a warning. E.g. like the warning on this page [2].

Modified

> 18b.
> SUGGESTION (split the para into multiple sentences). e.g.
> Setting up bidirectional logical replication across nodes requires
> multiple steps to be performed on various nodes. Because not all
> operations are transactional, the user is advised to take backups.

Modified

> ======
>
> 19. doc/src/sgml/ref/alter_subscription.sgml
>
> +         <para>
> +          There is some interaction between the <literal>only_local</literal>
> +          option and <literal>copy_data</literal> option. Refer to the
> +          <command>CREATE SUBSCRIPTION</command>
> +          <xref linkend="sql-createsubscription-notes" /> for interaction
> +          details and usage of <literal>force</literal> for
> +          <literal>copy_data</literal> option.
>           </para>
>
> "option" -> "parameter" (3x)

Modified

> ======
>
> 20. doc/src/sgml/ref/create_subscription.sgml
>
> +         <para>
> +          There is some interaction between the <literal>only_local</literal>
> +          option and <literal>copy_data</literal> option. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for interaction
> +          details and usage of <literal>force</literal> for
> +          <literal>copy_data</literal> option.
> +         </para>
>
> "option" -> "parameter" (3x)

Modified

> ======
>
> 21. src/backend/commands/subscriptioncmds.c
>
> @@ -69,6 +69,18 @@
>  /* check if the 'val' has 'bits' set */
>  #define IsSet(val, bits)  (((val) & (bits)) == (bits))
>
> +#define IS_COPY_DATA_ON_OR_FORCE(copy_data) ((copy_data) != COPY_DATA_OFF)
> +
> +/*
> + * Represents whether copy_data option is specified with off, on or force.
> + */
> +typedef enum CopyData
> +{
> + COPY_DATA_OFF,
> + COPY_DATA_ON,
> + COPY_DATA_FORCE
> +} CopyData;
> +
>
> 21a.
> "option" -> "parameter"

Modified

> 21b.
> I think the new #define is redundant and can be removed (see the next
> review comment)

Modified

> 21c.
> I think you set the OFF enum value like:
> typedef enum CopyData
> {
>  COPY_DATA_OFF = 0,
>  COPY_DATA_ON,
>  COPY_DATA_FORCE
> } CopyData;
>
> then it will greatly simplify things; many changes in this file will
> be unnecessary (see subsequent review comments)

Modified

> ~~~
>
> 22. src/backend/commands/subscriptioncmds.c
>
> +/*
> + * Validate the value specified for copy_data option.
> + */
> +static CopyData
> +DefGetCopyData(DefElem *def)
>
> "option" -> "parameter"

Modified

> ~~~
>
> 23. src/backend/commands/subscriptioncmds.c
>
> + /*
> + * Allow 0, 1, "true", "false", "on", "off" or "force".
> + */
> + switch (nodeTag(def->arg))
> + {
> + case T_Integer:
> + switch (intVal(def->arg))
> + {
> + case 0:
> + return COPY_DATA_OFF;
> + case 1:
> + return COPY_DATA_ON;
> + default:
> + /* otherwise, error out below */
> + break;
> + }
> + break;
>
> What about also allowing copy_data = 2, and making it equivalent to "force"?

I felt the existing looks ok, no need to support 2. It might confuse the user.

> ~~~
>
> 24. src/backend/commands/subscriptioncmds.c
>
> @@ -333,17 +400,17 @@ parse_subscription_options(ParseState *pstate,
> List *stmt_options,
>   errmsg("%s and %s are mutually exclusive options",
>   "connect = false", "create_slot = true")));
>
> - if (opts->copy_data &&
> + if (IS_COPY_DATA_ON_OR_FORCE(opts->copy_data) &&
>   IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
>
> I think this change would not be needed if you set the OFF enum = 0.

Modified

> ~~~
> 25. src/backend/commands/subscriptioncmds.c
>
> @@ -671,13 +738,14 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>   * Set sync state based on if we were asked to do data copy or
>   * not.
>   */
> - table_state = opts.copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY;
> + table_state = IS_COPY_DATA_ON_OR_FORCE(opts.copy_data) ?
> SUBREL_STATE_INIT : SUBREL_STATE_READY;
>
> I think this change would not be needed if you set the OFF enum = 0.

Modified

> ~~~
>
> 26. src/backend/commands/subscriptioncmds.c
>
> @@ -720,7 +788,8 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>   * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
>   * PUBLICATION to work.
>   */
> - if (opts.twophase && !opts.copy_data && tables != NIL)
> + if (opts.twophase && opts.copy_data == COPY_DATA_OFF &&
> + tables != NIL)
>   twophase_enabled = true;
>
> I think this change would not be needed if you set the OFF enum = 0.

Modified

> ~~~
>
> 27. src/backend/commands/subscriptioncmds.c
>
> @@ -851,7 +921,7 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
>   list_length(subrel_states), sizeof(Oid), oid_cmp))
>   {
>   AddSubscriptionRelState(sub->oid, relid,
> - copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
> + IS_COPY_DATA_ON_OR_FORCE(copy_data) ? SUBREL_STATE_INIT : SUBREL_STATE_READY,
>   InvalidXLogRecPtr);
>
> I think this change would not be needed if you set the OFF enum = 0.
>

Modified

> ~~~
> 28. src/backend/commands/subscriptioncmds.c
>
> @@ -1157,7 +1227,7 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>   * See ALTER_SUBSCRIPTION_REFRESH for details why this is
>   * not allowed.
>   */
> - if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
> + if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
> IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))
>   ereport(ERROR,
>   (errcode(ERRCODE_SYNTAX_ERROR),
>   errmsg("ALTER SUBSCRIPTION with refresh and copy_data is not allowed
> when two_phase is enabled"),
>
> I think this change would not be needed if you set the OFF enum = 0.

Modified

> ~~~
>
> 29. src/backend/commands/subscriptioncmds.c
>
> @@ -1209,7 +1279,7 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>   * See ALTER_SUBSCRIPTION_REFRESH for details why this is
>   * not allowed.
>   */
> - if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
> + if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
> IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))
>   ereport(ERROR,
>   (errcode(ERRCODE_SYNTAX_ERROR),
>   errmsg("ALTER SUBSCRIPTION with refresh and copy_data is not allowed
> when two_phase is enabled"),
>
> I think this change would not be needed if you set the OFF enum = 0.

Modified

> ~~~
>
> 30. src/backend/commands/subscriptioncmds.c
>
> @@ -1255,7 +1325,8 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>   *
>   * For more details see comments atop worker.c.
>   */
> - if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
> + if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
> + IS_COPY_DATA_ON_OR_FORCE(opts.copy_data))
>   ereport(ERROR,
>   (errcode(ERRCODE_SYNTAX_ERROR),
>   errmsg("ALTER SUBSCRIPTION ... REFRESH with copy_data is not allowed
> when two_phase is enabled"),
>
> I think this change would not be needed if you set the OFF enum = 0.

Modified

> ~~~
>
> 31. src/backend/commands/subscriptioncmds.c
>
> + appendStringInfoString(&cmd,
> +    "SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename,
> PS.srrelid as replicated\n"
> +    "FROM pg_publication P,\n"
> +    "LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> +    "LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> +    "pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> +    "WHERE C.oid = GPT.relid AND P.pubname in (");
>
> use upper case "in" -> "IN"

Modified

> ======
>
> 32.  src/test/regress/sql/subscription.sql
>
> @@ -40,6 +40,9 @@ SET SESSION AUTHORIZATION 'regress_subscription_user';
>
>  -- fail - invalid option combinations
>  CREATE SUBSCRIPTION regress_testsub2 CONNECTION
> 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
> false, copy_data = true);
> +CREATE SUBSCRIPTION regress_testsub2 CONNECTION
> 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
> false, copy_data = on);
> +CREATE SUBSCRIPTION regress_testsub2 CONNECTION
> 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
> false, copy_data = 1);
> +CREATE SUBSCRIPTION regress_testsub2 CONNECTION
> 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect =
> false, copy_data = force);
>
> 32a.
> The test for "copy_data = 1" should confirm it is the same as "on".

We have the test for copy_data = on and copy_data = 1, both the
results are the same. Isn't that sufficient?

> 32b.
> Should also test copy_data = 2 (and confirm it is the same as "force").

Currently only on, off, true, false, 0 & 1 is supported. 2 is not
supported. Added a test for copy_data as 2

> ======
>
> 33. src/test/subscription/t/032_localonly.pl - cosmetic changes
>
> 33a.
> +# Detach node C from the node-group of (A, B, C) and clean the table contents
> +# from all nodes.
>
> SUGGESTION
> Detach node_C from the node-group of (node_A, node_B, node_C) and ...

Modified

> 33b.
> +# Subroutine to create subscription and wait till the initial sync is
> completed.
>
> "till" -> "until"

Modified

> 33c.
> +# Subroutine to create subscription and wait till the initial sync is
> completed.
> +# Subroutine expects subscriber node, publisher node, subscription name,
> +# destination connection string, publication name and the subscription with
> +# options to be passed as input parameters.
> +sub create_subscription
> +{
> + my ($node_subscriber, $node_publisher, $sub_name, $node_connstr,
> + $pub_name, $with_options)
> +   = @_;
>
> "subscription with options" => "subscription parameters"
> "$with_options" -> "$sub_params"

Modified

> 33d.
> +# Specifying only_local 'on' which indicates that the publisher should only
>
> "Specifying" => "Specify"

Modified

Thanks for the comments, the v17 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com

Regards,
Vignesh

  110. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:48:07Z

    On Thu, May 26, 2022 at 4:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 20, 2022 at 3:31 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, May 18, 2022 at 4:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > 5.
> > >  * It is quite possible that subscriber has not yet pulled data to
> > > + * the tables, but in ideal cases the table data will be subscribed.
> > > + * To keep the code simple it is not checked if the subscriber table
> > > + * has pulled the data or not.
> > > + */
> > > + if (copydata == COPY_DATA_ON && local_only && !slot_attisnull(slot, 3))
> > >
> > > Sorry, but I don't understand what you intend to say by the above
> > > comment. Can you please explain?
> >
> > When the user specifies copy_data as on, we should check if the
> > publisher has the publication tables being subscribed from a remote
> > publisher. If so throw an error as it remote origin data present.
> > Ex:
> > Node1 - pub1 for table t1 -- no data
> > Node2 - Sub1 subscribing to data from pub1
> > Node2 - pub2 for table t1 -- no data
> > Node3 - create subscription to Node2 with copy_data = ON
> >
> > In this case even though the table does not have any remote origin
> > data,  as Node2 is subscribing to data from Node1, throw an error.
> > We throw an error irrespective of data present in Node1 or not to keep
> > the code simple.
> >
>
> I think we can change the contents of comments something like: "Throw
> an error if the publisher has subscribed to the same table from some
> other publisher. We cannot differentiate between the local and
> non-local data that is present in the HEAP during the initial sync.
> Identification of local data can be done only from the WAL by using
> the origin id.  XXX: For simplicity, we don't check whether the table
> has any data or not. If the table doesn't have any data then we don't
> need to distinguish between local and non-local data so we can avoid
> throwing error in that case."

Modified

> Few more comments:
> ==================
> Patch 0002
> ======
> 1.
> + if (copydata == COPY_DATA_ON && only_local && !slot_attisnull(slot, 3))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("CREATE/ALTER SUBSCRIPTION with only_local and copy_data as
> true is not allowed when the publisher might have replicated data,
> table:%s.%s might have replicated data in the publisher",
> +    nspname, relname),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force"));
>
> Can we split the error message? errmsg: table:%s.%s has replicated
> data in the publisher; errdetail:CREATE/ALTER SUBSCRIPTION with
> only_local and copy_data as true is not allowed when the publisher has
> replicated data

Modified

> 2.
> +   <para>
> +    Lock the required tables in the new node until the setup is complete.
> +   </para>
> +   <para>
> +    Create subscriptions on existing nodes pointing to publication on
> +    the new node with <literal>only_local</literal> option specified as
> +    <literal>on</literal> and <literal>copy_data</literal> specified as
> +    <literal>on</literal>.
> +   </para>
> +   <para>
> +    Wait for data to be copied from the new node to existing nodes.
> +   </para>
> +   <para>
> +    Alter the publication in new node so that the truncate operation is not
> +    replicated to the subscribers.
> +   </para>
>
> Here and at other places, we should specify that the lock mode should
> to acquire the lock on table should be EXCLUSIVE so that no concurrent
> DML is allowed on it. Also, it is better if somewhere we explain why
> and which nodes need locks?

Modified

> Patch 0001:
> ==========
> 1.
> +$node_A->append_conf(
> + 'postgresql.conf', qq(
> +max_prepared_transactions = 10
> +logical_decoding_work_mem = 64kB
> +));
>
> I don't see why you need to set these parameters. There is no test
> case that needs these parameters. Please remove these from here and
> all other similar places in 032_onlylocal.pl.

Removed it.

Thanks for the comments, the v17 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com

Regards,
Vignesh

  111. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:49:03Z

    On Fri, May 27, 2022 at 7:54 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Apr 6, 2022 at 9:36 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Apr 5, 2022 at 7:06 PM Ashutosh Bapat
> > <ashutosh.bapat.oss@gmail.com> wrote:
> > >
> > > On Tue, Apr 5, 2022 at 6:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Below are some other name ideas. Maybe they are not improvements, but
> > > > it might help other people to come up with something better.
> > > >
> > > > subscribe_publocal_only = true/false
> > > > origin = pub_only/any
> > > > adjacent_data_only = true/false
> > > > direct_subscriptions_only = true/false
> > > > ...
> > >
> > > FWIW, The subscriber wants "changes originated on publisher". From
> > > that angle origin = publisher/any looks attractive. It also leaves
> > > open the possibility that the subscriber may ask changes from a set of
> > > origins or even non-publisher origins.
> > >
> >
> > So, how are you imagining extending it for multiple origins? I think
> > we can have multiple origins data on a node when there are multiple
> > subscriptions pointing to the different or same node. The origin names
> > are internally generated names but are present in
> > pg_replication_origin. So, are we going to ask the user to check that
> > table and specify it as an option? But, note, that the user may not be
> > able to immediately recognize which origin data is useful for her.
> >
>
> I still don't have a very clear answer for the usability aspect but it
> seems this was discussed in PGCon-Unconference [1] (Section: Origin
> filter) and there also it was suggested to allow users to specify
> multiple origin names. So, probably Ashutosh's idea makes sense and we
> should use "origin = publisher/any or origin=local/any". Among these,
> I would prefer later.

I have changed it to origin = local/any.

The v17 patch attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com

Regards,
Vignesh

  112. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:53:15Z

    On Fri, May 27, 2022 at 10:56 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some more review comments for v15-0002 I found while working
> through the documented examples. These are all for
> doc/src/sgml/logical-replication.sgml. The section numbers (e.g.
> 31.11.1) are the assigned numbers after HTML rendering.
>
> The general comment is that the sections appeared at first to be
> independent of each other, but they are not really - sometimes
> existing data and existing pub/sub are assumed to remain. Therefore, I
> think some more text needs to be added to clarify the initial state
> for each of these sections.
>
> ======
>
> 1. Section id.
>
> <sect1 id="bidirectional-logical-replication">
>
> IMO this section ID should be renamed
> "logical-replication-bidirectional" then it will be more consistent
> filenames with all the other logical replication section names.

Modified

> ~~~
>
> 2. Section 31.11.2. Adding a new node when there is no data in any of the nodes
>
> 2a.
> This seems like a continuation of 31.11.1 because pub/sub of
> node1,node2 is assumed to exist still.

Added a note

> 2b.
> The bottom of this section should say after these steps any subsequent
> incremental data changes on any node will be replicated to all nodes
> (node1, node2, node3)

Modified

> ~~~
>
> 3. Section 31.11.3. Adding a new node when data is present in the existing nodes
>
> 3a.
> This seems to be a continuation of 31.11.1 because pub/sub (and
> initial data) of node1/node2 is assumed to exist.

Added a note

> 3b.
>
> The bottom of this section should say after these steps that any
> initial data of node1/node2 will be seen in node3, and any subsequent
> incremental data changes on any node will be replicated to all nodes
> (node1, node2, node3)

Modified

> ~~~
>
> 4. Section 31.11.4. Adding a new node when data is present in the new node
>
> 4a.
> This seems to be a continuation of 31.11.1 because pub/sub (and
> initial data) of node1/node2 is assumed to exist.

This section contents have been removed because this is not feasible currently

> 4b.
> It is not made very clear up-front if the tables on node1 and node2
> are empty or not. Apparently, they are considered NOT empty because
> later in the example you are using "force" when you create the
> subscription on node3.

This section contents have been removed because this is not feasible currently

> 4c.
> The SQL wrapping here is strangely different from others (put the
> subscription name on 1st line)
> node3=# CREATE SUBSCRIPTION
> node3-# sub_node3_node1 CONNECTION 'dbname=foo host=node1 user=repuser'
> node3-# PUBLICATION pub_node1
> node3-# WITH (copy_data = force, only_local = on);
> CREATE SUBSCRIPTION

This section contents have been removed because this is not feasible currently

> 4d.
> The SQL wrapping here is strangely different from others (put the
> subscription name on 1st line)
> node3=# CREATE SUBSCRIPTION
> node3-# sub_node3_node2 CONNECTION 'dbname=foo host=node2 user=repuser'
> node3-# PUBLICATION pub_node2
> node3-# WITH (copy_data = off, only_local = on);
> CREATE SUBSCRIPTION

This section contents have been removed because this is not feasible currently

> 4e.
> The bottom of this section should say after this that any initial data
> of node1/node2 will be seen in node3, and any subsequent incremental
> data changes on any node will be replicated to all nodes (node1,
> node2, node3)

This section contents have been removed because this is not feasible currently

> ~~~
>
> 5. Section 31.11.5. Generic steps to add a new node to the existing set of nodes
>
> 5a
> Create subscriptions on the new node pointing to publication on the
> first node with only_local option specified as on and copy_data option
> specified as "force".
>
> -> that should say "Create a subscription" (singular)

Modified

> 5b.
> Create subscriptions on the new node pointing to publications on the
> remaining node with only_local option specified as on and copy_data
> option specified as off.
>
> -> that should say "on the remaining node" (plural)

Modified

Thanks for the comments, the v17 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com

Regards,
Vignesh

  113. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:55:02Z

    On Fri, May 27, 2022 at 12:34 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Wed, May 25, 2022 7:55 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v16 patch has the changes for the same.
> >
>
> Thanks for updating the patch.
>
> Some comments for the document in 0002 patch.
>
> 1.
> +   <para>
> +    Lock the required tables in <literal>node1</literal> and
> +    <literal>node2</literal> till the setup is completed.
> +   </para>
> +
> +   <para>
> +    Create a publication in <literal>node1</literal>:
> +<programlisting>
> +node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1;
> +CREATE PUBLICATION
> +</programlisting></para>
>
> If the table is locked in the very beginning, we will not be able to create the
> publication (because the locks have conflict). Maybe we should switch the order
> of creating publication and locking tables here.

Modified

> 2.
> In the case of "Adding a new node when data is present in the new node", we need
> to truncate table t1 in node3, but the truncate operation would be blocked
> because the table has be locked before. Maybe we need some changes for it.

This section contents have been removed because this is not feasible currently

Thanks for the comments, the v17 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com

Regards,
Vignesh

  114. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-01T17:57:36Z

    On Fri, May 27, 2022 at 2:08 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, May 27, 2022 at 5:04 PM shiy.fnst@fujitsu.com
> <shiy.fnst@fujitsu.com> wrote:
> >
> > On Wed, May 25, 2022 7:55 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > The attached v16 patch has the changes for the same.
> > >
> >
> > Thanks for updating the patch.
> >
> > Some comments for the document in 0002 patch.
> >
> > 1.
> > +   <para>
> > +    Lock the required tables in <literal>node1</literal> and
> > +    <literal>node2</literal> till the setup is completed.
> > +   </para>
> > +
> > +   <para>
> > +    Create a publication in <literal>node1</literal>:
> > +<programlisting>
> > +node1=# CREATE PUBLICATION pub_node1 FOR TABLE t1;
> > +CREATE PUBLICATION
> > +</programlisting></para>
> >
> > If the table is locked in the very beginning, we will not be able to create the
> > publication (because the locks have conflict). Maybe we should switch the order
> > of creating publication and locking tables here.
> >
>
> I agree. It seems some of the locking instructions in the earlier
> sections 31.11.1 - 31.11.4 are contradictory to the later "generic"
> steps given in "31.11.5. Generic steps to add a new node to the
> existing set of nodes". I'm assuming the generic steps are the
> "correct" steps
>
> e.g. generic steps say get the lock on new node tables AFTER the
> publication of new node.
> e.g. generic steps say do NOT get a table lock on the node one you are
> (first) joining to.

Yes, the generic steps are correct. modified

> ~~~
>
> Furthermore, the generic steps are describing attaching a new N+1th
> node to some (1 ... N) other nodes.
>
> So I think in the TWO-node case (section 31.11.1) node2 should be
> treated as the "new" node that you are attaching to the "first" node1.
> IMO the section 31.11.1 example should be reversed so that it obeys
> the "generic" pattern.
> e.g. It should be doing the CREATE PUBLICATION pub_node2 first (since
> that is the "new" node)

In generic steps we mention the publication must be created in a new
node and we don't say about the existing nodes, because the
publication would be existing already. I feel the existing order of
publication creation in the TWO-node case (section 31.11.1) is ok.

Thanks for the comments, the v17 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1rMihO7daiFyLdxkqArrC%2BdtM61oPXc-XrTYBYnJg3nw%40mail.gmail.com

Regards,
Vignesh

  115. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-03T05:26:29Z

    Please find below my review comments for patch v17-0001:

======

1. Commit message
Add a missing test to verify only-local option in test_decoding plugin.

"option" -> "parameter"

======

2. contrib/test_decoding/sql/replorigin.sql

2a.
+-- Verify that remote origin data is not returned with only-local option

SUGGESTION
-- Verify the behaviour of the only-local parameter

2b.
+-- Remote origin data returned when only-local option is not set

"option" -> "parameter"

2c.
+-- Remote origin data not returned when only-local option is set

"option" -> "parameter"

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  116. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-03T05:30:49Z

    Please find below my review comments for patch v17-0002:

======

1. Commit message

This patch adds a new SUBSCRIPTION parameter "origin". It Specifies whether
the subscription will request the publisher to only send changes that
originated locally, or to send any changes regardless of origin.

"It Specifies" -> "It specifies"

~~~

2. Commit message

Usage:
CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres port=9999'
PUBLICATION pub1 with (origin = local);

"with" -> "WITH"

======

3. doc/src/sgml/catalogs.sgml

+      <para>
+       Possible origin values are <literal>local</literal>,
+       <literal>any</literal>, or <literal>NULL</literal> if none is specified.
+       If <literal>local</literal>, the subscription will request the
+       publisher to only send changes that originated locally. If
+       <literal>any</literal>, the publisher sends any changes regardless of
+       their origin.
+      </para></entry>

Should this also mention that NULL (default) is equivalent to 'any'?

SUGGESTION
If <literal>any</literal> (or <literal>NULL</literal>), the publisher
sends any changes regardless of their origin.

======

4. src/backend/catalog/pg_subscription.c

@@ -72,6 +72,16 @@ GetSubscription(Oid subid, bool missing_ok)
  sub->twophasestate = subform->subtwophasestate;
  sub->disableonerr = subform->subdisableonerr;

+ datum = SysCacheGetAttr(SUBSCRIPTIONOID,
+ tup,
+ Anum_pg_subscription_suborigin,
+ &isnull);
+
+ if (!isnull)
+ sub->origin = TextDatumGetCString(datum);
+ else
+ sub->origin = NULL;
+
  /* Get conninfo */
  datum = SysCacheGetAttr(SUBSCRIPTIONOID,
  tup,

Missing comment like the nearby code has:
/* Get origin */

======

5. src/backend/replication/logical/worker.c

+/* Macro for comparing string fields that might be NULL */
+#define equalstr(a, b) \
+ (((a) != NULL && (b) != NULL) ? (strcmp(a, b) == 0) : (a) == (b))
+

Should that have some extra parens for the macro args?
e.g. "strcmp((a), (b))"

~~~

6. src/backend/replication/logical/worker.c - maybe_reread_subscription

@@ -3059,6 +3063,7 @@ maybe_reread_subscription(void)
  strcmp(newsub->slotname, MySubscription->slotname) != 0 ||
  newsub->binary != MySubscription->binary ||
  newsub->stream != MySubscription->stream ||
+ equalstr(newsub->origin, MySubscription->origin) ||
  newsub->owner != MySubscription->owner ||
  !equal(newsub->publications, MySubscription->publications))
  {

Is that right? Shouldn't it say !equalstr(...)?

======

7. src/backend/replication/pgoutput/pgoutput.c - parse_output_parameters

@@ -380,6 +382,16 @@ parse_output_parameters(List *options, PGOutputData *data)

  data->two_phase = defGetBoolean(defel);
  }
+ else if (strcmp(defel->defname, "origin") == 0)
+ {
+ if (origin_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ origin_option_given = true;
+
+ data->origin = defGetString(defel);
+ }

Should this function also be validating that the origin parameter
value is only permitted to be one of "local" or "any"?

~~~

8. src/backend/replication/pgoutput/pgoutput.c - pgoutput_origin_filter

@@ -1698,12 +1710,20 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
 }

 /*
- * Currently we always forward.
+ * Return true if the data source (origin) is remote and user has requested
+ * only local data, false otherwise.
  */
 static bool
 pgoutput_origin_filter(LogicalDecodingContext *ctx,
     RepOriginId origin_id)
 {
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+
+ if (data->origin &&
+ (strcmp(data->origin, "local") == 0) &&
+ origin_id != InvalidRepOriginId)
+ return true;
+
  return false;
 }

8a.
Could rewrite the condition by putting the strcmp last so you can
avoid doing unnecessary strcmp.

e.g
+ if (data->origin &&
+ origin_id != InvalidRepOriginId &&
+ strcmp(data->origin, "local" == 0)


8b.
I also wondered if it might be worth considering caching the origin
parameter value when it was parsed so that you can avoid doing any
strcmp at all during this function. Because otherwise this might be
called millions of times, right?

======

9. src/include/catalog/pg_subscription.h

@@ -87,6 +87,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW

  /* List of publications subscribed to */
  text subpublications[1] BKI_FORCE_NOT_NULL;
+
+ /* Publish the data originated from the specified origin */
+ text suborigin;
 #endif
 } FormData_pg_subscription;

SUGGESTION (for the comment text)
/* Only publish data originating from the specified origin */

~~~

10 src/include/catalog/pg_subscription.h - Subscription

@@ -118,6 +121,9 @@ typedef struct Subscription
  char    *slotname; /* Name of the replication slot */
  char    *synccommit; /* Synchronous commit setting for worker */
  List    *publications; /* List of publication names to subscribe to */
+ char    *origin; /* Publish the data originated from the
+ * specified origin */
+
 } Subscription;

10a.
Reword comment same as suggested in review comment #9

10b.
Remove spurious blank line

======

11. src/include/replication/walreceiver.h

@@ -183,6 +183,8 @@ typedef struct
  bool streaming; /* Streaming of large transactions */
  bool twophase; /* Streaming of two-phase transactions at
  * prepare time */
+ char    *origin; /* Publish the data originated from the
+ * specified origin */
  } logical;

Reword comment same as suggested in review comment #9

======

12. src/test/subscription/t/032_origin.pl

+# Test logical replication using origin option.

# Test the CREATE SUBSCRIPTION 'origin' parameter

~~~

13. src/test/subscription/t/032_origin.pl

+# check that the data published from node_C to node_B is not sent to node_A
+$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full ORDER BY 1;");
+is($result, qq(11
+12), 'Remote data originating from another node (not the publisher)
is not replicated when origin option is local'
+);

"option" -> "parameter"

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  117. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-06T05:14:08Z

    On Fri, Jun 3, 2022 at 10:56 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Please find below my review comments for patch v17-0001:
>
> ======
>
> 1. Commit message
> Add a missing test to verify only-local option in test_decoding plugin.
>
> "option" -> "parameter"

Modified

> ======
>
> 2. contrib/test_decoding/sql/replorigin.sql
>
> 2a.
> +-- Verify that remote origin data is not returned with only-local option
>
> SUGGESTION
> -- Verify the behaviour of the only-local parameter

Modified

> 2b.
> +-- Remote origin data returned when only-local option is not set
>
> "option" -> "parameter"

Modified

> 2c.
> +-- Remote origin data not returned when only-local option is set
>
> "option" -> "parameter"

Modified

The attached v18 patch has the fixes for the same.

Regards,
Vignesh

  118. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-06T05:17:09Z

    On Fri, Jun 3, 2022 at 11:01 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Please find below my review comments for patch v17-0002:
>
> ======
>
> 1. Commit message
>
> This patch adds a new SUBSCRIPTION parameter "origin". It Specifies whether
> the subscription will request the publisher to only send changes that
> originated locally, or to send any changes regardless of origin.
>
> "It Specifies" -> "It specifies"

Modified

> ~~~
>
> 2. Commit message
>
> Usage:
> CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres port=9999'
> PUBLICATION pub1 with (origin = local);
>
> "with" -> "WITH"

Modified

> ======
>
> 3. doc/src/sgml/catalogs.sgml
>
> +      <para>
> +       Possible origin values are <literal>local</literal>,
> +       <literal>any</literal>, or <literal>NULL</literal> if none is specified.
> +       If <literal>local</literal>, the subscription will request the
> +       publisher to only send changes that originated locally. If
> +       <literal>any</literal>, the publisher sends any changes regardless of
> +       their origin.
> +      </para></entry>
>
> Should this also mention that NULL (default) is equivalent to 'any'?
>
> SUGGESTION
> If <literal>any</literal> (or <literal>NULL</literal>), the publisher
> sends any changes regardless of their origin.

Modified

> ======
>
> 4. src/backend/catalog/pg_subscription.c
>
> @@ -72,6 +72,16 @@ GetSubscription(Oid subid, bool missing_ok)
>   sub->twophasestate = subform->subtwophasestate;
>   sub->disableonerr = subform->subdisableonerr;
>
> + datum = SysCacheGetAttr(SUBSCRIPTIONOID,
> + tup,
> + Anum_pg_subscription_suborigin,
> + &isnull);
> +
> + if (!isnull)
> + sub->origin = TextDatumGetCString(datum);
> + else
> + sub->origin = NULL;
> +
>   /* Get conninfo */
>   datum = SysCacheGetAttr(SUBSCRIPTIONOID,
>   tup,
>
> Missing comment like the nearby code has:
> /* Get origin */

Modified

> ======
>
> 5. src/backend/replication/logical/worker.c
>
> +/* Macro for comparing string fields that might be NULL */
> +#define equalstr(a, b) \
> + (((a) != NULL && (b) != NULL) ? (strcmp(a, b) == 0) : (a) == (b))
> +
>
> Should that have some extra parens for the macro args?
> e.g. "strcmp((a), (b))"

Modified

> ~~~
>
> 6. src/backend/replication/logical/worker.c - maybe_reread_subscription
>
> @@ -3059,6 +3063,7 @@ maybe_reread_subscription(void)
>   strcmp(newsub->slotname, MySubscription->slotname) != 0 ||
>   newsub->binary != MySubscription->binary ||
>   newsub->stream != MySubscription->stream ||
> + equalstr(newsub->origin, MySubscription->origin) ||
>   newsub->owner != MySubscription->owner ||
>   !equal(newsub->publications, MySubscription->publications))
>   {
>
> Is that right? Shouldn't it say !equalstr(...)?

Modified

> ======
>
> 7. src/backend/replication/pgoutput/pgoutput.c - parse_output_parameters
>
> @@ -380,6 +382,16 @@ parse_output_parameters(List *options, PGOutputData *data)
>
>   data->two_phase = defGetBoolean(defel);
>   }
> + else if (strcmp(defel->defname, "origin") == 0)
> + {
> + if (origin_option_given)
> + ereport(ERROR,
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("conflicting or redundant options")));
> + origin_option_given = true;
> +
> + data->origin = defGetString(defel);
> + }
>
> Should this function also be validating that the origin parameter
> value is only permitted to be one of "local" or "any"?

Modified

> ~~~
>
> 8. src/backend/replication/pgoutput/pgoutput.c - pgoutput_origin_filter
>
> @@ -1698,12 +1710,20 @@ pgoutput_message(LogicalDecodingContext *ctx,
> ReorderBufferTXN *txn,
>  }
>
>  /*
> - * Currently we always forward.
> + * Return true if the data source (origin) is remote and user has requested
> + * only local data, false otherwise.
>   */
>  static bool
>  pgoutput_origin_filter(LogicalDecodingContext *ctx,
>      RepOriginId origin_id)
>  {
> + PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
> +
> + if (data->origin &&
> + (strcmp(data->origin, "local") == 0) &&
> + origin_id != InvalidRepOriginId)
> + return true;
> +
>   return false;
>  }
>
> 8a.
> Could rewrite the condition by putting the strcmp last so you can
> avoid doing unnecessary strcmp.
>
> e.g
> + if (data->origin &&
> + origin_id != InvalidRepOriginId &&
> + strcmp(data->origin, "local" == 0)

I have avoided the strcmp by caching locally as suggested in 8b. I
have not made any change for this.

>
> 8b.
> I also wondered if it might be worth considering caching the origin
> parameter value when it was parsed so that you can avoid doing any
> strcmp at all during this function. Because otherwise this might be
> called millions of times, right?

Modified

> ======
>
> 9. src/include/catalog/pg_subscription.h
>
> @@ -87,6 +87,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>
>   /* List of publications subscribed to */
>   text subpublications[1] BKI_FORCE_NOT_NULL;
> +
> + /* Publish the data originated from the specified origin */
> + text suborigin;
>  #endif
>  } FormData_pg_subscription;
>
> SUGGESTION (for the comment text)
> /* Only publish data originating from the specified origin */

Modified

> ~~~
>
> 10 src/include/catalog/pg_subscription.h - Subscription
>
> @@ -118,6 +121,9 @@ typedef struct Subscription
>   char    *slotname; /* Name of the replication slot */
>   char    *synccommit; /* Synchronous commit setting for worker */
>   List    *publications; /* List of publication names to subscribe to */
> + char    *origin; /* Publish the data originated from the
> + * specified origin */
> +
>  } Subscription;
>
> 10a.
> Reword comment same as suggested in review comment #9

Modified

> 10b.
> Remove spurious blank line

Modified

> ======
>
> 11. src/include/replication/walreceiver.h
>
> @@ -183,6 +183,8 @@ typedef struct
>   bool streaming; /* Streaming of large transactions */
>   bool twophase; /* Streaming of two-phase transactions at
>   * prepare time */
> + char    *origin; /* Publish the data originated from the
> + * specified origin */
>   } logical;
>
> Reword comment same as suggested in review comment #9

Modified

> ======
>
> 12. src/test/subscription/t/032_origin.pl
>
> +# Test logical replication using origin option.
>
> # Test the CREATE SUBSCRIPTION 'origin' parameter

Modified

> ~~~
>
> 13. src/test/subscription/t/032_origin.pl
>
> +# check that the data published from node_C to node_B is not sent to node_A
>+$result = $node_A->safe_psql('postgres', "SELECT * FROM tab_full ORDER BY 1;");
> +is($result, qq(11
> +12), 'Remote data originating from another node (not the publisher)
> is not replicated when origin option is local'
> +);
>
> "option" -> "parameter"

Modified

Thanks for the comments, the v18 patch attached at [1] has the fixes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0Haovukx2q7Yd987Sm8fbQ0nsh8F0EWaO_qsw0uObGBQ%40mail.gmail.com

Regards,
Vignesh

  119. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-06-06T08:58:57Z

    On Mon, Jun 6, 2022 1:14 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> The attached v18 patch has the fixes for the same.
> 

Thanks for updating the patch, here are some comments.

0002 patch
==============
1. 
+       <varlistentry>
+        <term><literal>origin</literal> (<type>string</type>)</term>
+        <listitem>
+         <para>

It maybe better if the type of  "origin" parameter is enum,  as it cannot be any
string and only has two valid values.

2.
@@ -607,6 +626,11 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
 					 LOGICALREP_TWOPHASE_STATE_PENDING :
 					 LOGICALREP_TWOPHASE_STATE_DISABLED);
 	values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
+	if (opts.origin)
+		values[Anum_pg_subscription_suborigin - 1] =
+			CStringGetTextDatum(opts.origin);
+	else
+		nulls[Anum_pg_subscription_suborigin - 1] = true;
 	values[Anum_pg_subscription_subconninfo - 1] =
 		CStringGetTextDatum(conninfo);
 	if (opts.slot_name)

Document of "CREATE SUBSCRIPTION" says, the default value of "origin" is "any", so why not set
suborigin to "any" when user doesn't specify this parameter?


0003 patch
==============
1.
@@ -300,6 +310,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
           replication from the publisher. The default is
           <literal>false</literal>.
          </para>
+         <para>
+          There is some interaction between the <literal>origin</literal>
+          parameter and <literal>copy_data</literal> parameter. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>
         </listitem>
        </varlistentry>

I think this change should be put together with "origin" parameter, instead of
"disable_on_error".


0004 patch
==============
1.
+   <para>
+    Now the bidirectional logical replication setup is complete between
+    <literal>node1</literal>, <literal>node2</literal> and
+    <literal>node2</literal>. Any subsequent changes in one node will
+    replicate the changes to the other nodes.
+   </para>

I think "node1, node2 and node2" should be "node1, node2 and node3".

Regards,
Shi yu

  120. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-08T10:51:52Z

    On Mon, Jun 6, 2022 at 2:29 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Mon, Jun 6, 2022 1:14 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v18 patch has the fixes for the same.
> >
>
> Thanks for updating the patch, here are some comments.
>
> 0002 patch
> ==============
> 1.
> +       <varlistentry>
> +        <term><literal>origin</literal> (<type>string</type>)</term>
> +        <listitem>
> +         <para>
>
> It maybe better if the type of  "origin" parameter is enum,  as it cannot be any
> string and only has two valid values.

Currently we only support local and any. But this was designed so that
it can be extended to support origin names. Users can provide a
particular origin name to be filtered.
The same was also discussed in pg unconference as mentioned in [1]

> 2.
> @@ -607,6 +626,11 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
>                                          LOGICALREP_TWOPHASE_STATE_PENDING :
>                                          LOGICALREP_TWOPHASE_STATE_DISABLED);
>         values[Anum_pg_subscription_subdisableonerr - 1] = BoolGetDatum(opts.disableonerr);
> +       if (opts.origin)
> +               values[Anum_pg_subscription_suborigin - 1] =
> +                       CStringGetTextDatum(opts.origin);
> +       else
> +               nulls[Anum_pg_subscription_suborigin - 1] = true;
>         values[Anum_pg_subscription_subconninfo - 1] =
>                 CStringGetTextDatum(conninfo);
>         if (opts.slot_name)
>
> Document of "CREATE SUBSCRIPTION" says, the default value of "origin" is "any", so why not set
> suborigin to "any" when user doesn't specify this parameter?

Modified

>
> 0003 patch
> ==============
> 1.
> @@ -300,6 +310,11 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
>            replication from the publisher. The default is
>            <literal>false</literal>.
>           </para>
> +         <para>
> +          There is some interaction between the <literal>origin</literal>
> +          parameter and <literal>copy_data</literal> parameter. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>          </listitem>
>         </varlistentry>
>
> I think this change should be put together with "origin" parameter, instead of
> "disable_on_error".

Modified

>
> 0004 patch
> ==============
> 1.
> +   <para>
> +    Now the bidirectional logical replication setup is complete between
> +    <literal>node1</literal>, <literal>node2</literal> and
> +    <literal>node2</literal>. Any subsequent changes in one node will
> +    replicate the changes to the other nodes.
> +   </para>
>
> I think "node1, node2 and node2" should be "node1, node2 and node3".

Modified

Thanks for the comments, the attached v19 patch has the changes for the same.
[1] - https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Logical_Replication_Origin_Filtering_and_Consistency

Regards,
Vignesh

  121. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-10T04:53:27Z

    Below are some review comments for the patch v18-0003

1. Commit message

This patch does a couple of things:
change 1) Checks and throws an error if the publication tables were also
subscribing data in the publisher from other publishers when copy_data
parameter is specified as 'ON' and origin parameter is specified as
'local'.
change 2) Adds force value for copy_data parameter.

SUGGESTION
This patch does a couple of things:
change 1) Checks and throws an error if 'copy_data = on' and 'origin =
local' but the publication tables were also subscribing data in the
publisher from other publishers.
change 2) Adds 'force' value for copy_data parameter.

~~~

2. Commit message - about the example

All my following review comments for the commit message are assuming
that the example steps are as they are written in the patch, but
actually I felt that the example might be more complex than it needed
to be: e.g
- You don’t really need the node2 to have data
- Therefore you don’t need all the added TRUNCATE complications

E.g. I think you only need node1 (with data) and node2 (no data).

Then node1 subscribes node2 with (origin=local, copy_data=off).
Then node2 subscribes node1 with (origin=local, copy_data=on).
- Demonstrates exception happens because node1 already had a subscription
- Demonstrates need for the copy_data=force to override that exception

So please consider using a simpler example for this commit message

~~~

3. Commit message

The below help us understand how the first change will be useful:

If copy_data parameter was used with 'on' in step 5, then an error
will be thrown
to alert the user to prevent inconsistent data being populated:
CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '<node1 details>'
PUBLICATION pub_node1 WITH (copy_data = on, origin = local);
ERROR:  CREATE/ALTER SUBSCRIPTION with origin and copy_data as true is not
allowed when the publisher might have replicated data

SUGGESTION
The steps below help to demonstrate how the new exception is useful:

The initial copy phase has no way to know the origin of the row data,
so if 'copy_data = on' in the step 5 below, then an error will be
thrown to prevent any potentially non-local data from being copied:
<blank line>
e.g
CREATE SUBSCRIPTION ...

~~~

4. Commit message

The following will help us understand how the second change will be useful:
Let's take a simple case where user is trying to setup bidirectional logical
replication between node1 and node2 where the two nodes have some pre-existing
data like below:
node1: Table t1 (c1 int) has data 11, 12, 13, 14
node2: Table t1 (c1 int) has data 21, 22, 23, 24

SUGGESTION
The following steps help to demonstrate how the 'copy_data = force'
change will be useful:
<blank line>
Let's take a scenario where the user wants to set up bidirectional
logical replication between node1 and node2 where the same table on
each node has pre-existing data. e.g.
node1: Table t1 (c1 int) has data 11, 12, 13, 14
node2: Table t1 (c1 int) has data 21, 22, 23, 24

~~~

5. Commit message

step 4:
node2=# CREATE SUBSCRIPTION sub_node2_node1 Connection '<node1 details>'
node2-# PUBLICATION pub_node1;
CREATE SUBSCRIPTION

"Connection" => "CONNECTION"

~~~

6. Commit message

If table t1 has a unique key, it will cause a unique key
violation and replication won't proceed.

SUGGESTION
In case, table t1 has a unique key, it will lead to a unique key
violation and replication won't proceed.

~~~

7. Commit message

step 3: Create a subscription in node1 to subscribe to node2. Use
copy_data specified as on so that the existing table data is copied during
initial sync:

SUGGESTION
step 3: Create a subscription in node1 to subscribe to node2. Use
'copy_data = on' so that the existing table data is copied during
initial sync:

~~~

8. Commit message

step 4: Adjust the publication publish settings so that truncate is not
published to the subscribers and truncate the table data in node2:

SUGGESTION (only added a comma)
step 4: Adjust the publication publish settings so that truncate is
not published to the subscribers, and truncate the table data in
node2:

~~~

9. Commit message

step 5: Create a subscription in node2 to subscribe to node1. Use copy_data
specified as force when creating a subscription to node1 so that the existing
table data is copied during initial sync:

SUGGESTION
step 5: Create a subscription in node2 to subscribe to node1. Use
'copy_data = force' when creating a subscription to node1 so that the
existing table data is copied during initial sync:

======

10. doc/src/sgml/ref/create_subscription.sgml

@@ -383,6 +398,15 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   If subscription is created with <literal>origin = local</literal> and
+   <literal>copy_data = on</literal>, it will check if the publisher tables are
+   being subscribed to any other publisher and throw an error to prevent
+   inconsistent data in the subscription. The user can continue with the copy
+   operation without throwing any error in this case by specifying
+   <literal>copy_data = force</literal>.
+  </para>

SUGGESTION (minor rewording)
If the subscription is created with <literal>origin = local</literal>
and <literal>copy_data = on</literal>, it will check if the publisher
tables are being subscribed to any other publisher and, if so, then
throw an error to prevent possible non-local data from being copied.
The user can override this check and continue with the copy operation
by specifying <literal>copy_data = force</literal>.

======

11. src/backend/commands/subscriptioncmds.c - parse_subscription_options

>From [1]:
>> What about also allowing copy_data = 2, and making it equivalent to "force"?
> Vignesh: I felt the existing looks ok, no need to support 2. It might confuse the user.

I don't think it would be confusing, but I also don't feel strongly
enough to debate it. Anyway, I could not find a similar precedent, so
your decision is fine.

~~~

12. src/backend/commands/subscriptioncmds.c - parse_subscription_options

@@ -339,17 +406,16 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
  errmsg("%s and %s are mutually exclusive options",
  "connect = false", "create_slot = true")));

- if (opts->copy_data &&
- IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
+ if (opts->copy_data && IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
  ereport(ERROR,

This is just a formatting change. Is it needed for this patch? patch.

~~~

13. src/backend/commands/subscriptioncmds.c - AlterSubscription_refresh

@@ -730,7 +798,8 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
  * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
  * PUBLICATION to work.
  */
- if (opts.twophase && !opts.copy_data && tables != NIL)
+ if (opts.twophase && opts.copy_data == COPY_DATA_OFF &&
+ tables != NIL)
  twophase_enabled = true;

Why is this change needed? I thought the original code is OK now since
COPY_DATA_OFF = 0

~~~

14. src/backend/commands/subscriptioncmds.c - AlterSubscription

@@ -1265,7 +1337,8 @@ AlterSubscription(ParseState *pstate,
AlterSubscriptionStmt *stmt,
  *
  * For more details see comments atop worker.c.
  */
- if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
+ opts.copy_data)

This is just a formatting change. Is it needed for this patch?

~~~

15. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

+
+ if (!origin || (strcmp(origin, "local") != 0) || copydata != COPY_DATA_ON)
+ return;
+

This condition could be rearranged to put the strcmp last so it is not
called unless absolutely necessary.

~~~

16. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

+ appendStringInfoString(&cmd,
+    "SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename,
PS.srrelid as replicated\n"
+    "FROM pg_publication P,\n"

The line is too long; needs wrapping.

~~~

17. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

+ if (!slot_attisnull(slot, 3))
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("table:%s.%s might have replicated data in the publisher",
+    nspname, relname),
+ errdetail("CREATE/ALTER SUBSCRIPTION with origin as local and
copy_data as true is not allowed when the publisher might have
replicated data."),
+ errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force."));

I felt the errmsg may be easier to read using "=" instead of "as".
Anyway, it would be more consistent with the errhint. Also, change the
"true" to "on" to be consistent with the errhint.

SUGGESTION
errdetail("CREATE/ALTER SUBSCRIPTION with origin = local and copy_data
= on is not allowed when the publisher might have replicated data."),


------
[1] https://www.postgresql.org/message-id/CALDaNm3iLLxP4OV%2ByQHs-c71P6zQ9W8D30DGsve1SQs_1pFsSQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  122. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-10T08:39:04Z

    Below are some review comments for the patch v18-0004

======

1. Commit message

Document the steps for the following:
a) Create a two-node bidirectional replication when there is no data in both the
nodes.
b) Adding a new node when there is no data in any of the nodes.
c) Adding a new node when data is present in the existing nodes.
d) Generic steps to add a new node to the existing set of nodes.

SUGGESTION (minor changes to make the tense consistent)
Documents the steps for the following:
a) Creating a two-node bidirectional replication when there is no data
in both nodes.
b) Adding a new node when there is no data in any of the nodes.
c) Adding a new node when data is present in the existing nodes.
d) Generic steps for adding a new node to an existing set of nodes.

======

2. doc/src/sgml/logical-replication.sgml - blurb

+   <para>
+    Bidirectional replication is useful in creating a multi-master database
+    which helps in performing read/write operations from any of the nodes.

SUGGESTION
Bidirectional replication is useful for creating a multi-master
database environment for replicating read/write operations performed
by any of the member nodes.

~~~

3. doc/src/sgml/logical-replication.sgml - warning

+   <warning>
+    <para>
+     Setting up bidirectional logical replication across nodes
requires multiple
+     steps to be performed on various nodes. Because not all operations are
+     transactional, the user is advised to take backups.
+    </para>
+   </warning>

SUGGESTION (but keep your formatting)
Setting up bidirectional logical replication requires multiple steps
to be performed on various nodes. Because...

~~~

4. doc/src/sgml/logical-replication.sgml - Setting bidirectional
replication between two nodes

+   <para>
+    The steps to create a two-node bidirectional replication when there is no
+    data in both the nodes <literal>node1</literal> and
+    <literal>node2</literal> are given below:
+   </para>

SUGGESTION (but keep your formatting)
The following steps demonstrate how to create a two-node bidirectional
replication when there is no table data present in both nodes node1
and node2:

~~~

5. doc/src/sgml/logical-replication.sgml - Setting bidirectional
replication between two nodes

+   <para>
+    Lock the required tables of <literal>node1</literal> and
+    <literal>node2</literal> in <literal>EXCLUSIVE</literal> mode until the
+    setup is completed.
+   </para>

Instead of "the required tables" shouldn't this just say "table t1"?

~~~

6. doc/src/sgml/logical-replication.sgml - Setting bidirectional
replication between two nodes

+   <para>
+    Now the bidirectional logical replication setup is complete between
+    <literal>node1</literal>, <literal>node2</literal> and
+    <literal>node2</literal>. Any subsequent changes in one node will
+    replicate the changes to the other nodes.
+   </para>

SUGGESTION (for 2nd sentence, and keep your formatting)
Any incremental changes from node1 will be replicated to node2, and
any incremental changes from node2 will be replicated to node1.

~~~

7. doc/src/sgml/logical-replication.sgml - Adding a new node when
there is no data in any of the nodes

+   <para>
+    Adding a new node <literal>node3</literal> to the existing
+    <literal>node1</literal> and <literal>node2</literal> when there is no data
+    in any of the nodes requires setting up subscription in
+    <literal>node1</literal> and <literal>node2</literal> to replicate the data
+    from <literal>node3</literal> and setting up subscription in
+    <literal>node3</literal> to replicate data from <literal>node1</literal>
+    and <literal>node2</literal>.
+   </para>

SUGGESTION (but keep your formatting)
The following steps demonstrate adding a new node node3 to the
existing node1 and node2 when there is no t1 data in any of the nodes.
This requires creating subscriptions in node1 and node2 to replicate
the data from node3 and creating subscriptions in node3 to replicate
data from node1 and node2.

~~~

8. doc/src/sgml/logical-replication.sgml - Adding a new node when
there is no data in any of the nodes

+   <note>
+    <para>
+     It is assumed that the bidirectional logical replication between
+     <literal>node1</literal> and <literal>node2</literal> is already
completed.
+    </para>
+   </note>

IMO this note should just be a text note in the previous paragraph
instead of an SGML <note>. e.g. "Note: These steps assume that..."

~~~

9. doc/src/sgml/logical-replication.sgml - Adding a new node when
there is no data in any of the nodes

+   <para>
+    Lock the required tables of all the nodes <literal>node1</literal>,
+    <literal>node2</literal> and <literal>node3</literal> in
+    <literal>EXCLUSIVE</literal> mode until the setup is completed.
+   </para>

Instead of "the required tables" shouldn't this just say "table t1"?

~~~

10. doc/src/sgml/logical-replication.sgml - Adding a new node when
there is no data in any of the nodes

+   <para>
+    Now the bidirectional logical replication setup is complete between
+    <literal>node1</literal>, <literal>node2</literal> and
+    <literal>node2</literal>. Any subsequent changes in one node will
+    replicate the changes to the other nodes.
+   </para>

SUGGESTION (2nd sentence)
Incremental changes made in any node will be replicated to the other two nodes.

~~~

11. doc/src/sgml/logical-replication.sgml - Adding a new node when
data is present in the existing nodes

+    <para>
+     Adding a new node <literal>node3</literal> which has no data to the
+     existing <literal>node1</literal> and <literal>node2</literal> when data
+     is present in existing nodes <literal>node1</literal> and
+     <literal>node2</literal> needs similar steps. The only change required
+     here is that <literal>node3</literal> should create a subscription with
+     <literal>copy_data = force</literal> to one of the existing nodes to
+     receive the existing data during initial data synchronization.
+   </para>

SUGGESTION (but keep your formatting)
The following steps demonstrate adding a new node node3 which has no
t1 data to the existing node1 and node2 where t1 data is present. This
needs similar steps; the only change required here is that node3
should create a subscription with copy_data = force to one of the
existing nodes so it can receive the existing t1 data during initial
data synchronization.

~~~

12 doc/src/sgml/logical-replication.sgml - Adding a new node when data
is present in the existing nodes

+   <note>
+    <para>
+     It is assumed that the bidirectional logical replication between
+     <literal>node1</literal> and <literal>node2</literal> is already
completed.
+     The nodes <literal>node1</literal> and <literal>node2</literal> has some
+     pre-existing data in table t1 that is synchronized in both the nodes.
+    </para>
+   </note>

12a.
IMO this note should just be text in the previous paragraph instead of
an SGML <note>. e.g. "Note: These steps assume that..."

12b.
SUGGESTION (minor rewording; keep your formatting)
Note: These steps assume that the bidirectional logical replication
between node1 and node2 is already completed, and the pre-existing
data in table t1 is already synchronized in both those nodes.

~~~

13. doc/src/sgml/logical-replication.sgml - Adding a new node when
data is present in the existing nodes

+   <para>
+    Lock the required tables of <literal>node2</literal> and
+    <literal>node3</literal> in <literal>EXCLUSIVE</literal> mode until the
+    setup is completed. No need to lock the tables in <literal>node1</literal>
+    as any data changes made will be synchronized while creating the
+    subscription with <literal>copy_data</literal> specified as
+    <literal>force</literal>.
+   </para>

13a.
Instead of "the required tables" shouldn't this just say "table t1"?

13b.
SUGGESTION (2nd sentence; keep your formatting)
There is no need to lock table t1 in node1 because any data changes
made will be synchronized while creating the subscription with
copy_data = force.

~~~

14. doc/src/sgml/logical-replication.sgml - Adding a new node when
data is present in the existing nodes

+   <para>
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node1</literal>. Use <literal>copy_data</literal> specified as
+    <literal>force</literal> so that the existing table data is
+    copied during initial sync:

SUGGESTION (2nd sentence; keep your formatting)
Use copy_data = force so that the existing table data is copied during
the initial sync:

~~~

15. doc/src/sgml/logical-replication.sgml - Adding a new node when
data is present in the existing nodes

+   <para>
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node2</literal>. Use <literal>copy_data</literal> specified as
+    <literal>off</literal> because the initial table data would have been
+    already copied in the previous step:

SUGGESTION (2nd sentence; keep your formatting)
Use copy_data = off because the initial table data would have been
already copied in the previous step:

~~~

16. doc/src/sgml/logical-replication.sgml - Adding a new node when
data is present in the existing nodes

+   <para>
+    Now the bidirectional logical replication setup is complete between
+    <literal>node1</literal>, <literal>node2</literal> and
+    <literal>node2</literal>. Any subsequent changes in one node will
+    replicate the changes to the other nodes.
+   </para>

16a.
Should say node1, node2 and node3

16b.
SUGGESTION (2nd sentence – same as the previous comment)
Incremental changes made in any node will be replicated to the other two nodes.

~~~

17. doc/src/sgml/logical-replication.sgml - Adding a new node when
data is present in the new node

+   <warning>
+    <para>
+     Adding a new node <literal>node3</literal> to the existing
+     <literal>node1</literal> and <literal>node2</literal> when data is present
+     in the new node <literal>node3</literal> is not possible.
+    </para>
+   </warning>

17a.
IMO
- Not really necessary to name the nodes, because this is a generic statement
- Maybe say "not supported" instead of "not possible". e.g. there is
no ERROR check for this case is there?

SUGGESTION
Adding a new node when data is present in the new node tables is not supported.

17b.
I am not sure but I felt this advice seemed more like an SGML <note>;
note a <warning>

~~~

18. doc/src/sgml/logical-replication.sgml - Generic steps to add a new
node to the existing set of nodes

+   <title>Generic steps to add a new node to the existing set of nodes</title>

SUGGESTION
Generic steps for adding a new node to an existing set of nodes

~~~

19. doc/src/sgml/logical-replication.sgml - Generic steps to add a new
node to the existing set of nodes

+   <para>
+    1. Create the required publication on the new node.
+   </para>
+   <para>
+    2. Lock the required tables of the new node in <literal>EXCLUSIVE</literal>
+    mode until the setup is complete. This is required to prevent any
+    modifications from happening in the new node. If data modifications occur
+    after step-3, there is a chance that the modifications will be published to
+    the first node and then synchronized back to the new node while creating
+    subscription in step-5 resulting in inconsistent data.
+   </para>
+   <para>
+    3. Create subscriptions on existing nodes pointing to publication on
+    the new node with <literal>origin</literal> parameter specified as
+    <literal>local</literal> and <literal>copy_data</literal> specified as
+    <literal>off</literal>.
+   </para>
+   <para>
+    4. Lock the required tables of the existing nodes except the first node in
+    <literal>EXCLUSIVE</literal> mode until the setup is complete. This is
+    required to prevent any modifications from happening. If data modifications
+    occur, there is a chance that the modifications done in between step-5 and
+    step-6 will not be synchronized to the new node resulting in inconsistent
+    data. No need to lock the tables in the first node as any data changes
+    made will be synchronized while creating the subscription with
+    <literal>copy_data</literal> specified as <literal>force</literal>.
+   </para>
+   <para>
+    5. Create a subscription on the new node pointing to publication on the
+    first node with <literal>origin</literal> parameter specified as
+    <literal>local</literal> and <literal>copy_data</literal> parameter
+    specified as <literal>force</literal>.
+   </para>
+   <para>
+    6. Create subscriptions on the new node pointing to publications on the
+    remaining nodes with <literal>origin</literal> parameter specified as
+    <literal>local</literal> and <literal>copy_data</literal> parameter
+    specifiedas <literal>off</literal>.
+   </para>

Following suggestions make the following changes:
- Some minor rewording
- Change the step names
- Use "=" instead of "specified as" for the parameters
- I felt it is more readable if the explanatory notes are separate
paragraphs from the steps. Maybe if you could indent them or something
it would be even better.
- Added a couple more explanatory notes

SUGGESTION (using those above changes; please keep your formatting)

Step-1: Create a publication on the new node.

Step-2: Lock the required tables of the new node in EXCLUSIVE mode
until the setup is complete.

(This lock is necessary to prevent any modifications from happening in
the new node because if data modifications occurred after Step-3,
there is a chance that the modifications will be published to the
first node and then synchronized back to the new node while creating
the subscription in Step-5. This would result in inconsistent data).

Step-3. Create subscriptions on existing nodes to the publication on
the new node with origin = local and copy_data = off.

(The copy_data = off is OK here because it is asserted that the
published tables of the new node will have no pre-existing data).

Step-4. Lock the required tables of the existing nodes except the
first node in EXCLUSIVE mode until the setup is complete.

(This lock is necessary to prevent any modifications from happening.
If data modifications occur, there is a chance that modifications done
between Step-5 and Step-6 will not be synchronized to the new node.
This would result in inconsistent data. There is no need to lock table
t1 in node1 because any data changes made will be synchronized while
creating the subscription with copy_data = force).

Step-5. Create a subscription on the new node to the publication on
the first node with origin = local and copy_data = force.

(This will copy the same table data from the existing nodes to the new node)

Step-6. Create subscriptions on the new node to publications on the
remaining nodes with origin = local and copy_data  = off.

(The copy_data = off is OK here because the existing node data was
already copied to the new node in Step-5)

======

20. doc/src/sgml/ref/create_subscription.sgml

-   <literal>copy_data = force</literal>.
+   <literal>copy_data = force</literal>. Refer to the
+   <xref linkend="logical-replication-bidirectional"/> on how
+   <literal>copy_data</literal> and <literal>origin</literal> can be used
+   in bidirectional replication.
   </para>

SUGGESTION (keep your formatting)
Refer to <xref linkend="logical-replication-bidirectional"/> for how
<literal>copy_data</literal> and <literal>origin</literal> can be used
in bidirectional replication.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  123. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-14T12:04:52Z

    On Fri, Jun 10, 2022 at 10:23 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Below are some review comments for the patch v18-0003
>
> 1. Commit message
>
> This patch does a couple of things:
> change 1) Checks and throws an error if the publication tables were also
> subscribing data in the publisher from other publishers when copy_data
> parameter is specified as 'ON' and origin parameter is specified as
> 'local'.
> change 2) Adds force value for copy_data parameter.
>
> SUGGESTION
> This patch does a couple of things:
> change 1) Checks and throws an error if 'copy_data = on' and 'origin =
> local' but the publication tables were also subscribing data in the
> publisher from other publishers.
> change 2) Adds 'force' value for copy_data parameter.

Modified

> ~~~
>
> 2. Commit message - about the example
>
> All my following review comments for the commit message are assuming
> that the example steps are as they are written in the patch, but
> actually I felt that the example might be more complex than it needed
> to be: e.g
> - You don’t really need the node2 to have data
> - Therefore you don’t need all the added TRUNCATE complications
>
> E.g. I think you only need node1 (with data) and node2 (no data).
>
> Then node1 subscribes node2 with (origin=local, copy_data=off).
> Then node2 subscribes node1 with (origin=local, copy_data=on).
> - Demonstrates exception happens because node1 already had a subscription
> - Demonstrates need for the copy_data=force to override that exception
>
> So please consider using a simpler example for this commit message

Modified

> ~~~
>
> 3. Commit message
>
> The below help us understand how the first change will be useful:
>
> If copy_data parameter was used with 'on' in step 5, then an error
> will be thrown
> to alert the user to prevent inconsistent data being populated:
> CREATE SUBSCRIPTION sub_node2_node1 CONNECTION '<node1 details>'
> PUBLICATION pub_node1 WITH (copy_data = on, origin = local);
> ERROR:  CREATE/ALTER SUBSCRIPTION with origin and copy_data as true is not
> allowed when the publisher might have replicated data
>
> SUGGESTION
> The steps below help to demonstrate how the new exception is useful:
>
> The initial copy phase has no way to know the origin of the row data,
> so if 'copy_data = on' in the step 5 below, then an error will be
> thrown to prevent any potentially non-local data from being copied:
> <blank line>
> e.g
> CREATE SUBSCRIPTION ...

Modified

> ~~~
>
> 4. Commit message
>
> The following will help us understand how the second change will be useful:
> Let's take a simple case where user is trying to setup bidirectional logical
> replication between node1 and node2 where the two nodes have some pre-existing
> data like below:
> node1: Table t1 (c1 int) has data 11, 12, 13, 14
> node2: Table t1 (c1 int) has data 21, 22, 23, 24
>
> SUGGESTION
> The following steps help to demonstrate how the 'copy_data = force'
> change will be useful:
> <blank line>
> Let's take a scenario where the user wants to set up bidirectional
> logical replication between node1 and node2 where the same table on
> each node has pre-existing data. e.g.
> node1: Table t1 (c1 int) has data 11, 12, 13, 14
> node2: Table t1 (c1 int) has data 21, 22, 23, 24

Modified

> ~~~
>
> 5. Commit message
>
> step 4:
> node2=# CREATE SUBSCRIPTION sub_node2_node1 Connection '<node1 details>'
> node2-# PUBLICATION pub_node1;
> CREATE SUBSCRIPTION
>
> "Connection" => "CONNECTION"

Modified

> ~~~
>
> 6. Commit message
>
> If table t1 has a unique key, it will cause a unique key
> violation and replication won't proceed.
>
> SUGGESTION
> In case, table t1 has a unique key, it will lead to a unique key
> violation and replication won't proceed.

Modified

> ~~~
>
> 7. Commit message
>
> step 3: Create a subscription in node1 to subscribe to node2. Use
> copy_data specified as on so that the existing table data is copied during
> initial sync:
>
> SUGGESTION
> step 3: Create a subscription in node1 to subscribe to node2. Use
> 'copy_data = on' so that the existing table data is copied during
> initial sync:

Modified

> ~~~
>
> 8. Commit message
>
> step 4: Adjust the publication publish settings so that truncate is not
> published to the subscribers and truncate the table data in node2:
>
> SUGGESTION (only added a comma)
> step 4: Adjust the publication publish settings so that truncate is
> not published to the subscribers, and truncate the table data in
> node2:

This content is not required any more, I have removed it.

> ~~~
>
> 9. Commit message
>
> step 5: Create a subscription in node2 to subscribe to node1. Use copy_data
> specified as force when creating a subscription to node1 so that the existing
> table data is copied during initial sync:
>
> SUGGESTION
> step 5: Create a subscription in node2 to subscribe to node1. Use
> 'copy_data = force' when creating a subscription to node1 so that the
> existing table data is copied during initial sync:

Modified

> ======
>
> 10. doc/src/sgml/ref/create_subscription.sgml
>
> @@ -383,6 +398,15 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   If subscription is created with <literal>origin = local</literal> and
> +   <literal>copy_data = on</literal>, it will check if the publisher tables are
> +   being subscribed to any other publisher and throw an error to prevent
> +   inconsistent data in the subscription. The user can continue with the copy
> +   operation without throwing any error in this case by specifying
> +   <literal>copy_data = force</literal>.
> +  </para>
>
> SUGGESTION (minor rewording)
> If the subscription is created with <literal>origin = local</literal>
> and <literal>copy_data = on</literal>, it will check if the publisher
> tables are being subscribed to any other publisher and, if so, then
> throw an error to prevent possible non-local data from being copied.
> The user can override this check and continue with the copy operation
> by specifying <literal>copy_data = force</literal>.

Modified

> ======
>
> 11. src/backend/commands/subscriptioncmds.c - parse_subscription_options
>
> From [1]:
> >> What about also allowing copy_data = 2, and making it equivalent to "force"?
> > Vignesh: I felt the existing looks ok, no need to support 2. It might confuse the user.
>
> I don't think it would be confusing, but I also don't feel strongly
> enough to debate it. Anyway, I could not find a similar precedent, so
> your decision is fine.

Ok

> ~~~
>
> 12. src/backend/commands/subscriptioncmds.c - parse_subscription_options
>
> @@ -339,17 +406,16 @@ parse_subscription_options(ParseState *pstate,
> List *stmt_options,
>   errmsg("%s and %s are mutually exclusive options",
>   "connect = false", "create_slot = true")));
>
> - if (opts->copy_data &&
> - IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
> + if (opts->copy_data && IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
>   ereport(ERROR,
>
> This is just a formatting change. Is it needed for this patch? patch.

Modified

> ~~~
>
> 13. src/backend/commands/subscriptioncmds.c - AlterSubscription_refresh
>
> @@ -730,7 +798,8 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>   * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
>   * PUBLICATION to work.
>   */
> - if (opts.twophase && !opts.copy_data && tables != NIL)
> + if (opts.twophase && opts.copy_data == COPY_DATA_OFF &&
> + tables != NIL)
>   twophase_enabled = true;
>
> Why is this change needed? I thought the original code is OK now since
> COPY_DATA_OFF = 0

Modified

> ~~~
>
> 14. src/backend/commands/subscriptioncmds.c - AlterSubscription
>
> @@ -1265,7 +1337,8 @@ AlterSubscription(ParseState *pstate,
> AlterSubscriptionStmt *stmt,
>   *
>   * For more details see comments atop worker.c.
>   */
> - if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED && opts.copy_data)
> + if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED &&
> + opts.copy_data)
>
> This is just a formatting change. Is it needed for this patch?

Modified

> ~~~
>
> 15. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> +
> + if (!origin || (strcmp(origin, "local") != 0) || copydata != COPY_DATA_ON)
> + return;
> +
>
> This condition could be rearranged to put the strcmp last so it is not
> called unless absolutely necessary.

Modified

> ~~~
>
> 16. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + appendStringInfoString(&cmd,
> +    "SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename,
> PS.srrelid as replicated\n"
> +    "FROM pg_publication P,\n"
>
> The line is too long; needs wrapping.

Modified

> ~~~
>
> 17. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + if (!slot_attisnull(slot, 3))
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("table:%s.%s might have replicated data in the publisher",
> +    nspname, relname),
> + errdetail("CREATE/ALTER SUBSCRIPTION with origin as local and
> copy_data as true is not allowed when the publisher might have
> replicated data."),
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force."));
>
> I felt the errmsg may be easier to read using "=" instead of "as".
> Anyway, it would be more consistent with the errhint. Also, change the
> "true" to "on" to be consistent with the errhint.
>
> SUGGESTION
> errdetail("CREATE/ALTER SUBSCRIPTION with origin = local and copy_data
> = on is not allowed when the publisher might have replicated data."),

Modified

Thanks for the comments, the attached v20 patch has the changes for the same.

Regards,
Vignesh

  124. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-14T12:08:50Z

    On Fri, Jun 10, 2022 at 2:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Below are some review comments for the patch v18-0004
>
> ======
>
> 1. Commit message
>
> Document the steps for the following:
> a) Create a two-node bidirectional replication when there is no data in both the
> nodes.
> b) Adding a new node when there is no data in any of the nodes.
> c) Adding a new node when data is present in the existing nodes.
> d) Generic steps to add a new node to the existing set of nodes.
>
> SUGGESTION (minor changes to make the tense consistent)
> Documents the steps for the following:
> a) Creating a two-node bidirectional replication when there is no data
> in both nodes.
> b) Adding a new node when there is no data in any of the nodes.
> c) Adding a new node when data is present in the existing nodes.
> d) Generic steps for adding a new node to an existing set of nodes.

Modified

> ======
>
> 2. doc/src/sgml/logical-replication.sgml - blurb
>
> +   <para>
> +    Bidirectional replication is useful in creating a multi-master database
> +    which helps in performing read/write operations from any of the nodes.
>
> SUGGESTION
> Bidirectional replication is useful for creating a multi-master
> database environment for replicating read/write operations performed
> by any of the member nodes.

Modified

> ~~~
>
> 3. doc/src/sgml/logical-replication.sgml - warning
>
> +   <warning>
> +    <para>
> +     Setting up bidirectional logical replication across nodes
> requires multiple
> +     steps to be performed on various nodes. Because not all operations are
> +     transactional, the user is advised to take backups.
> +    </para>
> +   </warning>
>
> SUGGESTION (but keep your formatting)
> Setting up bidirectional logical replication requires multiple steps
> to be performed on various nodes. Because...

Modified

> ~~~
>
> 4. doc/src/sgml/logical-replication.sgml - Setting bidirectional
> replication between two nodes
>
> +   <para>
> +    The steps to create a two-node bidirectional replication when there is no
> +    data in both the nodes <literal>node1</literal> and
> +    <literal>node2</literal> are given below:
> +   </para>
>
> SUGGESTION (but keep your formatting)
> The following steps demonstrate how to create a two-node bidirectional
> replication when there is no table data present in both nodes node1
> and node2:

Modified

> ~~~
>
> 5. doc/src/sgml/logical-replication.sgml - Setting bidirectional
> replication between two nodes
>
> +   <para>
> +    Lock the required tables of <literal>node1</literal> and
> +    <literal>node2</literal> in <literal>EXCLUSIVE</literal> mode until the
> +    setup is completed.
> +   </para>
>
> Instead of "the required tables" shouldn't this just say "table t1"?

Modified

> ~~~
>
> 6. doc/src/sgml/logical-replication.sgml - Setting bidirectional
> replication between two nodes
>
> +   <para>
> +    Now the bidirectional logical replication setup is complete between
> +    <literal>node1</literal>, <literal>node2</literal> and
> +    <literal>node2</literal>. Any subsequent changes in one node will
> +    replicate the changes to the other nodes.
> +   </para>
>
> SUGGESTION (for 2nd sentence, and keep your formatting)
> Any incremental changes from node1 will be replicated to node2, and
> any incremental changes from node2 will be replicated to node1.

Modified

> ~~~
>
> 7. doc/src/sgml/logical-replication.sgml - Adding a new node when
> there is no data in any of the nodes
>
> +   <para>
> +    Adding a new node <literal>node3</literal> to the existing
> +    <literal>node1</literal> and <literal>node2</literal> when there is no data
> +    in any of the nodes requires setting up subscription in
> +    <literal>node1</literal> and <literal>node2</literal> to replicate the data
> +    from <literal>node3</literal> and setting up subscription in
> +    <literal>node3</literal> to replicate data from <literal>node1</literal>
> +    and <literal>node2</literal>.
> +   </para>
>
> SUGGESTION (but keep your formatting)
> The following steps demonstrate adding a new node node3 to the
> existing node1 and node2 when there is no t1 data in any of the nodes.
> This requires creating subscriptions in node1 and node2 to replicate
> the data from node3 and creating subscriptions in node3 to replicate
> data from node1 and node2.

Modified

> ~~~
>
> 8. doc/src/sgml/logical-replication.sgml - Adding a new node when
> there is no data in any of the nodes
>
> +   <note>
> +    <para>
> +     It is assumed that the bidirectional logical replication between
> +     <literal>node1</literal> and <literal>node2</literal> is already
> completed.
> +    </para>
> +   </note>
>
> IMO this note should just be a text note in the previous paragraph
> instead of an SGML <note>. e.g. "Note: These steps assume that..."

Modified

> ~~~
>
> 9. doc/src/sgml/logical-replication.sgml - Adding a new node when
> there is no data in any of the nodes
>
> +   <para>
> +    Lock the required tables of all the nodes <literal>node1</literal>,
> +    <literal>node2</literal> and <literal>node3</literal> in
> +    <literal>EXCLUSIVE</literal> mode until the setup is completed.
> +   </para>
>
> Instead of "the required tables" shouldn't this just say "table t1"?

Modified

> ~~~
>
> 10. doc/src/sgml/logical-replication.sgml - Adding a new node when
> there is no data in any of the nodes
>
> +   <para>
> +    Now the bidirectional logical replication setup is complete between
> +    <literal>node1</literal>, <literal>node2</literal> and
> +    <literal>node2</literal>. Any subsequent changes in one node will
> +    replicate the changes to the other nodes.
> +   </para>
>
> SUGGESTION (2nd sentence)
> Incremental changes made in any node will be replicated to the other two nodes.

Modified

> ~~~
>
> 11. doc/src/sgml/logical-replication.sgml - Adding a new node when
> data is present in the existing nodes
>
> +    <para>
> +     Adding a new node <literal>node3</literal> which has no data to the
> +     existing <literal>node1</literal> and <literal>node2</literal> when data
> +     is present in existing nodes <literal>node1</literal> and
> +     <literal>node2</literal> needs similar steps. The only change required
> +     here is that <literal>node3</literal> should create a subscription with
> +     <literal>copy_data = force</literal> to one of the existing nodes to
> +     receive the existing data during initial data synchronization.
> +   </para>
>
> SUGGESTION (but keep your formatting)
> The following steps demonstrate adding a new node node3 which has no
> t1 data to the existing node1 and node2 where t1 data is present. This
> needs similar steps; the only change required here is that node3
> should create a subscription with copy_data = force to one of the
> existing nodes so it can receive the existing t1 data during initial
> data synchronization.

Modified

> ~~~
>
> 12 doc/src/sgml/logical-replication.sgml - Adding a new node when data
> is present in the existing nodes
>
> +   <note>
> +    <para>
> +     It is assumed that the bidirectional logical replication between
> +     <literal>node1</literal> and <literal>node2</literal> is already
> completed.
> +     The nodes <literal>node1</literal> and <literal>node2</literal> has some
> +     pre-existing data in table t1 that is synchronized in both the nodes.
> +    </para>
> +   </note>
>
> 12a.
> IMO this note should just be text in the previous paragraph instead of
> an SGML <note>. e.g. "Note: These steps assume that..."

Modified

> 12b.
> SUGGESTION (minor rewording; keep your formatting)
> Note: These steps assume that the bidirectional logical replication
> between node1 and node2 is already completed, and the pre-existing
> data in table t1 is already synchronized in both those nodes.

Modified

> ~~~
>
> 13. doc/src/sgml/logical-replication.sgml - Adding a new node when
> data is present in the existing nodes
>
> +   <para>
> +    Lock the required tables of <literal>node2</literal> and
> +    <literal>node3</literal> in <literal>EXCLUSIVE</literal> mode until the
> +    setup is completed. No need to lock the tables in <literal>node1</literal>
> +    as any data changes made will be synchronized while creating the
> +    subscription with <literal>copy_data</literal> specified as
> +    <literal>force</literal>.
> +   </para>
>
> 13a.
> Instead of "the required tables" shouldn't this just say "table t1"?

Modified

> 13b.
> SUGGESTION (2nd sentence; keep your formatting)
> There is no need to lock table t1 in node1 because any data changes
> made will be synchronized while creating the subscription with
> copy_data = force.

Modified

> ~~~
>
> 14. doc/src/sgml/logical-replication.sgml - Adding a new node when
> data is present in the existing nodes
>
> +   <para>
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node1</literal>. Use <literal>copy_data</literal> specified as
> +    <literal>force</literal> so that the existing table data is
> +    copied during initial sync:
>
> SUGGESTION (2nd sentence; keep your formatting)
> Use copy_data = force so that the existing table data is copied during
> the initial sync:

Modified

> ~~~
>
> 15. doc/src/sgml/logical-replication.sgml - Adding a new node when
> data is present in the existing nodes
>
> +   <para>
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node2</literal>. Use <literal>copy_data</literal> specified as
> +    <literal>off</literal> because the initial table data would have been
> +    already copied in the previous step:
>
> SUGGESTION (2nd sentence; keep your formatting)
> Use copy_data = off because the initial table data would have been
> already copied in the previous step:

Modified

> ~~~
>
> 16. doc/src/sgml/logical-replication.sgml - Adding a new node when
> data is present in the existing nodes
>
> +   <para>
> +    Now the bidirectional logical replication setup is complete between
> +    <literal>node1</literal>, <literal>node2</literal> and
> +    <literal>node2</literal>. Any subsequent changes in one node will
> +    replicate the changes to the other nodes.
> +   </para>
>
> 16a.
> Should say node1, node2 and node3

Modified it in v19

> 16b.
> SUGGESTION (2nd sentence – same as the previous comment)
> Incremental changes made in any node will be replicated to the other two nodes.

Modified

> ~~~
>
> 17. doc/src/sgml/logical-replication.sgml - Adding a new node when
> data is present in the new node
>
> +   <warning>
> +    <para>
> +     Adding a new node <literal>node3</literal> to the existing
> +     <literal>node1</literal> and <literal>node2</literal> when data is present
> +     in the new node <literal>node3</literal> is not possible.
> +    </para>
> +   </warning>
>
> 17a.
> IMO
> - Not really necessary to name the nodes, because this is a generic statement
> - Maybe say "not supported" instead of "not possible". e.g. there is
> no ERROR check for this case is there?
>
> SUGGESTION
> Adding a new node when data is present in the new node tables is not supported.

Modified

> 17b.
> I am not sure but I felt this advice seemed more like an SGML <note>;
> note a <warning>

Modified

> ~~~
>
> 18. doc/src/sgml/logical-replication.sgml - Generic steps to add a new
> node to the existing set of nodes
>
> +   <title>Generic steps to add a new node to the existing set of nodes</title>
>
> SUGGESTION
> Generic steps for adding a new node to an existing set of nodes

Modified

> ~~~
>
> 19. doc/src/sgml/logical-replication.sgml - Generic steps to add a new
> node to the existing set of nodes
>
> +   <para>
> +    1. Create the required publication on the new node.
> +   </para>
> +   <para>
> +    2. Lock the required tables of the new node in <literal>EXCLUSIVE</literal>
> +    mode until the setup is complete. This is required to prevent any
> +    modifications from happening in the new node. If data modifications occur
> +    after step-3, there is a chance that the modifications will be published to
> +    the first node and then synchronized back to the new node while creating
> +    subscription in step-5 resulting in inconsistent data.
> +   </para>
> +   <para>
> +    3. Create subscriptions on existing nodes pointing to publication on
> +    the new node with <literal>origin</literal> parameter specified as
> +    <literal>local</literal> and <literal>copy_data</literal> specified as
> +    <literal>off</literal>.
> +   </para>
> +   <para>
> +    4. Lock the required tables of the existing nodes except the first node in
> +    <literal>EXCLUSIVE</literal> mode until the setup is complete. This is
> +    required to prevent any modifications from happening. If data modifications
> +    occur, there is a chance that the modifications done in between step-5 and
> +    step-6 will not be synchronized to the new node resulting in inconsistent
> +    data. No need to lock the tables in the first node as any data changes
> +    made will be synchronized while creating the subscription with
> +    <literal>copy_data</literal> specified as <literal>force</literal>.
> +   </para>
> +   <para>
> +    5. Create a subscription on the new node pointing to publication on the
> +    first node with <literal>origin</literal> parameter specified as
> +    <literal>local</literal> and <literal>copy_data</literal> parameter
> +    specified as <literal>force</literal>.
> +   </para>
> +   <para>
> +    6. Create subscriptions on the new node pointing to publications on the
> +    remaining nodes with <literal>origin</literal> parameter specified as
> +    <literal>local</literal> and <literal>copy_data</literal> parameter
> +    specifiedas <literal>off</literal>.
> +   </para>
>
> Following suggestions make the following changes:
> - Some minor rewording
> - Change the step names
> - Use "=" instead of "specified as" for the parameters
> - I felt it is more readable if the explanatory notes are separate
> paragraphs from the steps. Maybe if you could indent them or something
> it would be even better.
> - Added a couple more explanatory notes
>
> SUGGESTION (using those above changes; please keep your formatting)
>
> Step-1: Create a publication on the new node.
>
> Step-2: Lock the required tables of the new node in EXCLUSIVE mode
> until the setup is complete.
>
> (This lock is necessary to prevent any modifications from happening in
> the new node because if data modifications occurred after Step-3,
> there is a chance that the modifications will be published to the
> first node and then synchronized back to the new node while creating
> the subscription in Step-5. This would result in inconsistent data).
>
> Step-3. Create subscriptions on existing nodes to the publication on
> the new node with origin = local and copy_data = off.
>
> (The copy_data = off is OK here because it is asserted that the
> published tables of the new node will have no pre-existing data).
>
> Step-4. Lock the required tables of the existing nodes except the
> first node in EXCLUSIVE mode until the setup is complete.
>
> (This lock is necessary to prevent any modifications from happening.
> If data modifications occur, there is a chance that modifications done
> between Step-5 and Step-6 will not be synchronized to the new node.
> This would result in inconsistent data. There is no need to lock table
> t1 in node1 because any data changes made will be synchronized while
> creating the subscription with copy_data = force).
>
> Step-5. Create a subscription on the new node to the publication on
> the first node with origin = local and copy_data = force.
>
> (This will copy the same table data from the existing nodes to the new node)
>
> Step-6. Create subscriptions on the new node to publications on the
> remaining nodes with origin = local and copy_data  = off.
>
> (The copy_data = off is OK here because the existing node data was
> already copied to the new node in Step-5)

Modified

> ======
>
> 20. doc/src/sgml/ref/create_subscription.sgml
>
> -   <literal>copy_data = force</literal>.
> +   <literal>copy_data = force</literal>. Refer to the
> +   <xref linkend="logical-replication-bidirectional"/> on how
> +   <literal>copy_data</literal> and <literal>origin</literal> can be used
> +   in bidirectional replication.
>    </para>
>
> SUGGESTION (keep your formatting)
> Refer to <xref linkend="logical-replication-bidirectional"/> for how
> <literal>copy_data</literal> and <literal>origin</literal> can be used
> in bidirectional replication.

Modified

Thanks for the comments. The comments are handled as part of the v20
patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0XtQVX3taeLKWE-gPQyppqs34ipXawAPOyO%3Dhe37MQSg%40mail.gmail.com

Regards,
Vignesh

  125. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-15T06:39:29Z

    Here are some review comments for patch v20-0002

======

1. General comment

Something subtle but significant changed since I last reviewed v18*.
Now the describe.c is changed so that the catalog will never display a
NULL origin column; it would always be "any". But now I am not sure if
it is a good idea to still allow the NULL in this catalog column while
at the same time you are pretending it is not there. I felt it might
be less confusing, and would simplify the code (e.g. remove quite a
few null checks) to have just used a single concept of the default -
e.g. Just assign the default as "any" everywhere. The column would be
defined as NOT NULL. Most of the following review comments are related
to this point.

======

2. doc/src/sgml/catalogs.sgml

+      <para>
+       Possible origin values are <literal>local</literal>,
+       <literal>any</literal>, or <literal>NULL</literal> if none is specified.
+       If <literal>local</literal>, the subscription will request the
+       publisher to only send changes that originated locally. If
+       <literal>any</literal> (or <literal>NULL</literal>), the publisher sends
+       any changes regardless of their origin.
+      </para></entry>

Is NULL still possible? Perhaps it would be better if it was not and
the default "any" was always written instead.

======

3. src/backend/catalog/pg_subscription.c

+ if (!isnull)
+ sub->origin = TextDatumGetCString(datum);
+ else
+ sub->origin = NULL;
+

Maybe better to either disallow NULL in the first place or assign the
"any" here instead of NULL.

======

4. src/backend/commands/subscriptioncmds.c - parse_subscription_options

@@ -137,6 +139,8 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
  opts->twophase = false;
  if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
  opts->disableonerr = false;
+ if (IsSet(supported_opts, SUBOPT_ORIGIN))
+ opts->origin = NULL;

If opt->origin was assigned to "any" then other code would be simplified.

~~~

5. src/backend/commands/subscriptioncmds.c - CreateSubscription

@@ -607,6 +626,11 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
  LOGICALREP_TWOPHASE_STATE_PENDING :
  LOGICALREP_TWOPHASE_STATE_DISABLED);
  values[Anum_pg_subscription_subdisableonerr - 1] =
BoolGetDatum(opts.disableonerr);
+ if (opts.origin)
+ values[Anum_pg_subscription_suborigin - 1] =
+ CStringGetTextDatum(opts.origin);
+ else
+ values[Anum_pg_subscription_suborigin - 1] =
CStringGetTextDatum(LOGICALREP_ORIGIN_ANY);

If NULL was not possible then this would just be one line:
values[Anum_pg_subscription_suborigin - 1] = CStringGetTextDatum(opts.origin);

======

6. src/backend/replication/logical/worker.c

@@ -276,6 +276,10 @@ static TransactionId stream_xid = InvalidTransactionId;
 static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr;
 #define is_skipping_changes()
(unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn)))

+/* Macro for comparing string fields that might be NULL */
+#define equalstr(a, b) \
+ (((a) != NULL && (b) != NULL) ? (strcmp((a), (b)) == 0) : (a) == (b))
+

If the NULL was not allowed in the first place then I think this macro
would just become redundant.

7. src/backend/replication/logical/worker.c - ApplyWorkerMain

@@ -3741,6 +3746,11 @@ ApplyWorkerMain(Datum main_arg)
  options.proto.logical.streaming = MySubscription->stream;
  options.proto.logical.twophase = false;

+ if (MySubscription->origin)
+ options.proto.logical.origin = pstrdup(MySubscription->origin);
+ else
+ options.proto.logical.origin = NULL;
+

Can't the if/else be avoided if you always assigned the "any" default
in the first place?

======

8. src/backend/replication/pgoutput/pgoutput.c - parse_output_parameters

@@ -287,11 +289,13 @@ parse_output_parameters(List *options, PGOutputData *data)
  bool messages_option_given = false;
  bool streaming_given = false;
  bool two_phase_option_given = false;
+ bool origin_option_given = false;

  data->binary = false;
  data->streaming = false;
  data->messages = false;
  data->two_phase = false;
+ data->origin = NULL;

Consider assigning default "any" here instead of NULL.

======

9. src/bin/pg_dump/pg_dump.c - getSubscriptions

+ /* FIXME: 150000 should be changed to 160000 later for PG16. */
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, " s.suborigin\n");
+ else
+ appendPQExpBufferStr(query, " NULL AS suborigin\n");
+

Maybe say: 'any' AS suborigin?

~~~

10. src/bin/pg_dump/pg_dump.c - getSubscriptions

@@ -4517,6 +4525,11 @@ getSubscriptions(Archive *fout)
  subinfo[i].subdisableonerr =
  pg_strdup(PQgetvalue(res, i, i_subdisableonerr));

+ if (PQgetisnull(res, i, i_suborigin))
+ subinfo[i].suborigin = NULL;
+ else
+ subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
+

If you disallow the NULL in the first place this condition maybe is no
longer needed.

~~~

11. src/bin/pg_dump/pg_dump.c - dumpSubscription

@@ -4589,6 +4602,9 @@ dumpSubscription(Archive *fout, const
SubscriptionInfo *subinfo)
  if (strcmp(subinfo->subdisableonerr, "t") == 0)
  appendPQExpBufferStr(query, ", disable_on_error = true");

+ if (subinfo->suborigin)
+ appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+

If NULL cannot happen then maybe this test is also redundant.

======

12. src/bin/pg_dump/t/002_pg_dump.pl

AFAICT there is a test for no origin (default), and a test for
explicit origin = local, but there is no test case for the explicit
origin = any.

======

13. src/include/catalog/pg_subscription.h

@@ -31,6 +31,9 @@
 #define LOGICALREP_TWOPHASE_STATE_PENDING 'p'
 #define LOGICALREP_TWOPHASE_STATE_ENABLED 'e'

+#define LOGICALREP_ORIGIN_LOCAL "local"
+#define LOGICALREP_ORIGIN_ANY "any"
+

I thought there should be a comment above these new constants.

~~~

14. src/include/catalog/pg_subscription.h

@@ -87,6 +90,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW

  /* List of publications subscribed to */
  text subpublications[1] BKI_FORCE_NOT_NULL;
+
+ /* Only publish data originating from the specified origin */
+ text suborigin;
 #endif
 } FormData_pg_subscription;

Perhaps it would be better if this new column was also forced to be NOT NULL.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  126. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-15T06:41:21Z

    Here are some review comments for patch v20-0003.

======

1. Commit message

In case, table t1 has a unique key, it will lead to a unique key
violation and replication won't proceed.

SUGGESTION
If table t1 has a unique key, this will lead to a unique key
violation, and replication won't proceed.

~~~

2. Commit message

This problem can be solved by using...

SUGGESTION
This problem can be avoided by using...

~~~

3. Commit message

step 3: Create a subscription in node1 to subscribe to node2. Use
'copy_data = on' so that the existing table data is copied during
initial sync:
node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '<node2 details>'
node1-# PUBLICATION pub_node2 WITH (copy_data = off, origin = local);
CREATE SUBSCRIPTION

This is wrong information. The table on node2 has no data, so talking
about 'copy_data = on' is inappropriate here.

======

4. Commit message

IMO it might be better to refer to subscription/publication/table "on"
nodeXXX, instead of saying "in" nodeXXX.

4a.
"the publication tables were also subscribing data in the publisher
from other publishers." -> "the publication tables were also
subscribing from other publishers.

4b.
"After the subscription is created in node2" -> "After the
subscription is created on node2"

4c.
"step 3: Create a subscription in node1 to subscribe to node2." ->
"step 3: Create a subscription on node1 to subscribe to node2."

4d.
"step 4: Create a subscription in node2 to subscribe to node1." ->
"step 4: Create a subscription on node2 to subscribe to node1."

======

5. doc/src/sgml/ref/create_subscription.sgml

@@ -383,6 +397,15 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   If the subscription is created with <literal>origin = local</literal> and
+   <literal>copy_data = on</literal>, it will check if the publisher tables are
+   being subscribed to any other publisher and, if so, then throw an error to
+   prevent possible non-local data from being copied. The user can override
+   this check and continue with the copy operation by specifying
+   <literal>copy_data = force</literal>.
+  </para>

Perhaps it is better here to say 'copy_data = true' instead of
'copy_data = on', simply because the value 'true' was mentioned
earlier on this page (but this would be the first mention of 'on').

======

6. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

+ errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force."));

Saying "off or force" is not consistent with the other message wording
in this patch, which used "/" for multiple enums.
(e.g. "connect = false", "copy_data = true/force").

So perhaps this errhint should be worded similarly:
"Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  127. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-15T06:43:35Z

    Here are some review comments for patch v20-0004.

======

1. General

I thought that it is better to refer to the
subscription/publications/table "on" the node, rather than "in" the
node. Most of the review comments below are related to this point.

======

2. Commit message

a) Creating a two-node bidirectional replication when there is no data
in both nodes.
b) Adding a new node when there is no data in any of the nodes.
c) Adding a new node when data is present in the existing nodes.

"in both nodes" -> "on both nodes"
"in any of the nodes" -> "on any of the nodes"
"in the existing nodes" -> "on the existing nodes"

======

3. doc/src/sgml/logical-replication.sgml - Setting bidirectional
replication between two nodes

3a.
+   <para>
+    The following steps demonstrate how to create a two-node bidirectional
+    replication when there is no table data present in both nodes
+    <literal>node1</literal> and <literal>node2</literal>:
+   </para>
-> "on both nodes"

3b.
+    Create a publication in <literal>node1</literal>:
-> "on"

3c.
+    Create a publication in <literal>node2</literal>:
-> "on"

3d.
+   <para>
+    Lock the table <literal>t1</literal> in <literal>node1</literal> and
+    <literal>node2</literal> in <literal>EXCLUSIVE</literal> mode until the
+    setup is completed.
+   </para>
-> "on <literal>node1</literal>"

3e.
+    Create a subscription in <literal>node2</literal> to subscribe to
-> "on"

3f.
+    Create a subscription in <literal>node1</literal> to subscribe to
+    <literal>node2</literal>:
-> "on"

~~~

4. doc/src/sgml/logical-replication.sgml - Adding a new node when
there is no data in any of the nodes

4a.
+   <title>Adding a new node when there is no data in any of the nodes</title>
SUGGESTION
Adding a new node when there is no table data on any of the nodes

4b.
+   <para>
+    The following steps demonstrate adding a new node <literal>node3</literal>
+    to the existing <literal>node1</literal> and <literal>node2</literal> when
+    there is no <literal>t1</literal> data in any of the nodes. This requires
+    creating subscriptions in <literal>node1</literal> and
+    <literal>node2</literal> to replicate the data from
+    <literal>node3</literal> and creating subscriptions in
+    <literal>node3</literal> to replicate data from <literal>node1</literal>
+    and <literal>node2</literal>. Note: These steps assume that the
+    bidirectional logical replication between <literal>node1</literal> and
+    <literal>node2</literal> is already completed.
+   </para>

"data in any of the nodes" -> "data on any of the nodes"
"creating subscriptions in <literal>node1</literal>" -> "creating
subscriptions on <literal>node1</literal>"
"creating subscriptions in <literal>node3</literal>" -> "creating
subscriptions on <literal>node3</literal>"

4c.
+    Create a publication in <literal>node3</literal>:
-> "on"

4d.
+    Lock table <literal>t1</literal> in all the nodes
-> "on"

4e.
+    Create a subscription in <literal>node1</literal> to subscribe to
+    <literal>node3</literal>:
-> "on"

4f.
+    Create a subscription in <literal>node2</literal> to subscribe to
+    <literal>node3</literal>:
-> "on"

4g.
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node1</literal>:
-> "on"

4h.
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node2</literal>:

4i.
+    <literal>node3</literal>. Incremental changes made in any node will be
+    replicated to the other two nodes.
"in any node" -> "on any node"

~~~

5. doc/src/sgml/logical-replication.sgml - Adding a new node when data
is present in the existing nodes

5a.
+   <title>Adding a new node when data is present in the existing nodes</title>
SUGGESTION
Adding a new node when table data is present on the existing nodes

5b.
+     during initial data synchronization. Note: These steps assume that the
+     bidirectional logical replication between <literal>node1</literal> and
+     <literal>node2</literal> is already completed, and the pre-existing data
+     in table <literal>t1</literal> is already synchronized in both those
+     nodes.
+   </para>
"in both those nodes" -> "on both those nodes"

5c.
+    Create a publication in <literal>node3</literal>
-> "on"

5d.
+    Lock table <literal>t1</literal> in <literal>node2</literal> and
-> "on"

5e.
+    Create a subscription in <literal>node1</literal> to subscribe to
+    <literal>node3</literal>:
-> "on"

5f.
+    Create a subscription in <literal>node2</literal> to subscribe to
+    <literal>node3</literal>:
-> "on"

5g.
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node1</literal>. Use <literal>copy_data = force </literal> so that
+    the existing table data is copied during initial sync:
-> "on"


5h.
+    Create a subscription in <literal>node3</literal> to subscribe to
+    <literal>node2</literal>. Use <literal>copy_data = off</literal>
-> "on"

5i.
+    <literal>node3</literal>. Incremental changes made in any node will be
+    replicated to the other two nodes.
"in any node" -> "on any node"

~~~

6. doc/src/sgml/logical-replication.sgml - Adding a new node when data
is present in the new node

+   <title>Adding a new node when data is present in the new node</title>
SUGGESTION
Adding a new node when table data is present on the new node

~~~

7. doc/src/sgml/logical-replication.sgml - Generic steps for adding a
new node to an existing set of nodes

7a.
+   <para>
+    Step-2: Lock the required tables of the new node in EXCLUSIVE mode until
+    the setup is complete. (This lock is necessary to prevent any modifications
+    from happening in the new node because if data modifications occurred after
+    Step-3, there is a chance that the modifications will be published to the
+    first node and then synchronized back to the new node while creating the
+    subscription in Step-5. This would result in inconsistent data).
+   </para>
"happening in the new node" -> "happening on the new node"

7b.
+    not be synchronized to the new node. This would result in inconsistent
+    data. There is no need to lock the required tables in
+    <literal>node1</literal> because any data changes made will be synchronized
+    while creating the subscription with <literal>copy_data = force</literal>).
+   </para>
"no need to lock the required tables in" -> "no need to lock the
required tables on"

======

8. doc/src/sgml/ref/create_subscription.sgml

@@ -403,7 +403,10 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    being subscribed to any other publisher and, if so, then throw an error to
    prevent possible non-local data from being copied. The user can override
    this check and continue with the copy operation by specifying
-   <literal>copy_data = force</literal>.
+   <literal>copy_data = force</literal>. Refer to
+   <xref linkend="logical-replication-bidirectional"/> for how
+   <literal>copy_data</literal> and <literal>origin</literal> can be used
+   in bidirectional replication.
   </para>

"can be used in bidirectional replication" -> "can be used to set up
bidirectional replication"

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  128. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-15T08:56:24Z

    PSA a test script that demonstrates all the documented steps for
setting up n-way bidirectional replication. These steps are the same
as those documented [1] on the new page "Bidirectional logical
replication".

This script works using the current latest v20* patch set. Each of the
sections of 31.11.1 - 31.11.5 (see below) can be run independently
(just edit the script and at the bottom uncomment the part you want to
test):
31.11.1. Setting bidirectional replication between two nodes
31.11.2. Adding a new node when there is no data in any of the nodes
31.11.3. Adding a new node when data is present in the existing nodes
31.11.4. Adding a new node when data is present in the new node
31.11.5. Generic steps for adding a new node to an existing set of nodes

~~

Some sample output is also attached.

------
[1] https://www.postgresql.org/message-id/attachment/134464/v20-0004-Document-bidirectional-logical-replication-steps.patch

Kind Regards,
Peter Smith.
Fujitsu Australia

  129. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-16T10:17:50Z

    On Wed, Jun 15, 2022 at 12:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v20-0002
>
> ======
>
> 1. General comment
>
> Something subtle but significant changed since I last reviewed v18*.
> Now the describe.c is changed so that the catalog will never display a
> NULL origin column; it would always be "any". But now I am not sure if
> it is a good idea to still allow the NULL in this catalog column while
> at the same time you are pretending it is not there. I felt it might
> be less confusing, and would simplify the code (e.g. remove quite a
> few null checks) to have just used a single concept of the default -
> e.g. Just assign the default as "any" everywhere. The column would be
> defined as NOT NULL. Most of the following review comments are related
> to this point.

Ok, I was initially feeling having NULL value will help in the upgrade
cases where we upgrade from a lower version which does not have origin
to current. But setting the default value handles the upgrade scenario
too.

> ======
>
> 2. doc/src/sgml/catalogs.sgml
>
> +      <para>
> +       Possible origin values are <literal>local</literal>,
> +       <literal>any</literal>, or <literal>NULL</literal> if none is specified.
> +       If <literal>local</literal>, the subscription will request the
> +       publisher to only send changes that originated locally. If
> +       <literal>any</literal> (or <literal>NULL</literal>), the publisher sends
> +       any changes regardless of their origin.
> +      </para></entry>
>
> Is NULL still possible? Perhaps it would be better if it was not and
> the default "any" was always written instead.

Modified

> ======
>
> 3. src/backend/catalog/pg_subscription.c
>
> + if (!isnull)
> + sub->origin = TextDatumGetCString(datum);
> + else
> + sub->origin = NULL;
> +
>
> Maybe better to either disallow NULL in the first place or assign the
> "any" here instead of NULL.

Modified

> ======
>
> 4. src/backend/commands/subscriptioncmds.c - parse_subscription_options
>
> @@ -137,6 +139,8 @@ parse_subscription_options(ParseState *pstate,
> List *stmt_options,
>   opts->twophase = false;
>   if (IsSet(supported_opts, SUBOPT_DISABLE_ON_ERR))
>   opts->disableonerr = false;
> + if (IsSet(supported_opts, SUBOPT_ORIGIN))
> + opts->origin = NULL;
>
> If opt->origin was assigned to "any" then other code would be simplified.

Modified

> ~~~
>
> 5. src/backend/commands/subscriptioncmds.c - CreateSubscription
>
> @@ -607,6 +626,11 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>   LOGICALREP_TWOPHASE_STATE_PENDING :
>   LOGICALREP_TWOPHASE_STATE_DISABLED);
>   values[Anum_pg_subscription_subdisableonerr - 1] =
> BoolGetDatum(opts.disableonerr);
> + if (opts.origin)
> + values[Anum_pg_subscription_suborigin - 1] =
> + CStringGetTextDatum(opts.origin);
> + else
> + values[Anum_pg_subscription_suborigin - 1] =
> CStringGetTextDatum(LOGICALREP_ORIGIN_ANY);
>
> If NULL was not possible then this would just be one line:
> values[Anum_pg_subscription_suborigin - 1] = CStringGetTextDatum(opts.origin);

Modified

> ======
>
> 6. src/backend/replication/logical/worker.c
>
> @@ -276,6 +276,10 @@ static TransactionId stream_xid = InvalidTransactionId;
>  static XLogRecPtr skip_xact_finish_lsn = InvalidXLogRecPtr;
>  #define is_skipping_changes()
> (unlikely(!XLogRecPtrIsInvalid(skip_xact_finish_lsn)))
>
> +/* Macro for comparing string fields that might be NULL */
> +#define equalstr(a, b) \
> + (((a) != NULL && (b) != NULL) ? (strcmp((a), (b)) == 0) : (a) == (b))
> +
>
> If the NULL was not allowed in the first place then I think this macro
> would just become redundant.

Removed it

> 7. src/backend/replication/logical/worker.c - ApplyWorkerMain
>
> @@ -3741,6 +3746,11 @@ ApplyWorkerMain(Datum main_arg)
>   options.proto.logical.streaming = MySubscription->stream;
>   options.proto.logical.twophase = false;
>
> + if (MySubscription->origin)
> + options.proto.logical.origin = pstrdup(MySubscription->origin);
> + else
> + options.proto.logical.origin = NULL;
> +
>
> Can't the if/else be avoided if you always assigned the "any" default
> in the first place?

Modified

> ======
>
> 8. src/backend/replication/pgoutput/pgoutput.c - parse_output_parameters
>
> @@ -287,11 +289,13 @@ parse_output_parameters(List *options, PGOutputData *data)
>   bool messages_option_given = false;
>   bool streaming_given = false;
>   bool two_phase_option_given = false;
> + bool origin_option_given = false;
>
>   data->binary = false;
>   data->streaming = false;
>   data->messages = false;
>   data->two_phase = false;
> + data->origin = NULL;
>
> Consider assigning default "any" here instead of NULL.

This is no more required because of setting default to any and the
same value will be passed as an option

> ======
>
> 9. src/bin/pg_dump/pg_dump.c - getSubscriptions
>
> + /* FIXME: 150000 should be changed to 160000 later for PG16. */
> + if (fout->remoteVersion >= 150000)
> + appendPQExpBufferStr(query, " s.suborigin\n");
> + else
> + appendPQExpBufferStr(query, " NULL AS suborigin\n");
> +
>
> Maybe say: 'any' AS suborigin?

Modified

> ~~~
>
> 10. src/bin/pg_dump/pg_dump.c - getSubscriptions
>
> @@ -4517,6 +4525,11 @@ getSubscriptions(Archive *fout)
>   subinfo[i].subdisableonerr =
>   pg_strdup(PQgetvalue(res, i, i_subdisableonerr));
>
> + if (PQgetisnull(res, i, i_suborigin))
> + subinfo[i].suborigin = NULL;
> + else
> + subinfo[i].suborigin = pg_strdup(PQgetvalue(res, i, i_suborigin));
> +
>
> If you disallow the NULL in the first place this condition maybe is no
> longer needed.

Modified

> ~~~
>
> 11. src/bin/pg_dump/pg_dump.c - dumpSubscription
>
> @@ -4589,6 +4602,9 @@ dumpSubscription(Archive *fout, const
> SubscriptionInfo *subinfo)
>   if (strcmp(subinfo->subdisableonerr, "t") == 0)
>   appendPQExpBufferStr(query, ", disable_on_error = true");
>
> + if (subinfo->suborigin)
> + appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
> +
>
> If NULL cannot happen then maybe this test is also redundant.

Modified

> ======
>
> 12. src/bin/pg_dump/t/002_pg_dump.pl
>
> AFAICT there is a test for no origin (default), and a test for
> explicit origin = local, but there is no test case for the explicit
> origin = any.

Added a test for the same

> ======
>
> 13. src/include/catalog/pg_subscription.h
>
> @@ -31,6 +31,9 @@
>  #define LOGICALREP_TWOPHASE_STATE_PENDING 'p'
>  #define LOGICALREP_TWOPHASE_STATE_ENABLED 'e'
>
> +#define LOGICALREP_ORIGIN_LOCAL "local"
> +#define LOGICALREP_ORIGIN_ANY "any"
> +
>
> I thought there should be a comment above these new constants.

Added comments

> ~~~
>
> 14. src/include/catalog/pg_subscription.h
>
> @@ -87,6 +90,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>
>   /* List of publications subscribed to */
>   text subpublications[1] BKI_FORCE_NOT_NULL;
> +
> + /* Only publish data originating from the specified origin */
> + text suborigin;
>  #endif
>  } FormData_pg_subscription;
>
> Perhaps it would be better if this new column was also forced to be NOT NULL.

I have set a default value so need to set NOT NULL.

Thanks for the comments, the attached v21 patch has the changes for the same.

Regards,
Vignesh

  130. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-16T10:20:02Z

    On Wed, Jun 15, 2022 at 12:11 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v20-0003.
>
> ======
>
> 1. Commit message
>
> In case, table t1 has a unique key, it will lead to a unique key
> violation and replication won't proceed.
>
> SUGGESTION
> If table t1 has a unique key, this will lead to a unique key
> violation, and replication won't proceed.

Modified

> ~~~
>
> 2. Commit message
>
> This problem can be solved by using...
>
> SUGGESTION
> This problem can be avoided by using...

Modified

> ~~~
>
> 3. Commit message
>
> step 3: Create a subscription in node1 to subscribe to node2. Use
> 'copy_data = on' so that the existing table data is copied during
> initial sync:
> node1=# CREATE SUBSCRIPTION sub_node1_node2 CONNECTION '<node2 details>'
> node1-# PUBLICATION pub_node2 WITH (copy_data = off, origin = local);
> CREATE SUBSCRIPTION
>
> This is wrong information. The table on node2 has no data, so talking
> about 'copy_data = on' is inappropriate here.

Modified

> ======
>
> 4. Commit message
>
> IMO it might be better to refer to subscription/publication/table "on"
> nodeXXX, instead of saying "in" nodeXXX.
>
> 4a.
> "the publication tables were also subscribing data in the publisher
> from other publishers." -> "the publication tables were also
> subscribing from other publishers.

Modified

> 4b.
> "After the subscription is created in node2" -> "After the
> subscription is created on node2"

Modified

> 4c.
> "step 3: Create a subscription in node1 to subscribe to node2." ->
> "step 3: Create a subscription on node1 to subscribe to node2."

Modified

> 4d.
> "step 4: Create a subscription in node2 to subscribe to node1." ->
> "step 4: Create a subscription on node2 to subscribe to node1."

Modified

> ======
>
> 5. doc/src/sgml/ref/create_subscription.sgml
>
> @@ -383,6 +397,15 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   If the subscription is created with <literal>origin = local</literal> and
> +   <literal>copy_data = on</literal>, it will check if the publisher tables are
> +   being subscribed to any other publisher and, if so, then throw an error to
> +   prevent possible non-local data from being copied. The user can override
> +   this check and continue with the copy operation by specifying
> +   <literal>copy_data = force</literal>.
> +  </para>
>
> Perhaps it is better here to say 'copy_data = true' instead of
> 'copy_data = on', simply because the value 'true' was mentioned
> earlier on this page (but this would be the first mention of 'on').

Modified

> ======
>
> 6. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off or force."));
>
> Saying "off or force" is not consistent with the other message wording
> in this patch, which used "/" for multiple enums.
> (e.g. "connect = false", "copy_data = true/force").
>
> So perhaps this errhint should be worded similarly:
> "Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."

Modified

Thanks for the comments, the v21 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm3%2B6cey0rcDft1ZUCjSUtLDM0xmU_Q%2BYhcsBrqe1RH8%3Dw%40mail.gmail.com

Regards,
Vignesh

  131. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-16T10:23:18Z

    On Wed, Jun 15, 2022 at 12:13 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v20-0004.
>
> ======
>
> 1. General
>
> I thought that it is better to refer to the
> subscription/publications/table "on" the node, rather than "in" the
> node. Most of the review comments below are related to this point.

Modified

> ======
>
> 2. Commit message
>
> a) Creating a two-node bidirectional replication when there is no data
> in both nodes.
> b) Adding a new node when there is no data in any of the nodes.
> c) Adding a new node when data is present in the existing nodes.
>
> "in both nodes" -> "on both nodes"
> "in any of the nodes" -> "on any of the nodes"
> "in the existing nodes" -> "on the existing nodes"

Modified

> ======
>
> 3. doc/src/sgml/logical-replication.sgml - Setting bidirectional
> replication between two nodes
>
> 3a.
> +   <para>
> +    The following steps demonstrate how to create a two-node bidirectional
> +    replication when there is no table data present in both nodes
> +    <literal>node1</literal> and <literal>node2</literal>:
> +   </para>
> -> "on both nodes"

Modified

> 3b.
> +    Create a publication in <literal>node1</literal>:
> -> "on"

Modified

> 3c.
> +    Create a publication in <literal>node2</literal>:
> -> "on"

Modified

> 3d.
> +   <para>
> +    Lock the table <literal>t1</literal> in <literal>node1</literal> and
> +    <literal>node2</literal> in <literal>EXCLUSIVE</literal> mode until the
> +    setup is completed.
> +   </para>
> -> "on <literal>node1</literal>"

Modified

> 3e.
> +    Create a subscription in <literal>node2</literal> to subscribe to
> -> "on"

Modified

> 3f.
> +    Create a subscription in <literal>node1</literal> to subscribe to
> +    <literal>node2</literal>:
> -> "on"

Modified

> ~~~
>
> 4. doc/src/sgml/logical-replication.sgml - Adding a new node when
> there is no data in any of the nodes
>
> 4a.
> +   <title>Adding a new node when there is no data in any of the nodes</title>
> SUGGESTION
> Adding a new node when there is no table data on any of the nodes

Modified

> 4b.
> +   <para>
> +    The following steps demonstrate adding a new node <literal>node3</literal>
> +    to the existing <literal>node1</literal> and <literal>node2</literal> when
> +    there is no <literal>t1</literal> data in any of the nodes. This requires
> +    creating subscriptions in <literal>node1</literal> and
> +    <literal>node2</literal> to replicate the data from
> +    <literal>node3</literal> and creating subscriptions in
> +    <literal>node3</literal> to replicate data from <literal>node1</literal>
> +    and <literal>node2</literal>. Note: These steps assume that the
> +    bidirectional logical replication between <literal>node1</literal> and
> +    <literal>node2</literal> is already completed.
> +   </para>
>
> "data in any of the nodes" -> "data on any of the nodes"
> "creating subscriptions in <literal>node1</literal>" -> "creating
> subscriptions on <literal>node1</literal>"
> "creating subscriptions in <literal>node3</literal>" -> "creating
> subscriptions on <literal>node3</literal>"

Modified

> 4c.
> +    Create a publication in <literal>node3</literal>:
> -> "on"

Modified

> 4d.
> +    Lock table <literal>t1</literal> in all the nodes
> -> "on"

Modified

> 4e.
> +    Create a subscription in <literal>node1</literal> to subscribe to
> +    <literal>node3</literal>:
> -> "on"

Modified

> 4f.
> +    Create a subscription in <literal>node2</literal> to subscribe to
> +    <literal>node3</literal>:
> -> "on"

Modified

> 4g.
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node1</literal>:
> -> "on"

Modified

> 4h.
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node2</literal>:

Modified

> 4i.
> +    <literal>node3</literal>. Incremental changes made in any node will be
> +    replicated to the other two nodes.
> "in any node" -> "on any node"

Modified

> ~~~
>
> 5. doc/src/sgml/logical-replication.sgml - Adding a new node when data
> is present in the existing nodes
>
> 5a.
> +   <title>Adding a new node when data is present in the existing nodes</title>
> SUGGESTION
> Adding a new node when table data is present on the existing nodes

Modified

> 5b.
> +     during initial data synchronization. Note: These steps assume that the
> +     bidirectional logical replication between <literal>node1</literal> and
> +     <literal>node2</literal> is already completed, and the pre-existing data
> +     in table <literal>t1</literal> is already synchronized in both those
> +     nodes.
> +   </para>
> "in both those nodes" -> "on both those nodes"

Modified

> 5c.
> +    Create a publication in <literal>node3</literal>
> -> "on"

Modified

> 5d.
> +    Lock table <literal>t1</literal> in <literal>node2</literal> and
> -> "on"

Modified

> 5e.
> +    Create a subscription in <literal>node1</literal> to subscribe to
> +    <literal>node3</literal>:
> -> "on"

Modified

> 5f.
> +    Create a subscription in <literal>node2</literal> to subscribe to
> +    <literal>node3</literal>:
> -> "on"

Modified

> 5g.
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node1</literal>. Use <literal>copy_data = force </literal> so that
> +    the existing table data is copied during initial sync:
> -> "on"

Modified

>
> 5h.
> +    Create a subscription in <literal>node3</literal> to subscribe to
> +    <literal>node2</literal>. Use <literal>copy_data = off</literal>
> -> "on"

Modified

> 5i.
> +    <literal>node3</literal>. Incremental changes made in any node will be
> +    replicated to the other two nodes.
> "in any node" -> "on any node"

Modified

> ~~~
>
> 6. doc/src/sgml/logical-replication.sgml - Adding a new node when data
> is present in the new node
>
> +   <title>Adding a new node when data is present in the new node</title>
> SUGGESTION
> Adding a new node when table data is present on the new node

Modified

> ~~~
>
> 7. doc/src/sgml/logical-replication.sgml - Generic steps for adding a
> new node to an existing set of nodes
>
> 7a.
> +   <para>
> +    Step-2: Lock the required tables of the new node in EXCLUSIVE mode until
> +    the setup is complete. (This lock is necessary to prevent any modifications
> +    from happening in the new node because if data modifications occurred after
> +    Step-3, there is a chance that the modifications will be published to the
> +    first node and then synchronized back to the new node while creating the
> +    subscription in Step-5. This would result in inconsistent data).
> +   </para>
> "happening in the new node" -> "happening on the new node"

Modified

> 7b.
> +    not be synchronized to the new node. This would result in inconsistent
> +    data. There is no need to lock the required tables in
> +    <literal>node1</literal> because any data changes made will be synchronized
> +    while creating the subscription with <literal>copy_data = force</literal>).
> +   </para>
> "no need to lock the required tables in" -> "no need to lock the
> required tables on"

Modified

> ======
>
> 8. doc/src/sgml/ref/create_subscription.sgml
>
> @@ -403,7 +403,10 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     being subscribed to any other publisher and, if so, then throw an error to
>     prevent possible non-local data from being copied. The user can override
>     this check and continue with the copy operation by specifying
> -   <literal>copy_data = force</literal>.
> +   <literal>copy_data = force</literal>. Refer to
> +   <xref linkend="logical-replication-bidirectional"/> for how
> +   <literal>copy_data</literal> and <literal>origin</literal> can be used
> +   in bidirectional replication.
>    </para>
>
> "can be used in bidirectional replication" -> "can be used to set up
> bidirectional replication"

Modified

Thanks for the comments, the v21 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm3%2B6cey0rcDft1ZUCjSUtLDM0xmU_Q%2BYhcsBrqe1RH8%3Dw%40mail.gmail.com

Regards,
Vignesh

  132. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-06-20T03:52:01Z

    On Thu, Jun 16, 2022 6:18 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached v21 patch has the changes for the
> same.
> 

Thanks for updating the patch. Here are some comments.

0002 patch
==============
1.
+          publisher to only send changes that originated locally. Setting
+          <literal>origin</literal> to <literal>any</literal> means that that
+          the publisher sends any changes regardless of their origin. The
+          default is <literal>any</literal>.

It seems there's a redundant "that" at the end of second line.

2.
+    <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>suborigin</structfield> <type>text</type>
+      </para>
+      <para>
+       Possible origin values are <literal>local</literal> or
+       <literal>any</literal>. The default is <literal>any</literal>.
+       If <literal>local</literal>, the subscription will request the publisher
+       to only send changes that originated locally. If <literal>any</literal>
+       the publisher sends any changes regardless of their origin.
+      </para></entry>
+     </row>.

A comma can be added after "If any".

3.
@@ -4589,6 +4598,8 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
 	if (strcmp(subinfo->subdisableonerr, "t") == 0)
 		appendPQExpBufferStr(query, ", disable_on_error = true");
 
+	appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+
 	if (strcmp(subinfo->subsynccommit, "off") != 0)
 		appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));

Do we need to append anything if it's the default value ("any")? I saw that some
other parameters, they will be appended only if they are not the default value.

0003 patch
==============
1. 
in create_subscription.sgml:
          (You cannot combine setting <literal>connect</literal>
          to <literal>false</literal> with
          setting <literal>create_slot</literal>, <literal>enabled</literal>,
          or <literal>copy_data</literal> to <literal>true</literal>.)

In this description about "connect" parameter in CREATE SUBSCIPTION document,
maybe it would be better to change "copy_data to true" to "copy_data to
true/force".

2.
+	appendStringInfoString(&cmd,
+						   "SELECT DISTINCT N.nspname AS schemaname,\n"
+						   "				C.relname AS tablename,\n"
+						   "				PS.srrelid as replicated\n"
+						   "FROM pg_publication P,\n"
+						   "	 LATERAL pg_get_publication_tables(P.pubname) GPT\n"
+						   "	 LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
+						   "	 pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
+						   "WHERE C.oid = GPT.relid AND P.pubname IN (");

"PS.srrelid as replicated" can be modified to "PS.srrelid AS replicated".

Besides, I think we can filter out the tables which are not subscribing data in
this SQL statement, then later processing can be simplified.

Something like:
SELECT DISTINCT N.nspname AS schemaname,
                                C.relname AS tablename
FROM pg_publication P,
         LATERAL pg_get_publication_tables(P.pubname) GPT
         LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
         pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE C.oid = GPT.relid AND P.pubname IN ('pa') AND PS.srrelid IS NOT NULL;

0004 patch
==============
1. Generic steps for adding a new node to an existing set of nodes

+    Step-2: Lock the required tables of the new node in EXCLUSIVE mode until
+    the setup is complete. (This lock is necessary to prevent any modifications

+    Step-4. Lock the required tables of the existing nodes except the first node
+    in EXCLUSIVE mode until the setup is complete. (This lock is necessary to

Should "in EXCLUSIVE mode" be modified to "in <literal>EXCLUSIVE</literal>
mode"?

2. Generic steps for adding a new node to an existing set of nodes

+    data. There is no need to lock the required tables on
+    <literal>node1</literal> because any data changes made will be synchronized
+    while creating the subscription with <literal>copy_data = force</literal>).

I think it would be better to say "on the first node" here, instead of "node1",
because in this section, node1 is not mentioned before.

Regards,
Shi yu

  133. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-06-20T09:07:32Z

    On Thu, Jun 16, 2022 at 3:48 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, Jun 15, 2022 at 12:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >

> Thanks for the comments, the attached v21 patch has the changes for the same.

I have done some basic review of v21 and I have a few comments,

1.
+/*
+ * The subscription will request the publisher to only send changes that
+ * originated locally.
+ */
+#define LOGICALREP_ORIGIN_LOCAL "local"
+
+/*
+ * The subscription will request the publisher to send any changes regardless
+ * of their origin
+ */
+#define LOGICALREP_ORIGIN_ANY "any"

Are we planning to extend this to more options or are we planning to
support the actual origin name here? If not then why isn't it just
bool?  I think the comments and the patch commit message should
explain the details behind it if it has been already discussed and
concluded.

2.
+/*
+ * Check and throw an error if the publisher has subscribed to the same table
+ * from some other publisher. This check is required only if copydata is ON and
+ * the origin is local.
+ */

I think it should also explain why this combination is not allowed and
if it is already explained in code
then this code can add comments to refer to that part of the code.


-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  134. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-20T09:46:43Z

    On Mon, Jun 20, 2022 at 2:37 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jun 16, 2022 at 3:48 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, Jun 15, 2022 at 12:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
>
> > Thanks for the comments, the attached v21 patch has the changes for the same.
>
> I have done some basic review of v21 and I have a few comments,
>
> 1.
> +/*
> + * The subscription will request the publisher to only send changes that
> + * originated locally.
> + */
> +#define LOGICALREP_ORIGIN_LOCAL "local"
> +
> +/*
> + * The subscription will request the publisher to send any changes regardless
> + * of their origin
> + */
> +#define LOGICALREP_ORIGIN_ANY "any"
>
> Are we planning to extend this to more options or are we planning to
> support the actual origin name here? If not then why isn't it just
> bool?  I think the comments and the patch commit message should
> explain the details behind it if it has been already discussed and
> concluded.

Currently we only support local and any. But this was designed to
accept string instead of boolean type, so that it can be extended
later to support filtering of origin names specified by the user in
the later versions. The same was also discussed in pg unconference as
mentioned in [1]. I will add it to the commit message and a comment
for the same in the next version.

> 2.
> +/*
> + * Check and throw an error if the publisher has subscribed to the same table
> + * from some other publisher. This check is required only if copydata is ON and
> + * the origin is local.
> + */
>
> I think it should also explain why this combination is not allowed and
> if it is already explained in code
> then this code can add comments to refer to that part of the code.

In the same function, the reason for this is mentioned detailly just
above the place where error is thrown. I think that should be enough.
Have a look and let me know if that is not sufficient:
+               /*
+                * Throw an error if the publisher has subscribed to
the same table
+                * from some other publisher. We cannot differentiate
between the
+                * local and non-local data that is present in the
HEAP during the
+                * initial sync. Identification of local data can be
done only from
+                * the WAL by using the origin id. XXX: For
simplicity, we don't check
+                * whether the table has any data or not. If the table
doesn't have
+                * any data then we don't need to distinguish between local and
+                * non-local data so we can avoid throwing error in that case.
+                */
+               if (!slot_attisnull(slot, 3))
+                       ereport(ERROR,
+
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+                                       errmsg("table:%s.%s might have
replicated data in the publisher",
+                                                  nspname, relname),
+                                       errdetail("CREATE/ALTER
SUBSCRIPTION with origin = local and copy_data = on is not allowed
when the publisher might have replicated data."),
+                                       errhint("Use CREATE/ALTER
SUBSCRIPTION with copy_data = off/force."));


[1] - https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Logical_Replication_Origin_Filtering_and_Consistency

Regards,
Vignesh

  135. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-06-20T10:32:13Z

    On Mon, Jun 20, 2022 at 3:16 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Jun 20, 2022 at 2:37 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jun 16, 2022 at 3:48 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, Jun 15, 2022 at 12:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> >
> > > Thanks for the comments, the attached v21 patch has the changes for the same.
> >
> > I have done some basic review of v21 and I have a few comments,
> >
> > 1.
> > +/*
> > + * The subscription will request the publisher to only send changes that
> > + * originated locally.
> > + */
> > +#define LOGICALREP_ORIGIN_LOCAL "local"
> > +
> > +/*
> > + * The subscription will request the publisher to send any changes regardless
> > + * of their origin
> > + */
> > +#define LOGICALREP_ORIGIN_ANY "any"
> >
> > Are we planning to extend this to more options or are we planning to
> > support the actual origin name here? If not then why isn't it just
> > bool?  I think the comments and the patch commit message should
> > explain the details behind it if it has been already discussed and
> > concluded.
>
> Currently we only support local and any. But this was designed to
> accept string instead of boolean type, so that it can be extended
> later to support filtering of origin names specified by the user in
> the later versions. The same was also discussed in pg unconference as
> mentioned in [1]. I will add it to the commit message and a comment
> for the same in the next version.
>
> > 2.
> > +/*
> > + * Check and throw an error if the publisher has subscribed to the same table
> > + * from some other publisher. This check is required only if copydata is ON and
> > + * the origin is local.
> > + */
> >
> > I think it should also explain why this combination is not allowed and
> > if it is already explained in code
> > then this code can add comments to refer to that part of the code.
>
> In the same function, the reason for this is mentioned detailly just
> above the place where error is thrown. I think that should be enough.
> Have a look and let me know if that is not sufficient:

I think that should be sufficient, thanks.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  136. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-20T11:55:24Z

    On Mon, Jun 20, 2022 at 9:22 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Thu, Jun 16, 2022 6:18 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v21 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch. Here are some comments.
>
> 0002 patch
> ==============
> 1.
> +          publisher to only send changes that originated locally. Setting
> +          <literal>origin</literal> to <literal>any</literal> means that that
> +          the publisher sends any changes regardless of their origin. The
> +          default is <literal>any</literal>.
>
> It seems there's a redundant "that" at the end of second line.

Modified

> 2.
> +    <row>
> +      <entry role="catalog_table_entry"><para role="column_definition">
> +       <structfield>suborigin</structfield> <type>text</type>
> +      </para>
> +      <para>
> +       Possible origin values are <literal>local</literal> or
> +       <literal>any</literal>. The default is <literal>any</literal>.
> +       If <literal>local</literal>, the subscription will request the publisher
> +       to only send changes that originated locally. If <literal>any</literal>
> +       the publisher sends any changes regardless of their origin.
> +      </para></entry>
> +     </row>.
>
> A comma can be added after "If any".

Modified

> 3.
> @@ -4589,6 +4598,8 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
>         if (strcmp(subinfo->subdisableonerr, "t") == 0)
>                 appendPQExpBufferStr(query, ", disable_on_error = true");
>
> +       appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
> +
>         if (strcmp(subinfo->subsynccommit, "off") != 0)
>                 appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
>
> Do we need to append anything if it's the default value ("any")? I saw that some
> other parameters, they will be appended only if they are not the default value.

Modified

> 0003 patch
> ==============
> 1.
> in create_subscription.sgml:
>           (You cannot combine setting <literal>connect</literal>
>           to <literal>false</literal> with
>           setting <literal>create_slot</literal>, <literal>enabled</literal>,
>           or <literal>copy_data</literal> to <literal>true</literal>.)
>
> In this description about "connect" parameter in CREATE SUBSCIPTION document,
> maybe it would be better to change "copy_data to true" to "copy_data to
> true/force".

Modified

> 2.
> +       appendStringInfoString(&cmd,
> +                                                  "SELECT DISTINCT N.nspname AS schemaname,\n"
> +                                                  "                            C.relname AS tablename,\n"
> +                                                  "                            PS.srrelid as replicated\n"
> +                                                  "FROM pg_publication P,\n"
> +                                                  "     LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> +                                                  "     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> +                                                  "     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> +                                                  "WHERE C.oid = GPT.relid AND P.pubname IN (");
>
> "PS.srrelid as replicated" can be modified to "PS.srrelid AS replicated".
>
> Besides, I think we can filter out the tables which are not subscribing data in
> this SQL statement, then later processing can be simplified.
>
> Something like:
> SELECT DISTINCT N.nspname AS schemaname,
>                                 C.relname AS tablename
> FROM pg_publication P,
>          LATERAL pg_get_publication_tables(P.pubname) GPT
>          LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
>          pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> WHERE C.oid = GPT.relid AND P.pubname IN ('pa') AND PS.srrelid IS NOT NULL;

Modified

> 0004 patch
> ==============
> 1. Generic steps for adding a new node to an existing set of nodes
>
> +    Step-2: Lock the required tables of the new node in EXCLUSIVE mode until
> +    the setup is complete. (This lock is necessary to prevent any modifications
>
> +    Step-4. Lock the required tables of the existing nodes except the first node
> +    in EXCLUSIVE mode until the setup is complete. (This lock is necessary to
>
> Should "in EXCLUSIVE mode" be modified to "in <literal>EXCLUSIVE</literal>
> mode"?

Modified

> 2. Generic steps for adding a new node to an existing set of nodes
>
> +    data. There is no need to lock the required tables on
> +    <literal>node1</literal> because any data changes made will be synchronized
> +    while creating the subscription with <literal>copy_data = force</literal>).
>
> I think it would be better to say "on the first node" here, instead of "node1",
> because in this section, node1 is not mentioned before.

Modified

Thanks for the comment, the v22 patch attached has the changes for the same.

Regards,
Vignesh

  137. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-20T11:57:29Z

    On Mon, Jun 20, 2022 at 3:16 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Jun 20, 2022 at 2:37 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jun 16, 2022 at 3:48 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, Jun 15, 2022 at 12:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> >
> > > Thanks for the comments, the attached v21 patch has the changes for the same.
> >
> > I have done some basic review of v21 and I have a few comments,
> >
> > 1.
> > +/*
> > + * The subscription will request the publisher to only send changes that
> > + * originated locally.
> > + */
> > +#define LOGICALREP_ORIGIN_LOCAL "local"
> > +
> > +/*
> > + * The subscription will request the publisher to send any changes regardless
> > + * of their origin
> > + */
> > +#define LOGICALREP_ORIGIN_ANY "any"
> >
> > Are we planning to extend this to more options or are we planning to
> > support the actual origin name here? If not then why isn't it just
> > bool?  I think the comments and the patch commit message should
> > explain the details behind it if it has been already discussed and
> > concluded.
>
> Currently we only support local and any. But this was designed to
> accept string instead of boolean type, so that it can be extended
> later to support filtering of origin names specified by the user in
> the later versions. The same was also discussed in pg unconference as
> mentioned in [1]. I will add it to the commit message and a comment
> for the same in the next version.

Thanks for the comment, the v22 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1h-9UNi_Jo_K%2BPK34tXBmV7fhj5C_nB8YzGA9rmUwHEA%40mail.gmail.com

Regards,
Vignesh

  138. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-22T06:46:15Z

    Here are my review comments for the v22* patch set.

========
v22-0001
========

No comments. LGTM

========
V22-0002
========

2.1 doc/src/sgml/catalogs.sgml

+       Possible origin values are <literal>local</literal> or
+       <literal>any</literal>. The default is <literal>any</literal>.

IMO the word "Possible" here is giving a sense of vagueness.

SUGGESTION
The origin value must be either <literal>local</literal> or
<literal>any</literal>.

~~~

2.2 src/backend/commands/subscriptioncmds.c

@@ -265,6 +271,29 @@ parse_subscription_options(ParseState *pstate,
List *stmt_options,
  opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
  opts->disableonerr = defGetBoolean(defel);
  }
+ else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
+ strcmp(defel->defname, "origin") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_ORIGIN))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_ORIGIN;
+
+ /*
+ * Even though "origin" parameter allows only "local" and "any"
+ * values, the "origin" parameter type is implemented as string
+ * type instead of boolean to extend the "origin" parameter to
+ * support filtering of origin name specified by the user in the
+ * later versions.
+ */
+ opts->origin = defGetString(defel);
+
+ if ((strcmp(opts->origin, LOGICALREP_ORIGIN_LOCAL) != 0) &&
+ (strcmp(opts->origin, LOGICALREP_ORIGIN_ANY) != 0))
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized origin value: \"%s\"", opts->origin));
+ }

I was wondering if it might be wise now to do a pfree(opts->origin)
here before setting the new option value which overwrites the
strdup-ed default "any". OTOH maybe it is overkill to worry about the
tiny leak? I am not sure what is the convention for this.

~~~

2.3 src/backend/replication/pgoutput/pgoutput.c

@@ -1698,12 +1719,16 @@ pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
 }

 /*
- * Currently we always forward.
+ * Return true if the data source (origin) is remote and user has requested
+ * only local data, false otherwise.
  */
 static bool
 pgoutput_origin_filter(LogicalDecodingContext *ctx,
     RepOriginId origin_id)

"user" -> "the user"

~~~

2.4 src/include/catalog/pg_subscription.h

+/*
+ * The subscription will request the publisher to send any changes regardless
+ * of their origin
+ */
+#define LOGICALREP_ORIGIN_ANY "any"

SUGGESTION (remove 'any'; add period)
The subscription will request the publisher to send changes regardless
of their origin.

========
v22-0003
========

3.1 Commit message

change 1) Checks and throws an error if 'copy_data = on' and 'origin =
local' but the publication tables were also subscribing from other publishers.

"also subscribing from other publishers." -> "also replicating from
other publishers."

~~~

3.2 doc/src/sgml/ref/alter_subscription.sgml

+         <para>
+          There is some interaction between the <literal>origin</literal>
+          parameter and <literal>copy_data</literal> parameter.

"and copy_data parameter." -> "and the copy_data parameter."

~~~

3.3 doc/src/sgml/ref/create_subscription.sgml

+         <para>
+          There is some interaction between the <literal>origin</literal>
+          parameter and <literal>copy_data</literal> parameter.

"and copy_data parameter." -> "and the copy_data parameter."

~~~

3.4 doc/src/sgml/ref/create_subscription.sgml

+         <para>
+          There is some interaction between the <literal>origin</literal>
+          parameter and <literal>copy_data</literal> parameter. Refer to the
+          <xref linkend="sql-createsubscription-notes" /> for details.
+         </para>

"and copy_data parameter." -> "and the copy_data parameter."

~~~

3.5 doc/src/sgml/ref/create_subscription.sgml

@@ -383,6 +398,15 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   If the subscription is created with <literal>origin = local</literal> and
+   <literal>copy_data = true</literal>, it will check if the
publisher tables are
+   being subscribed to any other publisher and, if so, then throw an error to
+   prevent possible non-local data from being copied. The user can override
+   this check and continue with the copy operation by specifying
+   <literal>copy_data = force</literal>.
+  </para>
+

For the part "it will check if the publisher tables are being
subscribed to any other publisher...", the "being subscribed to"
sounds a bit strange. Maybe it is better to reword this like you
already worded some of the code comments.

SUGGESTION
-> "... it will check if the publisher has subscribed to the same
table from other publishers and ..."

~~~

3.6 src/backend/commands/subscriptioncmds.c

+ /*
+ * Throw an error if the publisher has subscribed to the same table
+ * from some other publisher. We cannot differentiate between the
+ * local and non-local data that is present in the HEAP during the
+ * initial sync. Identification of local data can be done only from
+ * the WAL by using the origin id. XXX: For simplicity, we don't check
+ * whether the table has any data or not. If the table doesn't have
+ * any data then we don't need to distinguish between local and
+ * non-local data so we can avoid throwing error in that case.
+ */

I think the XXX part should be after a blank like so it is more
prominent, instead of being buried in the other text.

e.g.

+ /*
+ * Throw an error if the publisher has subscribed to the same table
+ * from some other publisher. We cannot differentiate between the
+ * local and non-local data that is present in the HEAP during the
+ * initial sync. Identification of local data can be done only from
+ * the WAL by using the origin id.
+ *
+ * XXX: For simplicity, we don't check
+ * whether the table has any data or not. If the table doesn't have
+ * any data then we don't need to distinguish between local and
+ * non-local data so we can avoid throwing error in that case.
+ */

~~~

3.7 src/test/regress/sql/subscription.sql

Elsewhere, there was code in this patch that apparently accepts
'copy_data' parameter but with no parameter value specified. But this
combination has no test case.

========
v22-0004
========

4.1 doc/src/sgml/logical-replication.sgml

+  <sect2 id="add-node-data-present-on-new-node">
+   <title>Adding a new node when data is present on the new node</title>

"when data is present" -> "when table data is present"

~~~

4.2 doc/src/sgml/logical-replication.sgml

+   <note>
+    <para>
+     Adding a new node when data is present on the new node tables is not
+     supported.
+    </para>
+   </note>

I think the note text should match the title text (see #4.1)

SUGGESTION
Adding a new node when table data is present on the new node is not supported.

~~~

4.3

+   <para>
+    Step-2: Lock the required tables of the new node in
+    <literal>EXCLUSIVE</literal> mode untilthe setup is complete. (This lock is
+    necessary to prevent any modifications from happening on the new node
+    because if data modifications occurred after Step-3, there is a chance that
+    the modifications will be published to the first node and then synchronized
+    back to the new node while creating the subscription in Step-5. This would
+    result in inconsistent data).
+   </para>
+   <para>

4.3.a
typo: "untilthe" -> "until the"

4.3.b
SUGGESTION (just for the 2nd sentence)
This lock is necessary to prevent any modifications from happening on
the new node. If data modifications occurred after Step-3, there is a
chance they could be published to the first node and then synchronized
back to the new node while creating the subscription in Step-5.


------
Kind Regards,
Peter Smith.
Fujitsu Australia

  139. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-06-23T01:35:15Z

    On Mon, Jun 20, 2022 7:55 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comment, the v22 patch attached has the changes for the
> same.

Thanks for updating the patch, here are some comments on 0003 patch.

1. 032_origin.pl
+###############################################################################
+# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B) bidirectional
+# replication setup when the existing nodes (node_A & node_B) and the new node
+# (node_C) does not have any data.
+###############################################################################

"does not have any data" should be "do not have any data" I think.

2.
The comment for 032_origin.pl:

# Test the CREATE SUBSCRIPTION 'origin' parameter.

After applying this patch, this file tests no only 'origin' parameter, but also
"copy_data" parameter, so should we modify this comment?

Besides, should we change the file name in this patch? It looks more like test
cases for bidirectional logical replication.

3. subscriptioncmds.c
	/* Set default values for the boolean supported options. */
...
	if (IsSet(supported_opts, SUBOPT_CREATE_SLOT))
 		opts->create_slot = true;
 	if (IsSet(supported_opts, SUBOPT_COPY_DATA))
-		opts->copy_data = true;
+		opts->copy_data = COPY_DATA_ON;
 	if (IsSet(supported_opts, SUBOPT_REFRESH))
 		opts->refresh = true;
 	if (IsSet(supported_opts, SUBOPT_BINARY))

"copy_data" option is not Boolean now, which is inconsistent with the comment.
So maybe we can change the comment here? ("the boolean supported options" ->
"the supported options")

Regards,
Shi yu

  140. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-23T10:06:45Z

    On Wed, Jun 22, 2022 at 12:16 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the v22* patch set.
>
> ========
> v22-0001
> ========
>
> No comments. LGTM
>
> ========
> V22-0002
> ========
>
> 2.1 doc/src/sgml/catalogs.sgml
>
> +       Possible origin values are <literal>local</literal> or
> +       <literal>any</literal>. The default is <literal>any</literal>.
>
> IMO the word "Possible" here is giving a sense of vagueness.
>
> SUGGESTION
> The origin value must be either <literal>local</literal> or
> <literal>any</literal>.

Modified

> ~~~
>
> 2.2 src/backend/commands/subscriptioncmds.c
>
> @@ -265,6 +271,29 @@ parse_subscription_options(ParseState *pstate,
> List *stmt_options,
>   opts->specified_opts |= SUBOPT_DISABLE_ON_ERR;
>   opts->disableonerr = defGetBoolean(defel);
>   }
> + else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
> + strcmp(defel->defname, "origin") == 0)
> + {
> + if (IsSet(opts->specified_opts, SUBOPT_ORIGIN))
> + errorConflictingDefElem(defel, pstate);
> +
> + opts->specified_opts |= SUBOPT_ORIGIN;
> +
> + /*
> + * Even though "origin" parameter allows only "local" and "any"
> + * values, the "origin" parameter type is implemented as string
> + * type instead of boolean to extend the "origin" parameter to
> + * support filtering of origin name specified by the user in the
> + * later versions.
> + */
> + opts->origin = defGetString(defel);
> +
> + if ((strcmp(opts->origin, LOGICALREP_ORIGIN_LOCAL) != 0) &&
> + (strcmp(opts->origin, LOGICALREP_ORIGIN_ANY) != 0))
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("unrecognized origin value: \"%s\"", opts->origin));
> + }
>
> I was wondering if it might be wise now to do a pfree(opts->origin)
> here before setting the new option value which overwrites the
> strdup-ed default "any". OTOH maybe it is overkill to worry about the
> tiny leak? I am not sure what is the convention for this.

I have freed the memory to avoid the leak,  I felt it is better to
clean the memory as it is a positive flow.

> ~~~
>
> 2.3 src/backend/replication/pgoutput/pgoutput.c
>
> @@ -1698,12 +1719,16 @@ pgoutput_message(LogicalDecodingContext *ctx,
> ReorderBufferTXN *txn,
>  }
>
>  /*
> - * Currently we always forward.
> + * Return true if the data source (origin) is remote and user has requested
> + * only local data, false otherwise.
>   */
>  static bool
>  pgoutput_origin_filter(LogicalDecodingContext *ctx,
>      RepOriginId origin_id)
>
> "user" -> "the user"

Modified

> ~~~
>
> 2.4 src/include/catalog/pg_subscription.h
>
> +/*
> + * The subscription will request the publisher to send any changes regardless
> + * of their origin
> + */
> +#define LOGICALREP_ORIGIN_ANY "any"
>
> SUGGESTION (remove 'any'; add period)
> The subscription will request the publisher to send changes regardless
> of their origin.

Modified

> ========
> v22-0003
> ========
>
> 3.1 Commit message
>
> change 1) Checks and throws an error if 'copy_data = on' and 'origin =
> local' but the publication tables were also subscribing from other publishers.
>
> "also subscribing from other publishers." -> "also replicating from
> other publishers."

Modified

> ~~~
>
> 3.2 doc/src/sgml/ref/alter_subscription.sgml
>
> +         <para>
> +          There is some interaction between the <literal>origin</literal>
> +          parameter and <literal>copy_data</literal> parameter.
>
> "and copy_data parameter." -> "and the copy_data parameter."

Modified

> ~~~
>
> 3.3 doc/src/sgml/ref/create_subscription.sgml
>
> +         <para>
> +          There is some interaction between the <literal>origin</literal>
> +          parameter and <literal>copy_data</literal> parameter.
>
> "and copy_data parameter." -> "and the copy_data parameter."

Modified

> ~~~
>
> 3.4 doc/src/sgml/ref/create_subscription.sgml
>
> +         <para>
> +          There is some interaction between the <literal>origin</literal>
> +          parameter and <literal>copy_data</literal> parameter. Refer to the
> +          <xref linkend="sql-createsubscription-notes" /> for details.
> +         </para>
>
> "and copy_data parameter." -> "and the copy_data parameter."

Modified

> ~~~
>
> 3.5 doc/src/sgml/ref/create_subscription.sgml
>
> @@ -383,6 +398,15 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   If the subscription is created with <literal>origin = local</literal> and
> +   <literal>copy_data = true</literal>, it will check if the
> publisher tables are
> +   being subscribed to any other publisher and, if so, then throw an error to
> +   prevent possible non-local data from being copied. The user can override
> +   this check and continue with the copy operation by specifying
> +   <literal>copy_data = force</literal>.
> +  </para>
> +
>
> For the part "it will check if the publisher tables are being
> subscribed to any other publisher...", the "being subscribed to"
> sounds a bit strange. Maybe it is better to reword this like you
> already worded some of the code comments.
>
> SUGGESTION
> -> "... it will check if the publisher has subscribed to the same
> table from other publishers and ..."

Modified

> ~~~
>
> 3.6 src/backend/commands/subscriptioncmds.c
>
> + /*
> + * Throw an error if the publisher has subscribed to the same table
> + * from some other publisher. We cannot differentiate between the
> + * local and non-local data that is present in the HEAP during the
> + * initial sync. Identification of local data can be done only from
> + * the WAL by using the origin id. XXX: For simplicity, we don't check
> + * whether the table has any data or not. If the table doesn't have
> + * any data then we don't need to distinguish between local and
> + * non-local data so we can avoid throwing error in that case.
> + */
>
> I think the XXX part should be after a blank like so it is more
> prominent, instead of being buried in the other text.
>
> e.g.
>
> + /*
> + * Throw an error if the publisher has subscribed to the same table
> + * from some other publisher. We cannot differentiate between the
> + * local and non-local data that is present in the HEAP during the
> + * initial sync. Identification of local data can be done only from
> + * the WAL by using the origin id.
> + *
> + * XXX: For simplicity, we don't check
> + * whether the table has any data or not. If the table doesn't have
> + * any data then we don't need to distinguish between local and
> + * non-local data so we can avoid throwing error in that case.
> + */

Modified

> ~~~
>
> 3.7 src/test/regress/sql/subscription.sql
>
> Elsewhere, there was code in this patch that apparently accepts
> 'copy_data' parameter but with no parameter value specified. But this
> combination has no test case.

I have added a test for the same

> ========
> v22-0004
> ========
>
> 4.1 doc/src/sgml/logical-replication.sgml
>
> +  <sect2 id="add-node-data-present-on-new-node">
> +   <title>Adding a new node when data is present on the new node</title>
>
> "when data is present" -> "when table data is present"

Modified

> ~~~
>
> 4.2 doc/src/sgml/logical-replication.sgml
>
> +   <note>
> +    <para>
> +     Adding a new node when data is present on the new node tables is not
> +     supported.
> +    </para>
> +   </note>
>
> I think the note text should match the title text (see #4.1)
>
> SUGGESTION
> Adding a new node when table data is present on the new node is not supported.

Modified

> ~~~
>
> 4.3
>
> +   <para>
> +    Step-2: Lock the required tables of the new node in
> +    <literal>EXCLUSIVE</literal> mode untilthe setup is complete. (This lock is
> +    necessary to prevent any modifications from happening on the new node
> +    because if data modifications occurred after Step-3, there is a chance that
> +    the modifications will be published to the first node and then synchronized
> +    back to the new node while creating the subscription in Step-5. This would
> +    result in inconsistent data).
> +   </para>
> +   <para>
>
> 4.3.a
> typo: "untilthe" -> "until the"

Modified

> 4.3.b
> SUGGESTION (just for the 2nd sentence)
> This lock is necessary to prevent any modifications from happening on
> the new node. If data modifications occurred after Step-3, there is a
> chance they could be published to the first node and then synchronized
> back to the new node while creating the subscription in Step-5.

Modified

Thanks for the comments, the attached v23 patch has the changes for the same.

Regards,
Vignesh

  141. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-23T10:08:50Z

    On Thu, Jun 23, 2022 at 7:05 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Mon, Jun 20, 2022 7:55 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comment, the v22 patch attached has the changes for the
> > same.
>
> Thanks for updating the patch, here are some comments on 0003 patch.
>
> 1. 032_origin.pl
> +###############################################################################
> +# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B) bidirectional
> +# replication setup when the existing nodes (node_A & node_B) and the new node
> +# (node_C) does not have any data.
> +###############################################################################
>
> "does not have any data" should be "do not have any data" I think.

I felt the existing is ok, I did not find it gramatically wrong. I did
not  make any changes for this.

> 2.
> The comment for 032_origin.pl:
>
> # Test the CREATE SUBSCRIPTION 'origin' parameter.
>
> After applying this patch, this file tests no only 'origin' parameter, but also
> "copy_data" parameter, so should we modify this comment?

Modified

> Besides, should we change the file name in this patch? It looks more like test
> cases for bidirectional logical replication.

I felt let's keep it as origin, as most of the tests are based on
'origin' parameter, also later features will be able to filter a
particular origin, those tests can easily fit in the same file.

> 3. subscriptioncmds.c
>         /* Set default values for the boolean supported options. */
> ...
>         if (IsSet(supported_opts, SUBOPT_CREATE_SLOT))
>                 opts->create_slot = true;
>         if (IsSet(supported_opts, SUBOPT_COPY_DATA))
> -               opts->copy_data = true;
> +               opts->copy_data = COPY_DATA_ON;
>         if (IsSet(supported_opts, SUBOPT_REFRESH))
>                 opts->refresh = true;
>         if (IsSet(supported_opts, SUBOPT_BINARY))
>
> "copy_data" option is not Boolean now, which is inconsistent with the comment.
> So maybe we can change the comment here? ("the boolean supported options" ->
> "the supported options")

Modified

Thanks for the comments, the v23 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1-ZrG%3DhaAoiB2yFKYc%2Bckcd1NLaU8QB3SWs32wPsph4w%40mail.gmail.com

Regards,
Vignesh

  142. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-24T04:50:19Z

    Here are my review comments for the v23* patch set.

========
v23-0001
========

No comments. LGTM

========
V23-0002
========

2.1 src/backend/commands/subscriptioncmds.c

+ opts->origin = defGetString(defel);
+
+ if ((strcmp(opts->origin, LOGICALREP_ORIGIN_LOCAL) != 0) &&
+ (strcmp(opts->origin, LOGICALREP_ORIGIN_ANY) != 0))
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized origin value: \"%s\"", opts->origin));
+ }

I thought maybe this should be ERRCODE_INVALID_PARAMETER_VALUE.

~~~

2.2 src/backend/replication/pgoutput/pgoutput.c

+ data->origin = defGetString(defel);
+ if (strcmp(data->origin, LOGICALREP_ORIGIN_LOCAL) == 0)
+ publish_local_origin = true;
+ else if (strcmp(data->origin, LOGICALREP_ORIGIN_ANY) == 0)
+ publish_local_origin = false;
+ else
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("unrecognized origin value: \"%s\"", data->origin));
+ }

I thought maybe this should be ERRCODE_INVALID_PARAMETER_VALUE.

========
v23-0003
========

3.1 Commit message

After the subscription is created on node2, node1 will be synced to
node2 and the newly synced data will be sent to node2, this process of
node1 sending data to node2 and node2 sending data to node1 will repeat
infinitely. If table t1 has a unique key, this will lead to a unique key
violation, and replication won't proceed.

~

31a.
"node2, this process of" -> "node2; this process of"
OR
"node2, this process of" -> "node2. This process of"

31b.
Also, my grammar checker recommends removing the comma after "violation"

~~~

3.2 doc/src/sgml/ref/create_subscription.sgml

@@ -115,7 +115,8 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
           (You cannot combine setting <literal>connect</literal>
           to <literal>false</literal> with
           setting <literal>create_slot</literal>, <literal>enabled</literal>,
-          or <literal>copy_data</literal> to <literal>true</literal>.)
+          or <literal>copy_data</literal> to
+          <literal>true</literal>/<literal>force</literal>.)
          </para>

I am not sure why that last sentence needs to be in parentheses, but
OTOH it seems to be that way already in PG15.

~~~

3.3 doc/src/sgml/ref/create_subscription.sgml

@@ -383,6 +398,15 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   If the subscription is created with <literal>origin = local</literal> and
+   <literal>copy_data = true</literal>, it will check if the publisher has
+   subscribed to the same table from other publishers and, if so, then throw an
+   error to prevent possible non-local data from being copied. The user can
+   override this check and continue with the copy operation by specifying
+   <literal>copy_data = force</literal>.
+  </para>

"and, if so, then throw..." -> "and, if so, throw..."

~~~

3.4 src/backend/commands/subscriptioncmds.c

+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires a boolean or \"force\"", def->defname));
+ return COPY_DATA_OFF; /* keep compiler quiet */
+}

I thought maybe this should be ERRCODE_INVALID_PARAMETER_VALUE.

========
v23-0004
========

4.1 Commit message

Document the steps for the following:
a) Creating a two-node bidirectional replication when there is no data
on both nodes.
b) Adding a new node when there is no data on any of the nodes.
c) Adding a new node when data is present on the existing nodes.
d) Generic steps for adding a new node to an existing set of nodes.

~

These pgdocs titles have changed slightly. I think this commit message
text should use match the current pgdocs titles.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  143. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-27T13:38:44Z

    On Fri, Jun 24, 2022 at 10:20 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the v23* patch set.
>
> ========
> v23-0001
> ========
>
> No comments. LGTM
>
> ========
> V23-0002
> ========
>
> 2.1 src/backend/commands/subscriptioncmds.c
>
> + opts->origin = defGetString(defel);
> +
> + if ((strcmp(opts->origin, LOGICALREP_ORIGIN_LOCAL) != 0) &&
> + (strcmp(opts->origin, LOGICALREP_ORIGIN_ANY) != 0))
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("unrecognized origin value: \"%s\"", opts->origin));
> + }
>
> I thought maybe this should be ERRCODE_INVALID_PARAMETER_VALUE.

Modified

> ~~~
>
> 2.2 src/backend/replication/pgoutput/pgoutput.c
>
> + data->origin = defGetString(defel);
> + if (strcmp(data->origin, LOGICALREP_ORIGIN_LOCAL) == 0)
> + publish_local_origin = true;
> + else if (strcmp(data->origin, LOGICALREP_ORIGIN_ANY) == 0)
> + publish_local_origin = false;
> + else
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("unrecognized origin value: \"%s\"", data->origin));
> + }
>
> I thought maybe this should be ERRCODE_INVALID_PARAMETER_VALUE.

Modified

> ========
> v23-0003
> ========
>
> 3.1 Commit message
>
> After the subscription is created on node2, node1 will be synced to
> node2 and the newly synced data will be sent to node2, this process of
> node1 sending data to node2 and node2 sending data to node1 will repeat
> infinitely. If table t1 has a unique key, this will lead to a unique key
> violation, and replication won't proceed.
>
> ~
>
> 31a.
> "node2, this process of" -> "node2; this process of"
> OR
> "node2, this process of" -> "node2. This process of"

Modified

> 31b.
> Also, my grammar checker recommends removing the comma after "violation"

Modified

> ~~~
>
> 3.2 doc/src/sgml/ref/create_subscription.sgml
>
> @@ -115,7 +115,8 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>            (You cannot combine setting <literal>connect</literal>
>            to <literal>false</literal> with
>            setting <literal>create_slot</literal>, <literal>enabled</literal>,
> -          or <literal>copy_data</literal> to <literal>true</literal>.)
> +          or <literal>copy_data</literal> to
> +          <literal>true</literal>/<literal>force</literal>.)
>           </para>
>
> I am not sure why that last sentence needs to be in parentheses, but
> OTOH it seems to be that way already in PG15.

I feel since it is like that since PG15, let's keep it in parenthesis
like earlier. I have not made any changes for this.

> ~~~
>
> 3.3 doc/src/sgml/ref/create_subscription.sgml
>
> @@ -383,6 +398,15 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   If the subscription is created with <literal>origin = local</literal> and
> +   <literal>copy_data = true</literal>, it will check if the publisher has
> +   subscribed to the same table from other publishers and, if so, then throw an
> +   error to prevent possible non-local data from being copied. The user can
> +   override this check and continue with the copy operation by specifying
> +   <literal>copy_data = force</literal>.
> +  </para>
>
> "and, if so, then throw..." -> "and, if so, throw..."

Modified

> ~~~
>
> 3.4 src/backend/commands/subscriptioncmds.c
>
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("%s requires a boolean or \"force\"", def->defname));
> + return COPY_DATA_OFF; /* keep compiler quiet */
> +}
>
> I thought maybe this should be ERRCODE_INVALID_PARAMETER_VALUE.

Modified

> ========
> v23-0004
> ========
>
> 4.1 Commit message
>
> Document the steps for the following:
> a) Creating a two-node bidirectional replication when there is no data
> on both nodes.
> b) Adding a new node when there is no data on any of the nodes.
> c) Adding a new node when data is present on the existing nodes.
> d) Generic steps for adding a new node to an existing set of nodes.
>
> ~
>
> These pgdocs titles have changed slightly. I think this commit message
> text should use match the current pgdocs titles.

Modified

Thanks for the comments, the attached v24 patch has the changes for the same.

Regards,
Vignesh

  144. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-06-28T01:58:49Z

    Here are my review comments for the v24* patch set.

Now, these few comments are all trivial and non-functional. Apart from
these, everything looks good to me.

========
v24-0001
========

No comments. LGTM

========
V24-0002
========

2.1 doc/src/sgml/ref/create_subscription.sgml

+         <para>
+          Specifies whether the subscription will request the publisher to only
+          send changes that originated locally, or to send any changes
+          regardless of origin. Setting <literal>origin</literal> to
+          <literal>local</literal> means that the subscription will request the
+          publisher to only send changes that originated locally. Setting
+          <literal>origin</literal> to <literal>any</literal> means that the
+          publisher sends any changes regardless of their origin. The default
+          is <literal>any</literal>.
+         </para>

2.1a.
IMO remove the word "any" from "any changes". Then the text will match
what is written in catalogs.sgml.
"send any changes regardless of origin" -> "send changes regardless of
origin" (occurs 2x)

2.1b.
This same text is cut/paste to the commit message so that can also be updated.


~~~

2.2 src/backend/commands/subscriptioncmds.c

+ /*
+ * Even though "origin" parameter allows only "local" and "any"
+ * values, the "origin" parameter type is implemented as string
+ * type instead of boolean to extend the "origin" parameter to
+ * support filtering of origin name specified by the user in the
+ * later versions.
+ */

2.2a.
SUGGESTION
Even though "origin" parameter allows only "local" and "any" values,
it is implemented as a string type so that the parameter can be
extended in future versions to support filtering using origin names
specified by the user.

2.2b.
This same text is cut/paste to the commit message so that can also be updated.


========
v24-0003
========

3.1 Commit message

This patch does a couple of things:
change 1) Checks and throws an error if 'copy_data = on' and 'origin =
local' but the publication tables were also replicating from other publishers.
change 2) Adds 'force' value for copy_data parameter.

~

"replicating" -> "replicated"
"change 1)" -> "1)"
"change 2)" -> "2)"

========
v24-0004
========

No comments. LGTM

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  145. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-28T06:17:53Z

    On Tue, Jun 28, 2022 at 7:29 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the v24* patch set.
>
> Now, these few comments are all trivial and non-functional. Apart from
> these, everything looks good to me.
>
> ========
> v24-0001
> ========
>
> No comments. LGTM
>
> ========
> V24-0002
> ========
>
> 2.1 doc/src/sgml/ref/create_subscription.sgml
>
> +         <para>
> +          Specifies whether the subscription will request the publisher to only
> +          send changes that originated locally, or to send any changes
> +          regardless of origin. Setting <literal>origin</literal> to
> +          <literal>local</literal> means that the subscription will request the
> +          publisher to only send changes that originated locally. Setting
> +          <literal>origin</literal> to <literal>any</literal> means that the
> +          publisher sends any changes regardless of their origin. The default
> +          is <literal>any</literal>.
> +         </para>
>
> 2.1a.
> IMO remove the word "any" from "any changes". Then the text will match
> what is written in catalogs.sgml.
> "send any changes regardless of origin" -> "send changes regardless of
> origin" (occurs 2x)

Modified

> 2.1b.
> This same text is cut/paste to the commit message so that can also be updated.

Modified

>
> ~~~
>
> 2.2 src/backend/commands/subscriptioncmds.c
>
> + /*
> + * Even though "origin" parameter allows only "local" and "any"
> + * values, the "origin" parameter type is implemented as string
> + * type instead of boolean to extend the "origin" parameter to
> + * support filtering of origin name specified by the user in the
> + * later versions.
> + */
>
> 2.2a.
> SUGGESTION
> Even though "origin" parameter allows only "local" and "any" values,
> it is implemented as a string type so that the parameter can be
> extended in future versions to support filtering using origin names
> specified by the user.

Modified

> 2.2b.
> This same text is cut/paste to the commit message so that can also be updated.

Modified

>
> ========
> v24-0003
> ========
>
> 3.1 Commit message
>

> v24-0004
> ========
>
> No comments. LGTM
>
> ------
> Kind Regards,
> Peter Smith.
> This patch does a couple of things:
> change 1) Checks and throws an error if 'copy_data = on' and 'origin =
> local' but the publication tables were also replicating from other publishers.
> change 2) Adds 'force' value for copy_data parameter.
>
> ~
>
> "replicating" -> "replicated"
> "change 1)" -> "1)"
> "change 2)" -> "2)"

Modified

Thanks for the comments, the attached  v25 patch has the changes for the same.

Regards,
Vignesh

  146. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-06-30T03:47:24Z

    On Tue, Jun 28, 2022 2:18 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached  v25 patch has the changes for the
> same.
> 

Thanks for updating the patch. Here are some comments.

0002 patch:
==============
1.
+# Test the CREATE SUBSCRIPTION 'origin' parameter and its interaction with
+# 'copy_data' parameter.

It seems we should move "and its interaction with 'copy_data' parameter" to
0003 patch.

0003 patch
==============
1.
When using ALTER SUBSCRIPTION ... REFRESH, subscription will throw an error if
any table is subscribed in publisher, even if the table has been subscribed
before refresh (which won't do the initial copy when refreshing). It looks the
previously subscribed tables don't need this check. Would it be better that we
only check the tables which need to do the initial copy?

2.
+				errmsg("table:%s.%s might have replicated data in the publisher",
+					   nspname, relname),

I think the table name needs to be enclosed in double quotes, which is
consistent with other messages.

Regards,
Shi yu

  147. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-06-30T16:10:14Z

    On Thu, Jun 30, 2022 at 9:17 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Tue, Jun 28, 2022 2:18 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached  v25 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch. Here are some comments.
>
> 0002 patch:
> ==============
> 1.
> +# Test the CREATE SUBSCRIPTION 'origin' parameter and its interaction with
> +# 'copy_data' parameter.
>
> It seems we should move "and its interaction with 'copy_data' parameter" to
> 0003 patch.

Modified

> 0003 patch
> ==============
> 1.
> When using ALTER SUBSCRIPTION ... REFRESH, subscription will throw an error if
> any table is subscribed in publisher, even if the table has been subscribed
> before refresh (which won't do the initial copy when refreshing). It looks the
> previously subscribed tables don't need this check. Would it be better that we
> only check the tables which need to do the initial copy?

Modified

> 2.
> +                               errmsg("table:%s.%s might have replicated data in the publisher",
> +                                          nspname, relname),
>
> I think the table name needs to be enclosed in double quotes, which is
> consistent with other messages.

Modified

Thanks for the comments, the attached v26 patch has the changes for the same.

Regards,
Vignesh

  148. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-02T06:42:35Z

    On Thu, Jun 30, 2022 at 9:40 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, Jun 30, 2022 at 9:17 AM shiy.fnst@fujitsu.com
> <shiy.fnst@fujitsu.com> wrote:
> >

The first patch that adds a test case for existing functionality looks
good to me and I'll push that early next week (by Tuesday) unless
there are more comments on it.

Few minor comments on 0002
========================
1.
+ /* FIXME: 150000 should be changed to 160000 later for PG16. */
+ if (options->proto.logical.origin &&
+ PQserverVersion(conn->streamConn) >= 150000)
+ appendStringInfo(&cmd, ", origin '%s'",
+ options->proto.logical.origin);

...
...
+ /* FIXME: 150000 should be changed to 160000 later for PG16. */
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, " s.suborigin\n");
+ else
+ appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
...
...
+ /* FIXME: 150000 should be changed to 160000 later for PG16 */
+ if (pset.sversion >= 150000)
+ appendPQExpBuffer(&buf,
+   ", suborigin AS \"%s\"\n",
+   gettext_noop("Origin"));

All these should now change to 16.

2.
/* ALTER SUBSCRIPTION <name> SET ( */
  else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("SET", "("))
- COMPLETE_WITH("binary", "slot_name", "streaming",
"synchronous_commit", "disable_on_error");
+ COMPLETE_WITH("binary", "origin", "slot_name", "streaming",
"synchronous_commit", "disable_on_error");
  /* ALTER SUBSCRIPTION <name> SKIP ( */
  else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("SKIP", "("))
  COMPLETE_WITH("lsn");
@@ -3152,7 +3152,7 @@ psql_completion(const char *text, int start, int end)
  /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
  else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
  COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
-   "enabled", "slot_name", "streaming",
+   "enabled", "origin", "slot_name", "streaming",
    "synchronous_commit", "two_phase", "disable_on_error");

Why do you choose to add a new option in-between other parameters
instead of at the end which we normally do? The one possible reason I
can think of is that all the parameters at the end are boolean so you
want to add this before those but then why before slot_name, and again
I don't see such a rule being followed for other parameters.

-- 
With Regards,
Amit Kapila.

  149. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-03T14:59:37Z

    On Sat, Jul 2, 2022 at 12:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jun 30, 2022 at 9:40 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Thu, Jun 30, 2022 at 9:17 AM shiy.fnst@fujitsu.com
> > <shiy.fnst@fujitsu.com> wrote:
> > >
>
> The first patch that adds a test case for existing functionality looks
> good to me and I'll push that early next week (by Tuesday) unless
> there are more comments on it.
>
> Few minor comments on 0002
> ========================
> 1.
> + /* FIXME: 150000 should be changed to 160000 later for PG16. */
> + if (options->proto.logical.origin &&
> + PQserverVersion(conn->streamConn) >= 150000)
> + appendStringInfo(&cmd, ", origin '%s'",
> + options->proto.logical.origin);
>
> ...
> ...
> + /* FIXME: 150000 should be changed to 160000 later for PG16. */
> + if (fout->remoteVersion >= 150000)
> + appendPQExpBufferStr(query, " s.suborigin\n");
> + else
> + appendPQExpBuffer(query, " '%s' AS suborigin\n", LOGICALREP_ORIGIN_ANY);
> ...
> ...
> + /* FIXME: 150000 should be changed to 160000 later for PG16 */
> + if (pset.sversion >= 150000)
> + appendPQExpBuffer(&buf,
> +   ", suborigin AS \"%s\"\n",
> +   gettext_noop("Origin"));
>
> All these should now change to 16.

Modified

> 2.
> /* ALTER SUBSCRIPTION <name> SET ( */
>   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> TailMatches("SET", "("))
> - COMPLETE_WITH("binary", "slot_name", "streaming",
> "synchronous_commit", "disable_on_error");
> + COMPLETE_WITH("binary", "origin", "slot_name", "streaming",
> "synchronous_commit", "disable_on_error");
>   /* ALTER SUBSCRIPTION <name> SKIP ( */
>   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> TailMatches("SKIP", "("))
>   COMPLETE_WITH("lsn");
> @@ -3152,7 +3152,7 @@ psql_completion(const char *text, int start, int end)
>   /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
>   else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
>   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> -   "enabled", "slot_name", "streaming",
> +   "enabled", "origin", "slot_name", "streaming",
>     "synchronous_commit", "two_phase", "disable_on_error");
>
> Why do you choose to add a new option in-between other parameters
> instead of at the end which we normally do? The one possible reason I
> can think of is that all the parameters at the end are boolean so you
> want to add this before those but then why before slot_name, and again
> I don't see such a rule being followed for other parameters.

I was not sure if it should be maintained in alphabetical order,
anyway since the last option "disable_on_error" is at the end, I have
changed it to the end.

Thanks for the comments, the attached v27 patch has the changes for the same.

Regards,
Vignesh

  150. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-03T22:29:03Z

    On Mon, Jul 4, 2022 at 12:59 AM vignesh C <vignesh21@gmail.com> wrote:
...
> > 2.
> > /* ALTER SUBSCRIPTION <name> SET ( */
> >   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > TailMatches("SET", "("))
> > - COMPLETE_WITH("binary", "slot_name", "streaming",
> > "synchronous_commit", "disable_on_error");
> > + COMPLETE_WITH("binary", "origin", "slot_name", "streaming",
> > "synchronous_commit", "disable_on_error");
> >   /* ALTER SUBSCRIPTION <name> SKIP ( */
> >   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > TailMatches("SKIP", "("))
> >   COMPLETE_WITH("lsn");
> > @@ -3152,7 +3152,7 @@ psql_completion(const char *text, int start, int end)
> >   /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
> >   else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
> >   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> > -   "enabled", "slot_name", "streaming",
> > +   "enabled", "origin", "slot_name", "streaming",
> >     "synchronous_commit", "two_phase", "disable_on_error");
> >
> > Why do you choose to add a new option in-between other parameters
> > instead of at the end which we normally do? The one possible reason I
> > can think of is that all the parameters at the end are boolean so you
> > want to add this before those but then why before slot_name, and again
> > I don't see such a rule being followed for other parameters.
>
> I was not sure if it should be maintained in alphabetical order,
> anyway since the last option "disable_on_error" is at the end, I have
> changed it to the end.
>

Although it seems it is not a hard rule, mostly the COMPLETE_WITH are
coded using alphabetical order. Anyway, I think that was a clear
intention here too since 13 of 14 parameters were already in
alphabetical order; it is actually only that "disable_on_error"
parameter that was misplaced; not the new "origin" parameter.

Also, in practice, on <tab> those completions will get output in
alphabetical order, so IMO it makes more sense for the code to be
consistent with the output:

e.g.
test_sub=# create subscription sub xxx connection '' publication pub WITH (
BINARY              DISABLE_ON_ERROR    STREAMING
CONNECT             ENABLED             SYNCHRONOUS_COMMIT
COPY_DATA           ORIGIN              TWO_PHASE
CREATE_SLOT         SLOT_NAME
test_sub=#

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  151. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-04T03:52:58Z

    On Mon, Jul 4, 2022 at 3:59 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Jul 4, 2022 at 12:59 AM vignesh C <vignesh21@gmail.com> wrote:
> ...
> > > 2.
> > > /* ALTER SUBSCRIPTION <name> SET ( */
> > >   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > > TailMatches("SET", "("))
> > > - COMPLETE_WITH("binary", "slot_name", "streaming",
> > > "synchronous_commit", "disable_on_error");
> > > + COMPLETE_WITH("binary", "origin", "slot_name", "streaming",
> > > "synchronous_commit", "disable_on_error");
> > >   /* ALTER SUBSCRIPTION <name> SKIP ( */
> > >   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > > TailMatches("SKIP", "("))
> > >   COMPLETE_WITH("lsn");
> > > @@ -3152,7 +3152,7 @@ psql_completion(const char *text, int start, int end)
> > >   /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
> > >   else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
> > >   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> > > -   "enabled", "slot_name", "streaming",
> > > +   "enabled", "origin", "slot_name", "streaming",
> > >     "synchronous_commit", "two_phase", "disable_on_error");
> > >
> > > Why do you choose to add a new option in-between other parameters
> > > instead of at the end which we normally do? The one possible reason I
> > > can think of is that all the parameters at the end are boolean so you
> > > want to add this before those but then why before slot_name, and again
> > > I don't see such a rule being followed for other parameters.
> >
> > I was not sure if it should be maintained in alphabetical order,
> > anyway since the last option "disable_on_error" is at the end, I have
> > changed it to the end.
> >
>
> Although it seems it is not a hard rule, mostly the COMPLETE_WITH are
> coded using alphabetical order. Anyway, I think that was a clear
> intention here too since 13 of 14 parameters were already in
> alphabetical order; it is actually only that "disable_on_error"
> parameter that was misplaced; not the new "origin" parameter.
>

Agreed, but let's not change disable_on_error as part of this patch.

-- 
With Regards,
Amit Kapila.

  152. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-07-04T08:16:45Z

    On Sun, Jul 3, 2022 11:00 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached v27 patch has the changes for the
> same.
> 

Thanks for updating the patch.

A comment on 0003 patch:
+		/*
+		 * No need to throw an error for the tables that are in ready state,
+		 * as the walsender will send the changes from WAL in case of tables
+		 * in ready state.
+		 */
+		if (isreadytable)
+			continue;
+
...
+		ereport(ERROR,
+				errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+				errmsg("table: \"%s.%s\" might have replicated data in the publisher",
+					   nspname, relname),
+				errdetail("CREATE/ALTER SUBSCRIPTION with origin = local and copy_data = on is not allowed when the publisher might have replicated data."),
+				errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."));
+
+		ExecClearTuple(slot);

I think we should call ExecClearTuple() before getting next tuple, so it should
be called if the table is in ready state. How about modifying it to:
		if (!isreadytable)
			ereport(ERROR,
					errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
					errmsg("table: \"%s.%s\" might have replicated data in the publisher",
						   nspname, relname),
					errdetail("CREATE/ALTER SUBSCRIPTION with origin = local and copy_data = on is not allowed when the publisher might have replicated data."),
					errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."));

		ExecClearTuple(slot);

Regards,
Shi yu

  153. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-07-04T08:41:17Z

    On Mon, Jul 4, 2022 4:17 PM shiy.fnst@fujitsu.com <shiy.fnst@fujitsu.com> wrote:
> 
> On Sun, Jul 3, 2022 11:00 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v27 patch has the changes for the
> > same.
> >
> 
> Thanks for updating the patch.
> 
> A comment on 0003 patch:
> 
> I think we should call ExecClearTuple() before getting next tuple, so it should
> be called if the table is in ready state. How about modifying it to:
> 

By the way, I have tested pg_dump/psql changes in this patch with older version
(server: pg10 ~ pg15, pg_dump/psql: pg16), and it worked ok.

Regards,
Shi yu

  154. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-04T10:18:33Z

    On Sun, Jul 3, 2022 at 8:29 PM vignesh C <vignesh21@gmail.com> wrote:
>

Review comments
===============
1.
@@ -530,7 +557,7 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
    SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
    SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
    SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
-   SUBOPT_DISABLE_ON_ERR);
+   SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
  parse_subscription_options(pstate, stmt->options, supported_opts, &opts);

  /*
@@ -606,6 +633,8 @@ CreateSubscription(ParseState *pstate,
CreateSubscriptionStmt *stmt,
  LOGICALREP_TWOPHASE_STATE_PENDING :
  LOGICALREP_TWOPHASE_STATE_DISABLED);
  values[Anum_pg_subscription_subdisableonerr - 1] =
BoolGetDatum(opts.disableonerr);
+ values[Anum_pg_subscription_suborigin - 1] =
+ CStringGetTextDatum(opts.origin);
  values[Anum_pg_subscription_subconninfo - 1] =
  CStringGetTextDatum(conninfo);
  if (opts.slot_name)
...
...

  /* List of publications subscribed to */
  text subpublications[1] BKI_FORCE_NOT_NULL;
+
+ /* Only publish data originating from the specified origin */
+ text suborigin BKI_DEFAULT(LOGICALREP_ORIGIN_ANY);
 #endif
 } FormData_pg_subscription;

The order of declaration and assignment for 'suborigin' should match
in above usage.

2. Similarly the changes in GetSubscription() should also match the
declaration of the origin column.

3.
 GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
-              subbinary, substream, subtwophasestate,
subdisableonerr, subslotname,
-              subsynccommit, subpublications)
+              subbinary, substream, subtwophasestate, subdisableonerr,
+              suborigin, subslotname, subsynccommit, subpublications)
     ON pg_subscription TO public;

This should also match the order of columns as in pg_subscription.h
unless there is a reason for not doing so.

4.
+ /*
+ * Even though "origin" parameter allows only "local" and "any"
+ * values, it is implemented as a string type so that the parameter
+ * can be extended in future versions to support filtering using
+ * origin names specified by the user.

/Even though "origin" .../Even though the "origin" parameter ...

5.
+
+# Create tables on node_A
+$node_A->safe_psql('postgres', "CREATE TABLE tab_full (a int PRIMARY KEY)");
+
+# Create the same tables on node_B
+$node_B->safe_psql('postgres', "CREATE TABLE tab_full (a int PRIMARY KEY)");

In both the above comments, you should use table instead of tables as
the test creates only one table.

6.
+$node_A->safe_psql('postgres', "DELETE FROM tab_full;");

After this, the test didn't ensure that this operation is replicated.
Can't that lead to unpredictable results for the other tests after
this test?

7.
+# Setup logical replication
+# node_C (pub) -> node_B (sub)
+my $node_C_connstr = $node_C->connstr . ' dbname=postgres';
+$node_C->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_C FOR TABLE tab_full");
+
+my $appname_B2 = 'tap_sub_B2';
+$node_B->safe_psql(
+ 'postgres', "
+ CREATE SUBSCRIPTION tap_sub_B2
+ CONNECTION '$node_C_connstr application_name=$appname_B2'
+ PUBLICATION tap_pub_C
+ WITH (origin = local)");
+
+$node_C->wait_for_catchup($appname_B2);
+
+$node_C->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for subscriber to synchronize data";

This test allows the publisher (node_C) to poll for sync but it should
be the subscriber (node_B) that needs to poll to allow the initial
sync to finish.

8. Do you think it makes sense to see if this new option can also be
supported by pg_recvlogical? I see that previously we have not
extended pg_recvlogical for all the newly added options but I feel we
should keep pg_recvlogical up to date w.r.t new options. We can do
this as a separate patch if we agree?

-- 
With Regards,
Amit Kapila.

  155. Re: Handle infinite recursion in logical replication setup

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2022-07-04T14:14:02Z

    > From d8f8844f877806527b6f3f45320b6ba55a8e3154 Mon Sep 17 00:00:00 2001
> From: Vigneshwaran C <vignesh21@gmail.com>
> Date: Thu, 26 May 2022 19:29:33 +0530
> Subject: [PATCH v27 1/4] Add a missing test to verify only-local parameter in
>  test_decoding plugin.
> 
> Add a missing test to verify only-local parameter in test_decoding plugin.

I don't get it.  replorigin.sql already has some lines to test
local-only.  What is your patch adding that is new?  Maybe instead of
adding some more lines at the end of the script, you should add lines
where this stuff is already being tested.  But that assumes that there
is something new that is being tested; if so what is it?

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Investigación es lo que hago cuando no sé lo que estoy haciendo"
(Wernher von Braun)

  156. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-04T17:59:27Z

    On Mon, Jul 4, 2022 at 3:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Jul 3, 2022 at 8:29 PM vignesh C <vignesh21@gmail.com> wrote:
> >
>
> Review comments
> ===============
> 1.
> @@ -530,7 +557,7 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>     SUBOPT_SLOT_NAME | SUBOPT_COPY_DATA |
>     SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
>     SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
> -   SUBOPT_DISABLE_ON_ERR);
> +   SUBOPT_DISABLE_ON_ERR | SUBOPT_ORIGIN);
>   parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
>
>   /*
> @@ -606,6 +633,8 @@ CreateSubscription(ParseState *pstate,
> CreateSubscriptionStmt *stmt,
>   LOGICALREP_TWOPHASE_STATE_PENDING :
>   LOGICALREP_TWOPHASE_STATE_DISABLED);
>   values[Anum_pg_subscription_subdisableonerr - 1] =
> BoolGetDatum(opts.disableonerr);
> + values[Anum_pg_subscription_suborigin - 1] =
> + CStringGetTextDatum(opts.origin);
>   values[Anum_pg_subscription_subconninfo - 1] =
>   CStringGetTextDatum(conninfo);
>   if (opts.slot_name)
> ...
> ...
>
>   /* List of publications subscribed to */
>   text subpublications[1] BKI_FORCE_NOT_NULL;
> +
> + /* Only publish data originating from the specified origin */
> + text suborigin BKI_DEFAULT(LOGICALREP_ORIGIN_ANY);
>  #endif
>  } FormData_pg_subscription;
>
> The order of declaration and assignment for 'suborigin' should match
> in above usage.

Modified

> 2. Similarly the changes in GetSubscription() should also match the
> declaration of the origin column.

Modified

> 3.
>  GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
> -              subbinary, substream, subtwophasestate,
> subdisableonerr, subslotname,
> -              subsynccommit, subpublications)
> +              subbinary, substream, subtwophasestate, subdisableonerr,
> +              suborigin, subslotname, subsynccommit, subpublications)
>      ON pg_subscription TO public;
>
> This should also match the order of columns as in pg_subscription.h
> unless there is a reason for not doing so.

Modified

> 4.
> + /*
> + * Even though "origin" parameter allows only "local" and "any"
> + * values, it is implemented as a string type so that the parameter
> + * can be extended in future versions to support filtering using
> + * origin names specified by the user.
>
> /Even though "origin" .../Even though the "origin" parameter ...

Modified

> 5.
> +
> +# Create tables on node_A
> +$node_A->safe_psql('postgres', "CREATE TABLE tab_full (a int PRIMARY KEY)");
> +
> +# Create the same tables on node_B
> +$node_B->safe_psql('postgres', "CREATE TABLE tab_full (a int PRIMARY KEY)");
>
> In both the above comments, you should use table instead of tables as
> the test creates only one table.

Modified

> 6.
> +$node_A->safe_psql('postgres', "DELETE FROM tab_full;");
>
> After this, the test didn't ensure that this operation is replicated.
> Can't that lead to unpredictable results for the other tests after
> this test?

Modified to include wait_for_catchup and verified the data at the
beginning of the next test

> 7.
> +# Setup logical replication
> +# node_C (pub) -> node_B (sub)
> +my $node_C_connstr = $node_C->connstr . ' dbname=postgres';
> +$node_C->safe_psql('postgres',
> + "CREATE PUBLICATION tap_pub_C FOR TABLE tab_full");
> +
> +my $appname_B2 = 'tap_sub_B2';
> +$node_B->safe_psql(
> + 'postgres', "
> + CREATE SUBSCRIPTION tap_sub_B2
> + CONNECTION '$node_C_connstr application_name=$appname_B2'
> + PUBLICATION tap_pub_C
> + WITH (origin = local)");
> +
> +$node_C->wait_for_catchup($appname_B2);
> +
> +$node_C->poll_query_until('postgres', $synced_query)
> +  or die "Timed out while waiting for subscriber to synchronize data";
>
> This test allows the publisher (node_C) to poll for sync but it should
> be the subscriber (node_B) that needs to poll to allow the initial
> sync to finish.

Modified

> 8. Do you think it makes sense to see if this new option can also be
> supported by pg_recvlogical? I see that previously we have not
> extended pg_recvlogical for all the newly added options but I feel we
> should keep pg_recvlogical up to date w.r.t new options. We can do
> this as a separate patch if we agree?

I will analyze this and post my analysis soon.

Thanks for the comments, the v28 patch attached has the changes for the same.

Regards,
Vignesh

  157. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-04T18:00:53Z

    On Mon, Jul 4, 2022 at 1:46 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Sun, Jul 3, 2022 11:00 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v27 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch.
>
> A comment on 0003 patch:
> +               /*
> +                * No need to throw an error for the tables that are in ready state,
> +                * as the walsender will send the changes from WAL in case of tables
> +                * in ready state.
> +                */
> +               if (isreadytable)
> +                       continue;
> +
> ...
> +               ereport(ERROR,
> +                               errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> +                               errmsg("table: \"%s.%s\" might have replicated data in the publisher",
> +                                          nspname, relname),
> +                               errdetail("CREATE/ALTER SUBSCRIPTION with origin = local and copy_data = on is not allowed when the publisher might have replicated data."),
> +                               errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."));
> +
> +               ExecClearTuple(slot);
>
> I think we should call ExecClearTuple() before getting next tuple, so it should
> be called if the table is in ready state. How about modifying it to:
>                 if (!isreadytable)
>                         ereport(ERROR,
>                                         errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
>                                         errmsg("table: \"%s.%s\" might have replicated data in the publisher",
>                                                    nspname, relname),
>                                         errdetail("CREATE/ALTER SUBSCRIPTION with origin = local and copy_data = on is not allowed when the publisher might have replicated data."),
>                                         errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."));
>
>                 ExecClearTuple(slot);

Thanks for the comment, I have modified this in the v28 patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3MNK2hMYroTiHGS9HkSxiA-az1QC1mpa0YwDZ8nGmmZg%40mail.gmail.com

Regards,
Vignesh

  158. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-04T18:01:55Z

    On Mon, Jul 4, 2022 at 9:23 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 4, 2022 at 3:59 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Mon, Jul 4, 2022 at 12:59 AM vignesh C <vignesh21@gmail.com> wrote:
> > ...
> > > > 2.
> > > > /* ALTER SUBSCRIPTION <name> SET ( */
> > > >   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > > > TailMatches("SET", "("))
> > > > - COMPLETE_WITH("binary", "slot_name", "streaming",
> > > > "synchronous_commit", "disable_on_error");
> > > > + COMPLETE_WITH("binary", "origin", "slot_name", "streaming",
> > > > "synchronous_commit", "disable_on_error");
> > > >   /* ALTER SUBSCRIPTION <name> SKIP ( */
> > > >   else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
> > > > TailMatches("SKIP", "("))
> > > >   COMPLETE_WITH("lsn");
> > > > @@ -3152,7 +3152,7 @@ psql_completion(const char *text, int start, int end)
> > > >   /* Complete "CREATE SUBSCRIPTION <name> ...  WITH ( <opt>" */
> > > >   else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
> > > >   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> > > > -   "enabled", "slot_name", "streaming",
> > > > +   "enabled", "origin", "slot_name", "streaming",
> > > >     "synchronous_commit", "two_phase", "disable_on_error");
> > > >
> > > > Why do you choose to add a new option in-between other parameters
> > > > instead of at the end which we normally do? The one possible reason I
> > > > can think of is that all the parameters at the end are boolean so you
> > > > want to add this before those but then why before slot_name, and again
> > > > I don't see such a rule being followed for other parameters.
> > >
> > > I was not sure if it should be maintained in alphabetical order,
> > > anyway since the last option "disable_on_error" is at the end, I have
> > > changed it to the end.
> > >
> >
> > Although it seems it is not a hard rule, mostly the COMPLETE_WITH are
> > coded using alphabetical order. Anyway, I think that was a clear
> > intention here too since 13 of 14 parameters were already in
> > alphabetical order; it is actually only that "disable_on_error"
> > parameter that was misplaced; not the new "origin" parameter.
> >
>
> Agreed, but let's not change disable_on_error as part of this patch.

I have changed this in the v28 patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3MNK2hMYroTiHGS9HkSxiA-az1QC1mpa0YwDZ8nGmmZg%40mail.gmail.com

Regards,
Vignesh

  159. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-04T18:03:13Z

    On Mon, Jul 4, 2022 at 7:44 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> > From d8f8844f877806527b6f3f45320b6ba55a8e3154 Mon Sep 17 00:00:00 2001
> > From: Vigneshwaran C <vignesh21@gmail.com>
> > Date: Thu, 26 May 2022 19:29:33 +0530
> > Subject: [PATCH v27 1/4] Add a missing test to verify only-local parameter in
> >  test_decoding plugin.
> >
> > Add a missing test to verify only-local parameter in test_decoding plugin.
>
> I don't get it.  replorigin.sql already has some lines to test
> local-only.  What is your patch adding that is new?  Maybe instead of
> adding some more lines at the end of the script, you should add lines
> where this stuff is already being tested.  But that assumes that there
> is something new that is being tested; if so what is it?

The test is to check that remote origin data (i.e. replication origin
being set) will be filtered when only-local parameter is set. I felt
that this scenario is not covered, unless I'm missing something. I
have moved the test as suggested.
I have changed this in the v28 patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3MNK2hMYroTiHGS9HkSxiA-az1QC1mpa0YwDZ8nGmmZg%40mail.gmail.com

Regards,
Vignesh

  160. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-05T00:44:32Z

    I checked again the v26* patch set.

I had no more comments for v26-0001, v26-0002, or v26-0004, but below
are some comments for v26-0003.

========
v26-0003
========

3.1 src/backend/catalog/pg_subscription.c

3.1.a
+/*
+ * Get all relations for subscription that are in a ready state.
+ *
+ * Returned list is palloc'ed in current memory context.
+ */
+List *
+GetSubscriptionReadyRelations(Oid subid)

"subscription" -> "the subscription"

Also, It might be better to say in the function header what kind of
structures are in the returned List. E.g. Firstly, I'd assumed it was
the same return type as the other function
GetSubscriptionReadyRelations, but it isn’t.

3.1.b
I think there might be a case to be made for *combining* those
SubscriptionRelState and SubscripotionRel structs into a single common
struct. Then all those GetSubscriptionNotReadyRelations and
GetSubscriptionNotReadyRelations (also GetSubscriptionRelations?) can
be merged to be just one common function that takes a parameter to say
do you want to return the relation List of ALL / READY /  NOT_READY?
What do you think?

======

3.2 src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

+/*
+ * Check and throw an error if the publisher has subscribed to the same table
+ * from some other publisher. This check is required only if copydata is ON and
+ * the origin is local.
+ */
+static void
+check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
+    CopyData copydata, char *origin, Oid subid)

The function comment probably should say something about why the new
READY logic was added in v26.

~~~

3.3

3.3.a
+ /*
+ * The subid will be valid only for ALTER SUBSCRIPTION ... REFRESH
+ * PUBLICATION. Get the ready relations for the subscription only in case
+ * of ALTER SUBSCRIPTION case as there will be no relations in ready state
+ * while the subscription is created.
+ */
+ if (subid != InvalidOid)
+ subreadyrels = GetSubscriptionReadyRelations(subid);

The word "case" is 2x in the same sentence. I also paraphrased my
understanding of this comment below. Maybe it is simpler?

SUGGESTION
Get the ready relations for the subscription. The subid will be valid
only for ALTER SUBSCRIPTION ... REFRESH because there will be no
relations in ready state while the subscription is created.

3.3.b
+ if (subid != InvalidOid)
+ subreadyrels = GetSubscriptionReadyRelations(subid);

SUGGESTION
if (OidIsValid(subid))

======

3.4 src/test/subscription/t/032_localonly.pl

Now all the test cases are re-using the same data (e.g. 11,12,13) and
you are deleting the data between the tests. I guess it is OK, but IMO
the tests are easier to read when each test part was using unique data
(e.g. 11,12,13, then 12,22,32, then 13,23,33 etc)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  161. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-05T08:29:28Z

    On Mon, Jul 4, 2022 at 11:33 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Jul 4, 2022 at 7:44 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> >
> > > From d8f8844f877806527b6f3f45320b6ba55a8e3154 Mon Sep 17 00:00:00 2001
> > > From: Vigneshwaran C <vignesh21@gmail.com>
> > > Date: Thu, 26 May 2022 19:29:33 +0530
> > > Subject: [PATCH v27 1/4] Add a missing test to verify only-local parameter in
> > >  test_decoding plugin.
> > >
> > > Add a missing test to verify only-local parameter in test_decoding plugin.
> >
> > I don't get it.  replorigin.sql already has some lines to test
> > local-only.  What is your patch adding that is new?  Maybe instead of
> > adding some more lines at the end of the script, you should add lines
> > where this stuff is already being tested.  But that assumes that there
> > is something new that is being tested; if so what is it?
>
> The test is to check that remote origin data (i.e. replication origin
> being set) will be filtered when only-local parameter is set. I felt
> that this scenario is not covered, unless I'm missing something.
>

If we change only-local option to '0' in the below part of the test,
we can see a different result:
-- and magically the replayed xact will be filtered!
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local',
'1');

So, I think only-local functionality is being tested but I feel your
test is clear in the sense that it clearly shows that remote data can
only be fetched with 'only-local' as '0' when the origin is set. It
seems to me that in existing tests, we are not testing the combination
where the origin is set and 'only-local' is '0'. So, there is some
value to having the test you are proposing but if Alvaro, you, or
others don't think it is worth adding a new test for it then we can
probably drop this one.

-- 
With Regards,
Amit Kapila.

  162. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-05T16:03:18Z

    On Tue, Jul 5, 2022 at 1:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 4, 2022 at 11:33 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, Jul 4, 2022 at 7:44 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > >
> > > > From d8f8844f877806527b6f3f45320b6ba55a8e3154 Mon Sep 17 00:00:00 2001
> > > > From: Vigneshwaran C <vignesh21@gmail.com>
> > > > Date: Thu, 26 May 2022 19:29:33 +0530
> > > > Subject: [PATCH v27 1/4] Add a missing test to verify only-local parameter in
> > > >  test_decoding plugin.
> > > >
> > > > Add a missing test to verify only-local parameter in test_decoding plugin.
> > >
> > > I don't get it.  replorigin.sql already has some lines to test
> > > local-only.  What is your patch adding that is new?  Maybe instead of
> > > adding some more lines at the end of the script, you should add lines
> > > where this stuff is already being tested.  But that assumes that there
> > > is something new that is being tested; if so what is it?
> >
> > The test is to check that remote origin data (i.e. replication origin
> > being set) will be filtered when only-local parameter is set. I felt
> > that this scenario is not covered, unless I'm missing something.
> >
>
> If we change only-local option to '0' in the below part of the test,
> we can see a different result:
> -- and magically the replayed xact will be filtered!
> SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local',
> '1');
>
> So, I think only-local functionality is being tested but I feel your
> test is clear in the sense that it clearly shows that remote data can
> only be fetched with 'only-local' as '0' when the origin is set. It
> seems to me that in existing tests, we are not testing the combination
> where the origin is set and 'only-local' is '0'. So, there is some
> value to having the test you are proposing but if Alvaro, you, or
> others don't think it is worth adding a new test for it then we can
> probably drop this one.

Since the existing test is already handling the verification of this
scenario, I felt no need to add the test. Updated v29 patch removes
the 0001 patch which had the test case.

Regards,
Vignesh

  163. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-05T16:06:17Z

    On Tue, Jul 5, 2022 at 6:14 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> I checked again the v26* patch set.
>
> I had no more comments for v26-0001, v26-0002, or v26-0004, but below
> are some comments for v26-0003.
>
> ========
> v26-0003
> ========
>
> 3.1 src/backend/catalog/pg_subscription.c
>
> 3.1.a
> +/*
> + * Get all relations for subscription that are in a ready state.
> + *
> + * Returned list is palloc'ed in current memory context.
> + */
> +List *
> +GetSubscriptionReadyRelations(Oid subid)
>
> "subscription" -> "the subscription"
>
> Also, It might be better to say in the function header what kind of
> structures are in the returned List. E.g. Firstly, I'd assumed it was
> the same return type as the other function
> GetSubscriptionReadyRelations, but it isn’t.

This function has been removed and GetSubscriptionRelations is used
with slight changes.

> 3.1.b
> I think there might be a case to be made for *combining* those
> SubscriptionRelState and SubscripotionRel structs into a single common
> struct. Then all those GetSubscriptionNotReadyRelations and
> GetSubscriptionNotReadyRelations (also GetSubscriptionRelations?) can
> be merged to be just one common function that takes a parameter to say
> do you want to return the relation List of ALL / READY /  NOT_READY?
> What do you think?

I have used a common function that can get all relations, ready
relations and not ready relations instead of the 3 functions.

> ======
>
> 3.2 src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> +/*
> + * Check and throw an error if the publisher has subscribed to the same table
> + * from some other publisher. This check is required only if copydata is ON and
> + * the origin is local.
> + */
> +static void
> +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
> +    CopyData copydata, char *origin, Oid subid)
>
> The function comment probably should say something about why the new
> READY logic was added in v26.

Added a comment for the same.

> ~~~
>
> 3.3
>
> 3.3.a
> + /*
> + * The subid will be valid only for ALTER SUBSCRIPTION ... REFRESH
> + * PUBLICATION. Get the ready relations for the subscription only in case
> + * of ALTER SUBSCRIPTION case as there will be no relations in ready state
> + * while the subscription is created.
> + */
> + if (subid != InvalidOid)
> + subreadyrels = GetSubscriptionReadyRelations(subid);
>
> The word "case" is 2x in the same sentence. I also paraphrased my
> understanding of this comment below. Maybe it is simpler?
>
> SUGGESTION
> Get the ready relations for the subscription. The subid will be valid
> only for ALTER SUBSCRIPTION ... REFRESH because there will be no
> relations in ready state while the subscription is created.

Modified

> 3.3.b
> + if (subid != InvalidOid)
> + subreadyrels = GetSubscriptionReadyRelations(subid);
>
> SUGGESTION
> if (OidIsValid(subid))

Modified

> ======
>
> 3.4 src/test/subscription/t/032_localonly.pl
>
> Now all the test cases are re-using the same data (e.g. 11,12,13) and
> you are deleting the data between the tests. I guess it is OK, but IMO
> the tests are easier to read when each test part was using unique data
> (e.g. 11,12,13, then 12,22,32, then 13,23,33 etc)

Modified

Thanks for the comments, the v29 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1T5utq60qVx%3DRN60rHcg7wt2psM7PpCQ2fDiB-R8oLGg%40mail.gmail.com

Regards,
 Vignesh

  164. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-06T11:39:30Z

    On Tue, Jul 5, 2022 at 9:33 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Since the existing test is already handling the verification of this
> scenario, I felt no need to add the test. Updated v29 patch removes
> the 0001 patch which had the test case.
>

I have again looked at the first and it looks good to me. I would like
to push it after some more review but before that, I would like to
know if someone else has any suggestions/objections to this patch, so
let me summarize the idea of the first patch. It will allow users to
skip the replication of remote data. Here remote data is the data that
the publisher node has received from some other node. The primary use
case is to avoid loops (infinite replication of the same data) among
nodes as shown in the initial email of this thread.

To achieve that this patch adds a new SUBSCRIPTION parameter "origin".
It specifies whether the subscription will request the publisher to
only send changes that originated locally or to send changes
regardless of origin. Setting it to "local" means that the
subscription will request the publisher to only send changes that
originated locally. Setting it to "any" means that the publisher sends
changes regardless of their origin. The default is "any".

Usage:
CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres port=9999'
PUBLICATION pub1 WITH (origin = local);

For now, even though the "origin" parameter allows only "local" and
"any" values, it is implemented as a string type so that the parameter
can be extended in future versions to support filtering using origin
names specified by the user.

This feature allows filtering only the replication data originated
from WAL but for initial sync (initial copy of table data) we don't
have such a facility as we can only distinguish the data based on
origin from WAL. As a separate patch (v29-0002*), we are planning to
forbid the initial sync if we notice that the publication tables were
also replicated from other publishers to avoid duplicate data or
loops. They will be allowed to copy with the 'force' option in such
cases.

-- 
With Regards,
Amit Kapila.

  165. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-08T11:08:57Z

    On Tue, Jul 5, 2022 at 9:33 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Since the existing test is already handling the verification of this
> scenario, I felt no need to add the test. Updated v29 patch removes
> the 0001 patch which had the test case.
>

I am not able to apply 0001.
patching file src/bin/psql/tab-complete.c
Hunk #1 FAILED at 1873.
Hunk #2 FAILED at 3152.

Few comments on 0002
=====================
1.
+          <xref linkend="sql-createsubscription-notes" /> for interaction

Is there a need for space before / in above? If not, please remove it
with similar extra space from other similar usages.

2.
+         <para>
+          There is some interaction between the <literal>origin</literal>
+          parameter and the <literal>copy_data</literal> parameter. Refer to
+          the <command>CREATE SUBSCRIPTION</command>
+          <xref linkend="sql-createsubscription-notes" /> for interaction
+          details and usage of <literal>force</literal> for
+          <literal>copy_data</literal> parameter.
          </para>

I think this is bit long. We can try to reduce this by something like:
Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
<literal>force</literal> option and its interaction with the
<literal>origin</literal> parameter.

Also, adopt the same other places if you agree with the above change.

4.
@@ -601,16 +549,28 @@ GetSubscriptionNotReadyRelations(Oid subid)
  SysScanDesc scan;

  rel = table_open(SubscriptionRelRelationId, AccessShareLock);
-
  ScanKeyInit(&skey[nkeys++],
  Anum_pg_subscription_rel_srsubid,

Spurious line removal.

5.
+static void
+check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
+    CopyData copydata, char *origin, Oid subid)
{
...
...
+ /*
+ * Get the ready relations for the subscription. The subid will be valid
+ * only for ALTER SUBSCRIPTION ... REFRESH because there will be no
+ * relations in ready state while the subscription is created.
+ */
+ if (OidIsValid(subid))
+ subreadyrels = GetSubscriptionRelations(subid, READY_STATE);

Why do we want to consider only READY_STATE relations here? If you see
its caller AlterSubscription_refresh(), we don't consider copying the
relation if it exists on subscribers in any state. If my observation
is correct then you probably don't need to introduce SubRelStateType.

-- 
With Regards,
Amit Kapila.

  166. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-09T14:40:56Z

    On Fri, Jul 8, 2022 at 4:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jul 5, 2022 at 9:33 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Since the existing test is already handling the verification of this
> > scenario, I felt no need to add the test. Updated v29 patch removes
> > the 0001 patch which had the test case.
> >
>
> I am not able to apply 0001.
> patching file src/bin/psql/tab-complete.c
> Hunk #1 FAILED at 1873.
> Hunk #2 FAILED at 3152.

I have rebased the patch on top of HEAD.

> Few comments on 0002
> =====================
> 1.
> +          <xref linkend="sql-createsubscription-notes" /> for interaction
>
> Is there a need for space before / in above? If not, please remove it
> with similar extra space from other similar usages.

Modified

> 2.
> +         <para>
> +          There is some interaction between the <literal>origin</literal>
> +          parameter and the <literal>copy_data</literal> parameter. Refer to
> +          the <command>CREATE SUBSCRIPTION</command>
> +          <xref linkend="sql-createsubscription-notes" /> for interaction
> +          details and usage of <literal>force</literal> for
> +          <literal>copy_data</literal> parameter.
>           </para>
>
> I think this is bit long. We can try to reduce this by something like:
> Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
> <literal>force</literal> option and its interaction with the
> <literal>origin</literal> parameter.
>
> Also, adopt the same other places if you agree with the above change.

Modified with slight rewording.

> 4.
> @@ -601,16 +549,28 @@ GetSubscriptionNotReadyRelations(Oid subid)
>   SysScanDesc scan;
>
>   rel = table_open(SubscriptionRelRelationId, AccessShareLock);
> -
>   ScanKeyInit(&skey[nkeys++],
>   Anum_pg_subscription_rel_srsubid,
>
> Spurious line removal.

I have removed this

> 5.
> +static void
> +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
> +    CopyData copydata, char *origin, Oid subid)
> {
> ...
> ...
> + /*
> + * Get the ready relations for the subscription. The subid will be valid
> + * only for ALTER SUBSCRIPTION ... REFRESH because there will be no
> + * relations in ready state while the subscription is created.
> + */
> + if (OidIsValid(subid))
> + subreadyrels = GetSubscriptionRelations(subid, READY_STATE);
>
> Why do we want to consider only READY_STATE relations here? If you see
> its caller AlterSubscription_refresh(), we don't consider copying the
> relation if it exists on subscribers in any state. If my observation
> is correct then you probably don't need to introduce SubRelStateType.

I have changed it similar to the caller function and removed SubRelStateType

Thanks for the comments, the v30 patch attached has the changes for the same.

Regards,
Vignesh

  167. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-11T03:41:26Z

    Here are my review comments for the v30* patches:

========
v30-0001
========

1.1 <general>

I was wondering if it is better to implement a new defGetOrigin method
now instead of just using the defGetString to process the 'origin',
since you may need to do that in future anyway if the 'origin' name is
planned to become specifiable by the user. OTOH maybe you prefer to
change this code later when the time comes. I am not sure what way is
best.

~~~

1.2. src/include/replication/walreceiver.h

@@ -183,6 +183,8 @@ typedef struct
  bool streaming; /* Streaming of large transactions */
  bool twophase; /* Streaming of two-phase transactions at
  * prepare time */
+ char    *origin; /* Only publish data originating from the
+ * specified origin */
  } logical;
  } proto;
 } WalRcvStreamOptions;

Should the new comments be aligned with the other ones?

========
v30-0002
========

2.1 doc/src/sgml/ref/alter_subscription.sgml

+         <para>
+          Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
+          <literal>force</literal> for <literal>copy_data</literal> parameter
+          and its interaction with the <literal>origin</literal> parameter.
          </para>

IMO it's better to say "Refer to the Notes" or "Refer to CREATE
SUBSCRIPTION Notes" instead of just "Refer Notes"

~~~

2.2 doc/src/sgml/ref/create_subscription.sgml

2.2.a
+         <para>
+          Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
+          <literal>force</literal> for <literal>copy_data</literal> parameter
+          and its interaction with the <literal>origin</literal> parameter.
+         </para>

IMO it's better to say "Refer to the Notes" (same as other xref on
this page) instead of "Refer Notes"

2.2.b
@@ -316,6 +324,11 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
           publisher sends changes regardless of their origin. The default is
           <literal>any</literal>.
          </para>
+         <para>
+          Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
+          <literal>force</literal> for <literal>copy_data</literal> parameter
+          and its interaction with the <literal>origin</literal> parameter.
+         </para>

Ditto

~~~

2.3 src/backend/commands/subscriptioncmds.c - DefGetCopyData

+/*
+ * Validate the value specified for copy_data parameter.
+ */
+static CopyData
+DefGetCopyData(DefElem *def)

~~~

2.4

+ /*
+ * If no parameter given, assume "true" is meant.
+ */

Please modify the comment to match the recent push [1].

~~~

2.5 src/test/subscription/t/032_localonly.pl

2.5.a
+# Check Alter subscription ... refresh publication when there is a new
+# table that is subscribing data from a different publication
+$node_A->safe_psql('postgres', "CREATE TABLE tab_full1 (a int PRIMARY KEY)");
+$node_B->safe_psql('postgres', "CREATE TABLE tab_full1 (a int PRIMARY KEY)");

Unfortunately, I think tab_full1 is a terrible table name because in
my screen font the 'l' and the '1' look exactly the same so it just
looks like a typo. Maybe change it to "tab_new" or something?

2.5b
What exactly is the purpose of "full" in all these test table names?
AFAIK "full" is just some name baggage inherited from completely
different tests which were doing full versus partial table
replication. I'm not sure it is relevant here.

========
v30-0002
========

No comments.

------
[1] https://github.com/postgres/postgres/commit/8445f5a21d40b969673ca03918c74b4fbc882bf4

Kind Regards,
Peter Smith.
Fujitsu Australia

  168. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-11T11:33:34Z

    On Sat, Jul 9, 2022 at 8:11 PM vignesh C <vignesh21@gmail.com> wrote:
>

Thanks, a few more comments on v30_0002*
1.
+/*
+ * Represents whether copy_data parameter is specified with off, on or force.

A comma is required after on.

2.
  qsort(subrel_local_oids, list_length(subrel_states),
    sizeof(Oid), oid_cmp);

+ check_pub_table_subscribed(wrconn, sub->publications, copy_data,
+    sub->origin, subrel_local_oids,
+    list_length(subrel_states));

We can avoid using list_length by using an additional variable in this case.

3.
errmsg("table: \"%s.%s\" might have replicated data in the publisher",
+    nspname, relname),

Why ':' is used after the table in the above message? I don't see such
a convention at other places in the code. Also, having might in the
error messages makes it less clear, so, can we slightly change the
message as in the attached patch?

4. I have made some additional changes in the comments, kindly check
the attached and merge those if you are okay.

5.
+$node_C->safe_psql(
+     'postgres', "
+        DELETE FROM tab_full");
+$node_B->safe_psql(
+     'postgres', "
+        DELETE FROM tab_full where a = 13");

Don't we need to wait for these operations to replicate?

-- 
With Regards,
Amit Kapila.

  169. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-12T03:12:55Z

    On Mon, Jul 11, 2022 at 9:11 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the v30* patches:
>
> ========
> v30-0001
> ========
>
> 1.1 <general>
>
> I was wondering if it is better to implement a new defGetOrigin method
> now instead of just using the defGetString to process the 'origin',
> since you may need to do that in future anyway if the 'origin' name is
> planned to become specifiable by the user. OTOH maybe you prefer to
> change this code later when the time comes. I am not sure what way is
> best.

I preferred to do that change when the feature is getting extended.

> ~~~
>
> 1.2. src/include/replication/walreceiver.h
>
> @@ -183,6 +183,8 @@ typedef struct
>   bool streaming; /* Streaming of large transactions */
>   bool twophase; /* Streaming of two-phase transactions at
>   * prepare time */
> + char    *origin; /* Only publish data originating from the
> + * specified origin */
>   } logical;
>   } proto;
>  } WalRcvStreamOptions;
>
> Should the new comments be aligned with the other ones?

I kept it like this as pgindent also is aligning it as the current code.

> ========
> v30-0002
> ========
>
> 2.1 doc/src/sgml/ref/alter_subscription.sgml
>
> +         <para>
> +          Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
> +          <literal>force</literal> for <literal>copy_data</literal> parameter
> +          and its interaction with the <literal>origin</literal> parameter.
>           </para>
>
> IMO it's better to say "Refer to the Notes" or "Refer to CREATE
> SUBSCRIPTION Notes" instead of just "Refer Notes"

Modified to "Refer to the Notes"

> ~~~
>
> 2.2 doc/src/sgml/ref/create_subscription.sgml
>
> 2.2.a
> +         <para>
> +          Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
> +          <literal>force</literal> for <literal>copy_data</literal> parameter
> +          and its interaction with the <literal>origin</literal> parameter.
> +         </para>
>
> IMO it's better to say "Refer to the Notes" (same as other xref on
> this page) instead of "Refer Notes"

Modified to "Refer to the Notes"

> 2.2.b
> @@ -316,6 +324,11 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>            publisher sends changes regardless of their origin. The default is
>            <literal>any</literal>.
>           </para>
> +         <para>
> +          Refer <xref linkend="sql-createsubscription-notes"/> for the usage of
> +          <literal>force</literal> for <literal>copy_data</literal> parameter
> +          and its interaction with the <literal>origin</literal> parameter.
> +         </para>
>
> Ditto

Modified to "Refer to the Notes"

> ~~~
>
> 2.3 src/backend/commands/subscriptioncmds.c - DefGetCopyData
>
> +/*
> + * Validate the value specified for copy_data parameter.
> + */
> +static CopyData
> +DefGetCopyData(DefElem *def)

Changed it to defGetCopyData to keep the naming similar to others

> ~~~
>
> 2.4
>
> + /*
> + * If no parameter given, assume "true" is meant.
> + */
>
> Please modify the comment to match the recent push [1].

Modified

> ~~~
>
> 2.5 src/test/subscription/t/032_localonly.pl
>
> 2.5.a
> +# Check Alter subscription ... refresh publication when there is a new
> +# table that is subscribing data from a different publication
> +$node_A->safe_psql('postgres', "CREATE TABLE tab_full1 (a int PRIMARY KEY)");
> +$node_B->safe_psql('postgres', "CREATE TABLE tab_full1 (a int PRIMARY KEY)");
>
> Unfortunately, I think tab_full1 is a terrible table name because in
> my screen font the 'l' and the '1' look exactly the same so it just
> looks like a typo. Maybe change it to "tab_new" or something?

Modified to tab_new

> 2.5b
> What exactly is the purpose of "full" in all these test table names?
> AFAIK "full" is just some name baggage inherited from completely
> different tests which were doing full versus partial table
> replication. I'm not sure it is relevant here.

Modified to tab

Thanks for the comments, the v31 patch attached has the changes for the same.

Regards,
Vignesh

  170. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-12T03:17:06Z

    On Mon, Jul 11, 2022 at 5:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sat, Jul 9, 2022 at 8:11 PM vignesh C <vignesh21@gmail.com> wrote:
> >
>
> Thanks, a few more comments on v30_0002*
> 1.
> +/*
> + * Represents whether copy_data parameter is specified with off, on or force.
>
> A comma is required after on.

Modified

> 2.
>   qsort(subrel_local_oids, list_length(subrel_states),
>     sizeof(Oid), oid_cmp);
>
> + check_pub_table_subscribed(wrconn, sub->publications, copy_data,
> +    sub->origin, subrel_local_oids,
> +    list_length(subrel_states));
>
> We can avoid using list_length by using an additional variable in this case.

Modified

> 3.
> errmsg("table: \"%s.%s\" might have replicated data in the publisher",
> +    nspname, relname),
>
> Why ':' is used after the table in the above message? I don't see such
> a convention at other places in the code. Also, having might in the
> error messages makes it less clear, so, can we slightly change the
> message as in the attached patch?

Modified as suggested

> 4. I have made some additional changes in the comments, kindly check
> the attached and merge those if you are okay.

I have merged the changes

> 5.
> +$node_C->safe_psql(
> +     'postgres', "
> +        DELETE FROM tab_full");
> +$node_B->safe_psql(
> +     'postgres', "
> +        DELETE FROM tab_full where a = 13");
>
> Don't we need to wait for these operations to replicate?

Modified to include wait

Thanks for the comments, the v31 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2-D860yULtcmZAzDbdiof-Dg6Y_YaY4owbO6Rj%3DXEHMw%40mail.gmail.com

Regards,
Vignesh

  171. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-12T04:21:31Z

    On Tue, Jul 12, 2022 at 8:43 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Jul 11, 2022 at 9:11 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my review comments for the v30* patches:
> >
> > ========
> > v30-0001
> > ========
> >
> > 1.1 <general>
> >
> > I was wondering if it is better to implement a new defGetOrigin method
> > now instead of just using the defGetString to process the 'origin',
> > since you may need to do that in future anyway if the 'origin' name is
> > planned to become specifiable by the user. OTOH maybe you prefer to
> > change this code later when the time comes. I am not sure what way is
> > best.
>
> I preferred to do that change when the feature is getting extended.
>

+1.

*
+$node_C->safe_psql(
+     'postgres', "
+        DELETE FROM tab");
+$node_B->safe_psql(
+     'postgres', "
+        DELETE FROM tab where a = 32");
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);

Here, don't we need to use node_C instead of node_A for waiting as we
have performed an operation on node_C?

-- 
With Regards,
Amit Kapila.

  172. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-12T09:28:04Z

    On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jul 12, 2022 at 8:43 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, Jul 11, 2022 at 9:11 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Here are my review comments for the v30* patches:
> > >
> > > ========
> > > v30-0001
> > > ========
> > >
> > > 1.1 <general>
> > >
> > > I was wondering if it is better to implement a new defGetOrigin method
> > > now instead of just using the defGetString to process the 'origin',
> > > since you may need to do that in future anyway if the 'origin' name is
> > > planned to become specifiable by the user. OTOH maybe you prefer to
> > > change this code later when the time comes. I am not sure what way is
> > > best.
> >
> > I preferred to do that change when the feature is getting extended.
> >
>
> +1.
>
> *
> +$node_C->safe_psql(
> +     'postgres', "
> +        DELETE FROM tab");
> +$node_B->safe_psql(
> +     'postgres', "
> +        DELETE FROM tab where a = 32");
> +
> +$node_A->wait_for_catchup($subname_BA);
> +$node_B->wait_for_catchup($subname_AB);
>
> Here, don't we need to use node_C instead of node_A for waiting as we
> have performed an operation on node_C?

No need to wait for node_C as we have dropped the subscription before
the delete operation. I have added the comment to make it clear.
The attached v32 patch has the changes for the same.

Regards,
Vignesh

  173. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-07-13T11:19:56Z

    On Tue, Jul 12, 2022 at 2:58 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

I find one thing  confusing about this patch.  Basically, this has two
option 'local' and 'any', so I would assume that all the local server
changes should be covered under the 'local' but now if we set some
origin using 'select pg_replication_origin_session_setup('aa');' then
changes from that session will be ignored because it has an origin id.
I think actually the name is creating confusion, because by local it
seems like a change which originated locally and the document is also
specifying the same.

+       If <literal>local</literal>, the subscription will request the publisher
+       to only send changes that originated locally. If <literal>any</literal>,

I think if we want to keep the option local then we should look up all
the origin in the replication origin catalog and identify whether it
is a local origin id or remote origin id and based on that filter out
the changes.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  174. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-07-14T05:56:32Z

    On Wed, Jul 13, 2022 at 4:49 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jul 12, 2022 at 2:58 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> I find one thing  confusing about this patch.  Basically, this has two
> option 'local' and 'any', so I would assume that all the local server
> changes should be covered under the 'local' but now if we set some
> origin using 'select pg_replication_origin_session_setup('aa');' then
> changes from that session will be ignored because it has an origin id.
> I think actually the name is creating confusion, because by local it
> seems like a change which originated locally and the document is also
> specifying the same.
>
> +       If <literal>local</literal>, the subscription will request the publisher
> +       to only send changes that originated locally. If <literal>any</literal>,
>
> I think if we want to keep the option local then we should look up all
> the origin in the replication origin catalog and identify whether it
> is a local origin id or remote origin id and based on that filter out
> the changes.

On the other hand if we are interested in receiving the changes which
are generated without any origin then I think we should change 'local'
to 'none' and then in future we can provide a new option which can
send the changes generated by all the local origin?  I think other
than this the patch LGTM.


--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  175. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-14T13:04:02Z

    On Thu, Jul 14, 2022 at 11:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Jul 13, 2022 at 4:49 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jul 12, 2022 at 2:58 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > I find one thing  confusing about this patch.  Basically, this has two
> > option 'local' and 'any', so I would assume that all the local server
> > changes should be covered under the 'local' but now if we set some
> > origin using 'select pg_replication_origin_session_setup('aa');' then
> > changes from that session will be ignored because it has an origin id.
> > I think actually the name is creating confusion, because by local it
> > seems like a change which originated locally and the document is also
> > specifying the same.
> >
> > +       If <literal>local</literal>, the subscription will request the publisher
> > +       to only send changes that originated locally. If <literal>any</literal>,
> >
> > I think if we want to keep the option local then we should look up all
> > the origin in the replication origin catalog and identify whether it
> > is a local origin id or remote origin id and based on that filter out
> > the changes.
>
> On the other hand if we are interested in receiving the changes which
> are generated without any origin then I think we should change 'local'
> to 'none' and then in future we can provide a new option which can
> send the changes generated by all the local origin?  I think other
> than this the patch LGTM.

Thanks for the comment. The attached v33 patch has the changes to
specify origin as 'none' instead of 'local' which will not publish the
data having any origin.

Regards,
Vignesh

  176. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-07-14T13:12:11Z

    On Thu, 14 Jul 2022 at 6:34 PM, vignesh C <vignesh21@gmail.com> wrote:

> On Thu, Jul 14, 2022 at 11:26 AM Dilip Kumar <dilipbalaut@gmail.com>
> wrote:
> >
> > On Wed, Jul 13, 2022 at 4:49 PM Dilip Kumar <dilipbalaut@gmail.com>
> wrote:
> > >
> > > On Tue, Jul 12, 2022 at 2:58 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > >
> > > I find one thing  confusing about this patch.  Basically, this has two
> > > option 'local' and 'any', so I would assume that all the local server
> > > changes should be covered under the 'local' but now if we set some
> > > origin using 'select pg_replication_origin_session_setup('aa');' then
> > > changes from that session will be ignored because it has an origin id.
> > > I think actually the name is creating confusion, because by local it
> > > seems like a change which originated locally and the document is also
> > > specifying the same.
> > >
> > > +       If <literal>local</literal>, the subscription will request the
> publisher
> > > +       to only send changes that originated locally. If
> <literal>any</literal>,
> > >
> > > I think if we want to keep the option local then we should look up all
> > > the origin in the replication origin catalog and identify whether it
> > > is a local origin id or remote origin id and based on that filter out
> > > the changes.
> >
> > On the other hand if we are interested in receiving the changes which
> > are generated without any origin then I think we should change 'local'
> > to 'none' and then in future we can provide a new option which can
> > send the changes generated by all the local origin?  I think other
> > than this the patch LGTM.
>
> Thanks for the comment. The attached v33 patch has the changes to
> specify origin as 'none' instead of 'local' which will not publish the
> data having any origin.


I think the ‘none’ might have problem from expand ability pov?  what if in
future we support the actual origin name and than what none mean? no origin
or origin name none?  Should we just give origin name empty name ‘’?  Or is
there some other issue?

>
—
Dilip

> --
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  177. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-16T03:34:54Z

    On Thu, Jul 14, 2022 at 6:42 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, 14 Jul 2022 at 6:34 PM, vignesh C <vignesh21@gmail.com> wrote:
>>
>> On Thu, Jul 14, 2022 at 11:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> >
>> > On Wed, Jul 13, 2022 at 4:49 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> > >
>> > > On Tue, Jul 12, 2022 at 2:58 PM vignesh C <vignesh21@gmail.com> wrote:
>> > > >
>> > > > On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> > >
>> > > I find one thing  confusing about this patch.  Basically, this has two
>> > > option 'local' and 'any', so I would assume that all the local server
>> > > changes should be covered under the 'local' but now if we set some
>> > > origin using 'select pg_replication_origin_session_setup('aa');' then
>> > > changes from that session will be ignored because it has an origin id.
>> > > I think actually the name is creating confusion, because by local it
>> > > seems like a change which originated locally and the document is also
>> > > specifying the same.
>> > >
>> > > +       If <literal>local</literal>, the subscription will request the publisher
>> > > +       to only send changes that originated locally. If <literal>any</literal>,
>> > >
>> > > I think if we want to keep the option local then we should look up all
>> > > the origin in the replication origin catalog and identify whether it
>> > > is a local origin id or remote origin id and based on that filter out
>> > > the changes.
>> >
>> > On the other hand if we are interested in receiving the changes which
>> > are generated without any origin then I think we should change 'local'
>> > to 'none' and then in future we can provide a new option which can
>> > send the changes generated by all the local origin?  I think other
>> > than this the patch LGTM.
>>
>> Thanks for the comment. The attached v33 patch has the changes to
>> specify origin as 'none' instead of 'local' which will not publish the
>> data having any origin.
>
>
> I think the ‘none’ might have problem from expand ability pov?  what if in future we support the actual origin name and than what none mean? no origin or origin name none?  Should we just give origin name empty name ‘’?  Or is there some other issue?

Currently there is no restriction in the name we can specify for
origin, ex any, none, local, etc all are allowed as origin name. How
about extending it with another parameter "origin_name" when we
support filtering of a particular origin like:
1) origin = name, origin_name = 'orig1' -- Means that the publisher
will filter the changes having origin name as 'orig1' and send the
other changes.
2) origin = any -- Means that the publisher sends all changes
regardless of their origin.
3) origin = none -- Means that the subscription will request the
publisher to only send changes that have no origin associated.
When we need to specify filtering of a particular origin name we will
have to use both origin and origin_name option, like origin = name,
origin_name = 'orig1' as in the first example.

I'm not sure if there is a simpler way to do this with only a single
option as both 'none' and 'any' can be specified as the origin name.
Thoughts?

Regards,
Vignesh

  178. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-07-16T04:59:11Z

    On Sat, Jul 16, 2022 at 9:05 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, Jul 14, 2022 at 6:42 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, 14 Jul 2022 at 6:34 PM, vignesh C <vignesh21@gmail.com> wrote:
> >>
> >> On Thu, Jul 14, 2022 at 11:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >> >
> >> > On Wed, Jul 13, 2022 at 4:49 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >> > >
> >> > > On Tue, Jul 12, 2022 at 2:58 PM vignesh C <vignesh21@gmail.com> wrote:
> >> > > >
> >> > > > On Tue, Jul 12, 2022 at 9:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >> > >
> >> > > I find one thing  confusing about this patch.  Basically, this has two
> >> > > option 'local' and 'any', so I would assume that all the local server
> >> > > changes should be covered under the 'local' but now if we set some
> >> > > origin using 'select pg_replication_origin_session_setup('aa');' then
> >> > > changes from that session will be ignored because it has an origin id.
> >> > > I think actually the name is creating confusion, because by local it
> >> > > seems like a change which originated locally and the document is also
> >> > > specifying the same.
> >> > >
> >> > > +       If <literal>local</literal>, the subscription will request the publisher
> >> > > +       to only send changes that originated locally. If <literal>any</literal>,
> >> > >
> >> > > I think if we want to keep the option local then we should look up all
> >> > > the origin in the replication origin catalog and identify whether it
> >> > > is a local origin id or remote origin id and based on that filter out
> >> > > the changes.
> >> >
> >> > On the other hand if we are interested in receiving the changes which
> >> > are generated without any origin then I think we should change 'local'
> >> > to 'none' and then in future we can provide a new option which can
> >> > send the changes generated by all the local origin?  I think other
> >> > than this the patch LGTM.
> >>
> >> Thanks for the comment. The attached v33 patch has the changes to
> >> specify origin as 'none' instead of 'local' which will not publish the
> >> data having any origin.
> >
> >
> > I think the ‘none’ might have problem from expand ability pov?  what if in future we support the actual origin name and than what none mean? no origin or origin name none?  Should we just give origin name empty name ‘’?  Or is there some other issue?
>
> Currently there is no restriction in the name we can specify for
> origin, ex any, none, local, etc all are allowed as origin name. How
> about extending it with another parameter "origin_name" when we
> support filtering of a particular origin like:
> 1) origin = name, origin_name = 'orig1' -- Means that the publisher
> will filter the changes having origin name as 'orig1' and send the
> other changes.
> 2) origin = any -- Means that the publisher sends all changes
> regardless of their origin.
> 3) origin = none -- Means that the subscription will request the
> publisher to only send changes that have no origin associated.
> When we need to specify filtering of a particular origin name we will
> have to use both origin and origin_name option, like origin = name,
> origin_name = 'orig1' as in the first example.
>
> I'm not sure if there is a simpler way to do this with only a single
> option as both 'none' and 'any' can be specified as the origin name.
> Thoughts?

I think giving two options would be really confusing from the
usability perspective.  I think what we should be doing here is to
keep these three names 'none', 'any' and 'local' as reserved names for
the origin name so that those are not allowed to be set by the user
and they have some internal meaning.  And I don't think this is going
to create too much trouble for anyone because those are not really the
names someone wants to use for their replication origin.  So I think
pg_replication_origin_create() we can also check for the reserved
names just for the replication origin.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  179. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-18T04:52:50Z

    On Sat, Jul 16, 2022 at 10:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I think giving two options would be really confusing from the
> usability perspective.  I think what we should be doing here is to
> keep these three names 'none', 'any' and 'local' as reserved names for
> the origin name so that those are not allowed to be set by the user
> and they have some internal meaning.
>

This makes sense to me. I think we can avoid reserving 'local' for now
till we agree on its use case and implementation. One similar point
about slots is that we treat 'none' slot_name in subscription commands
as a special value indicating no slot name whereas we do allow
creating a slot with the name 'none' with
pg_create_logical_replication_slot(). So, if we want to follow a
similar convention here, we may not need to add any restriction for
origin names but personally, I think it is better to add such a
restriction to avoid confusion and in fact, as a separate patch we
should even disallow creating slot name as 'none'.

-- 
With Regards,
Amit Kapila.

  180. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-18T11:28:32Z

    On Mon, Jul 18, 2022 at 10:23 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sat, Jul 16, 2022 at 10:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I think giving two options would be really confusing from the
> > usability perspective.  I think what we should be doing here is to
> > keep these three names 'none', 'any' and 'local' as reserved names for
> > the origin name so that those are not allowed to be set by the user
> > and they have some internal meaning.
> >
>
> This makes sense to me. I think we can avoid reserving 'local' for now
> till we agree on its use case and implementation. One similar point
> about slots is that we treat 'none' slot_name in subscription commands
> as a special value indicating no slot name whereas we do allow
> creating a slot with the name 'none' with
> pg_create_logical_replication_slot(). So, if we want to follow a
> similar convention here, we may not need to add any restriction for
> origin names but personally, I think it is better to add such a
> restriction to avoid confusion and in fact, as a separate patch we
> should even disallow creating slot name as 'none'.

I have made changes to disallow the name "any" and "none" in
pg_replication_origin_create. The attached v34 patch has the changes
for the same. I will post a separate patch to disallow creation of
slots with names as 'none' separately later.

Regards,
Vignesh

  181. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-18T16:16:37Z

    On Mon, Jul 18, 2022 at 4:58 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Jul 18, 2022 at 10:23 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Sat, Jul 16, 2022 at 10:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I think giving two options would be really confusing from the
> > > usability perspective.  I think what we should be doing here is to
> > > keep these three names 'none', 'any' and 'local' as reserved names for
> > > the origin name so that those are not allowed to be set by the user
> > > and they have some internal meaning.
> > >
> >
> > This makes sense to me. I think we can avoid reserving 'local' for now
> > till we agree on its use case and implementation. One similar point
> > about slots is that we treat 'none' slot_name in subscription commands
> > as a special value indicating no slot name whereas we do allow
> > creating a slot with the name 'none' with
> > pg_create_logical_replication_slot(). So, if we want to follow a
> > similar convention here, we may not need to add any restriction for
> > origin names but personally, I think it is better to add such a
> > restriction to avoid confusion and in fact, as a separate patch we
> > should even disallow creating slot name as 'none'.
>
> I have made changes to disallow the name "any" and "none" in
> pg_replication_origin_create. The attached v34 patch has the changes
> for the same. I will post a separate patch to disallow creation of
> slots with names as 'none' separately later.

I have updated the patch to handle the origin value case
insensitively. The attached patch has the changes for the same.

Regards,
Vignesh

  182. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-19T11:34:37Z

    On Mon, Jul 18, 2022 at 9:46 PM vignesh C <vignesh21@gmail.com> wrote:
>
> I have updated the patch to handle the origin value case
> insensitively. The attached patch has the changes for the same.
>

Thanks, the patch looks mostly good to me. I have made a few changes
in 0001 patch which are as follows: (a) make a comparison of origin
names in maybe_reread_subscription similar to slot names as in future
we may support origin names other than 'any' and 'none', (b) made
comment changes at few places and minor change in one of the error
message, (c) ran pgindent and slightly changed the commit message.

I am planning to push this day after tomorrow unless there are any
comments/suggestions.

-- 
With Regards,
Amit Kapila.

  183. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-20T05:08:36Z

    On Tue, Jul 19, 2022 at 11:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 18, 2022 at 9:46 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > I have updated the patch to handle the origin value case
> > insensitively. The attached patch has the changes for the same.
> >
>
> Thanks, the patch looks mostly good to me. I have made a few changes
> in 0001 patch which are as follows: (a) make a comparison of origin
> names in maybe_reread_subscription similar to slot names as in future
> we may support origin names other than 'any' and 'none', (b) made
> comment changes at few places and minor change in one of the error
> message, (c) ran pgindent and slightly changed the commit message.
>
> I am planning to push this day after tomorrow unless there are any
> comments/suggestions.

FYI, the function name in the comment is not same as the function name here:

+/*
+ * IsReservedName
+ * True iff name is either "none" or "any".
+ */
+static bool
+IsReservedOriginName(const char *name)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  184. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-20T09:03:16Z

    On Wed, Jul 20, 2022 at 10:38 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jul 19, 2022 at 11:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Jul 18, 2022 at 9:46 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > I have updated the patch to handle the origin value case
> > > insensitively. The attached patch has the changes for the same.
> > >
> >
> > Thanks, the patch looks mostly good to me. I have made a few changes
> > in 0001 patch which are as follows: (a) make a comparison of origin
> > names in maybe_reread_subscription similar to slot names as in future
> > we may support origin names other than 'any' and 'none', (b) made
> > comment changes at few places and minor change in one of the error
> > message, (c) ran pgindent and slightly changed the commit message.
> >
> > I am planning to push this day after tomorrow unless there are any
> > comments/suggestions.
>
> FYI, the function name in the comment is not same as the function name here:
>
> +/*
> + * IsReservedName
> + * True iff name is either "none" or "any".
> + */
> +static bool
> +IsReservedOriginName(const char *name)

Modified. Apart from this I have run pgperltidy on the perl file and
renamed 032_origin.pl to 030_origin.pl as currently there is
029_on_error.pl, 031_column_list.pl and there is no 030_*****.pl file.
Thanks for the comment, the attached patch has the changes for the same.

Regards,
Vignesh

  185. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-21T08:36:31Z

    On Wed, Jul 20, 2022 at 2:33 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Modified. Apart from this I have run pgperltidy on the perl file and
> renamed 032_origin.pl to 030_origin.pl as currently there is
> 029_on_error.pl, 031_column_list.pl and there is no 030_*****.pl file.
> Thanks for the comment, the attached patch has the changes for the same.
>

Pushed. Kindly rebase the remaining patches.

-- 
With Regards,
Amit Kapila.

  186. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-21T10:34:22Z

    On Thu, Jul 21, 2022 at 2:06 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jul 20, 2022 at 2:33 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Modified. Apart from this I have run pgperltidy on the perl file and
> > renamed 032_origin.pl to 030_origin.pl as currently there is
> > 029_on_error.pl, 031_column_list.pl and there is no 030_*****.pl file.
> > Thanks for the comment, the attached patch has the changes for the same.
> >
>
> Pushed. Kindly rebase the remaining patches.

Thanks for pushing the patch.
The attached v37 version contains the rebased patch for the remaining patches.

Regards,
Vignesh

  187. Re: Handle infinite recursion in logical replication setup

    Jonathan S. Katz <jkatz@postgresql.org> — 2022-07-21T20:08:54Z

    Hi,

On 7/21/22 6:34 AM, vignesh C wrote:
> On Thu, Jul 21, 2022 at 2:06 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Wed, Jul 20, 2022 at 2:33 PM vignesh C <vignesh21@gmail.com> wrote:
>>>
>>> Modified. Apart from this I have run pgperltidy on the perl file and
>>> renamed 032_origin.pl to 030_origin.pl as currently there is
>>> 029_on_error.pl, 031_column_list.pl and there is no 030_*****.pl file.
>>> Thanks for the comment, the attached patch has the changes for the same.
>>>
>>
>> Pushed. Kindly rebase the remaining patches.
> 
> Thanks for pushing the patch.
> The attached v37 version contains the rebased patch for the remaining patches.

Thanks for the work on this feature -- this is definitely very helpful 
towards supporting more types of use cases with logical replication!

I've read through the proposed documentation and did some light testing 
of the patch. I have two general comments about the docs as they 
currently read:

1. I'm concerned by calling this "Bidirectional replication" in the docs 
that we are overstating the current capabilities. I think this is 
accentuated int he opening paragraph:

==snip==
  Bidirectional replication is useful for creating a multi-master database
  environment for replicating read/write operations performed by any of the
  member nodes.
==snip==

For one, we're not replicating reads, we're replicating writes. Amongst 
the writes, at this point we're only replicating DML. A reader could 
think that deploying can work for a full bidirectional solution.

(Even if we're aspirationally calling this section "Bidirectional 
replication", that does make it sound like we're limited to two nodes, 
when we can support more than two).

Perhaps "Logical replication between writers" or "Logical replication 
between primaries" or "Replicating changes between primaries", or 
something better.

2. There is no mention of conflicts in the documentation, e.g. 
referencing the "Conflicts" section of the documentation. It's very easy 
to create a conflicting transaction that causes a subscriber to be 
unable to continue to apply transactions:

   -- DB 1
   CREATE TABLE abc (id int);
   CREATE PUBLICATION node1 FOR ALL TABLES ;

   -- DB2
   CREATE TABLE abc (id int);
   CREATE PUBLICATION node2 FOR ALL TABLES ;
   CREATE SUBSCRIPTION node2_node1
     CONNECTION 'dbname=logi port=5433'
     PUBLICATION node1
     WITH (copy_data = off, origin = none);

   -- DB1
   CREATE SUBSCRIPTION node1_node2
     CONNECTION 'dbname=logi port=5434'
     PUBLICATION node2
     WITH (copy_data = off, origin = none);
   INSERT INTO abc VALUES (1);

   -- DB2
   INSERT INTO abc VALUES (2);

   -- DB1
   ALTER TABLE abc ADD PRIMARY KEY id;
   INSERT INTO abc VALUES (3);

   -- DB2
   INSERT INTO abc VALUES (3);

   -- DB1 cannot apply the transactions

At a minimum, I think we should reference the documentation we have in 
the logical replication section on conflicts. We may also want to advise 
that a user is responsible for designing their schemas in a way to 
minimize the risk of conflicts.

Thanks,

Jonathan

  188. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-22T04:47:01Z

    On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> Thanks for the work on this feature -- this is definitely very helpful
> towards supporting more types of use cases with logical replication!
>
> I've read through the proposed documentation and did some light testing
> of the patch. I have two general comments about the docs as they
> currently read:
>
> 1. I'm concerned by calling this "Bidirectional replication" in the docs
> that we are overstating the current capabilities. I think this is
> accentuated int he opening paragraph:
>
> ==snip==
>   Bidirectional replication is useful for creating a multi-master database
>   environment for replicating read/write operations performed by any of the
>   member nodes.
> ==snip==
>
> For one, we're not replicating reads, we're replicating writes. Amongst
> the writes, at this point we're only replicating DML. A reader could
> think that deploying can work for a full bidirectional solution.
>
> (Even if we're aspirationally calling this section "Bidirectional
> replication", that does make it sound like we're limited to two nodes,
> when we can support more than two).
>

Right, I think the system can support N-Way replication.

> Perhaps "Logical replication between writers" or "Logical replication
> between primaries" or "Replicating changes between primaries", or
> something better.
>

Among the above "Replicating changes between primaries" sounds good to
me or simply "Replication between primaries". As this is a sub-section
on the Logical Replication page, I feel it is okay to not use Logical
in the title.

> 2. There is no mention of conflicts in the documentation, e.g.
> referencing the "Conflicts" section of the documentation. It's very easy
> to create a conflicting transaction that causes a subscriber to be
> unable to continue to apply transactions:
>
>    -- DB 1
>    CREATE TABLE abc (id int);
>    CREATE PUBLICATION node1 FOR ALL TABLES ;
>
>    -- DB2
>    CREATE TABLE abc (id int);
>    CREATE PUBLICATION node2 FOR ALL TABLES ;
>    CREATE SUBSCRIPTION node2_node1
>      CONNECTION 'dbname=logi port=5433'
>      PUBLICATION node1
>      WITH (copy_data = off, origin = none);
>
>    -- DB1
>    CREATE SUBSCRIPTION node1_node2
>      CONNECTION 'dbname=logi port=5434'
>      PUBLICATION node2
>      WITH (copy_data = off, origin = none);
>    INSERT INTO abc VALUES (1);
>
>    -- DB2
>    INSERT INTO abc VALUES (2);
>
>    -- DB1
>    ALTER TABLE abc ADD PRIMARY KEY id;
>    INSERT INTO abc VALUES (3);
>
>    -- DB2
>    INSERT INTO abc VALUES (3);
>
>    -- DB1 cannot apply the transactions
>
> At a minimum, I think we should reference the documentation we have in
> the logical replication section on conflicts. We may also want to advise
> that a user is responsible for designing their schemas in a way to
> minimize the risk of conflicts.
>

This sounds reasonable to me.

One more point about docs, it appears to be added as the last
sub-section on the Logical Replication page. Is there a reason for
doing so? I feel this should be third sub-section after describing
Publication and Subscription.

BTW, do you have any opinion on the idea of the first remaining patch
where we accomplish two things: a) Checks and throws an error if
'copy_data = on' and 'origin = none' but the publication tables were
also replicated from other publishers. b) Adds 'force' value for
copy_data parameter to allow copying in such a case. The primary
reason for this patch is to avoid loops or duplicate data in the
initial phase. We can't skip copying based on origin as we can do
while replicating changes from WAL. So, we detect that the publisher
already has data from some other node and doesn't allow replication
unless the user uses the 'force' option for copy_data.

-- 
With Regards,
Amit Kapila.

  189. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-23T17:27:56Z

    On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> Hi,
>
> On 7/21/22 6:34 AM, vignesh C wrote:
> > On Thu, Jul 21, 2022 at 2:06 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Wed, Jul 20, 2022 at 2:33 PM vignesh C <vignesh21@gmail.com> wrote:
> >>>
> >>> Modified. Apart from this I have run pgperltidy on the perl file and
> >>> renamed 032_origin.pl to 030_origin.pl as currently there is
> >>> 029_on_error.pl, 031_column_list.pl and there is no 030_*****.pl file.
> >>> Thanks for the comment, the attached patch has the changes for the same.
> >>>
> >>
> >> Pushed. Kindly rebase the remaining patches.
> >
> > Thanks for pushing the patch.
> > The attached v37 version contains the rebased patch for the remaining patches.
>
> Thanks for the work on this feature -- this is definitely very helpful
> towards supporting more types of use cases with logical replication!
>
> I've read through the proposed documentation and did some light testing
> of the patch. I have two general comments about the docs as they
> currently read:
>
> 1. I'm concerned by calling this "Bidirectional replication" in the docs
> that we are overstating the current capabilities. I think this is
> accentuated int he opening paragraph:
>
> ==snip==
>   Bidirectional replication is useful for creating a multi-master database
>   environment for replicating read/write operations performed by any of the
>   member nodes.
> ==snip==
>
> For one, we're not replicating reads, we're replicating writes. Amongst
> the writes, at this point we're only replicating DML. A reader could
> think that deploying can work for a full bidirectional solution.

I have changed read/write operations to write operations. I have also
added a note saying "The logical replication restrictions applies to
the replication between primaries also.", to clarify that non DML
operations and other restrictions apply in this case too.

> (Even if we're aspirationally calling this section "Bidirectional
> replication", that does make it sound like we're limited to two nodes,
> when we can support more than two).
>
> Perhaps "Logical replication between writers" or "Logical replication
> between primaries" or "Replicating changes between primaries", or
> something better.

I have changed it to "Replication between primaries".

> 2. There is no mention of conflicts in the documentation, e.g.
> referencing the "Conflicts" section of the documentation. It's very easy
> to create a conflicting transaction that causes a subscriber to be
> unable to continue to apply transactions:
>
>    -- DB 1
>    CREATE TABLE abc (id int);
>    CREATE PUBLICATION node1 FOR ALL TABLES ;
>
>    -- DB2
>    CREATE TABLE abc (id int);
>    CREATE PUBLICATION node2 FOR ALL TABLES ;
>    CREATE SUBSCRIPTION node2_node1
>      CONNECTION 'dbname=logi port=5433'
>      PUBLICATION node1
>      WITH (copy_data = off, origin = none);
>
>    -- DB1
>    CREATE SUBSCRIPTION node1_node2
>      CONNECTION 'dbname=logi port=5434'
>      PUBLICATION node2
>      WITH (copy_data = off, origin = none);
>    INSERT INTO abc VALUES (1);
>
>    -- DB2
>    INSERT INTO abc VALUES (2);
>
>    -- DB1
>    ALTER TABLE abc ADD PRIMARY KEY id;
>    INSERT INTO abc VALUES (3);
>
>    -- DB2
>    INSERT INTO abc VALUES (3);
>
>    -- DB1 cannot apply the transactions
>
> At a minimum, I think we should reference the documentation we have in
> the logical replication section on conflicts. We may also want to advise
> that a user is responsible for designing their schemas in a way to
> minimize the risk of conflicts.

Added a  note for the same and referred it to the conflicts section.

Thanks for the comments, the attached v38 patch has the changes for the same.

Regards,
Vignesh

  190. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-23T17:32:31Z

    On Fri, Jul 22, 2022 at 10:17 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> >
> > Thanks for the work on this feature -- this is definitely very helpful
> > towards supporting more types of use cases with logical replication!
> >
> > I've read through the proposed documentation and did some light testing
> > of the patch. I have two general comments about the docs as they
> > currently read:
> >
> > 1. I'm concerned by calling this "Bidirectional replication" in the docs
> > that we are overstating the current capabilities. I think this is
> > accentuated int he opening paragraph:
> >
> > ==snip==
> >   Bidirectional replication is useful for creating a multi-master database
> >   environment for replicating read/write operations performed by any of the
> >   member nodes.
> > ==snip==
> >
> > For one, we're not replicating reads, we're replicating writes. Amongst
> > the writes, at this point we're only replicating DML. A reader could
> > think that deploying can work for a full bidirectional solution.
> >
> > (Even if we're aspirationally calling this section "Bidirectional
> > replication", that does make it sound like we're limited to two nodes,
> > when we can support more than two).
> >
>
> Right, I think the system can support N-Way replication.
>
> > Perhaps "Logical replication between writers" or "Logical replication
> > between primaries" or "Replicating changes between primaries", or
> > something better.
> >
>
> Among the above "Replicating changes between primaries" sounds good to
> me or simply "Replication between primaries". As this is a sub-section
> on the Logical Replication page, I feel it is okay to not use Logical
> in the title.

I have changed it to "Replication between primaries".

> > 2. There is no mention of conflicts in the documentation, e.g.
> > referencing the "Conflicts" section of the documentation. It's very easy
> > to create a conflicting transaction that causes a subscriber to be
> > unable to continue to apply transactions:
> >
> >    -- DB 1
> >    CREATE TABLE abc (id int);
> >    CREATE PUBLICATION node1 FOR ALL TABLES ;
> >
> >    -- DB2
> >    CREATE TABLE abc (id int);
> >    CREATE PUBLICATION node2 FOR ALL TABLES ;
> >    CREATE SUBSCRIPTION node2_node1
> >      CONNECTION 'dbname=logi port=5433'
> >      PUBLICATION node1
> >      WITH (copy_data = off, origin = none);
> >
> >    -- DB1
> >    CREATE SUBSCRIPTION node1_node2
> >      CONNECTION 'dbname=logi port=5434'
> >      PUBLICATION node2
> >      WITH (copy_data = off, origin = none);
> >    INSERT INTO abc VALUES (1);
> >
> >    -- DB2
> >    INSERT INTO abc VALUES (2);
> >
> >    -- DB1
> >    ALTER TABLE abc ADD PRIMARY KEY id;
> >    INSERT INTO abc VALUES (3);
> >
> >    -- DB2
> >    INSERT INTO abc VALUES (3);
> >
> >    -- DB1 cannot apply the transactions
> >
> > At a minimum, I think we should reference the documentation we have in
> > the logical replication section on conflicts. We may also want to advise
> > that a user is responsible for designing their schemas in a way to
> > minimize the risk of conflicts.
> >
>
> This sounds reasonable to me.
>
> One more point about docs, it appears to be added as the last
> sub-section on the Logical Replication page. Is there a reason for
> doing so? I feel this should be third sub-section after describing
> Publication and Subscription.

I had initially kept it at the end since we demonstrated the various
steps to create the replication  between the primaries. Since it gives
an introduction about the "Replication between primaries" and then
states the steps, it looks ok to move it as suggested. I have modified
this in the v38 patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm01x0sLz8YzfCSjxcMFxM4NDQxcFzZa%2B4eesUmD40DdTg%40mail.gmail.com

Regards,
Vignesh

  191. Re: Handle infinite recursion in logical replication setup

    Jonathan S. Katz <jkatz@postgresql.org> — 2022-07-24T16:51:48Z

    On 7/22/22 12:47 AM, Amit Kapila wrote:
> On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:

>> 1. I'm concerned by calling this "Bidirectional replication" in the docs
>> that we are overstating the current capabilities. I think this is
>> accentuated int he opening paragraph:
>>
>> ==snip==
>>    Bidirectional replication is useful for creating a multi-master database
>>    environment for replicating read/write operations performed by any of the
>>    member nodes.
>> ==snip==
>>
>> For one, we're not replicating reads, we're replicating writes. Amongst
>> the writes, at this point we're only replicating DML. A reader could
>> think that deploying can work for a full bidirectional solution.
>>
>> (Even if we're aspirationally calling this section "Bidirectional
>> replication", that does make it sound like we're limited to two nodes,
>> when we can support more than two).
>>
> 
> Right, I think the system can support N-Way replication.

I did some more testing of the feature, i.e. doing 3-node and 4-node 
tests. While logical replication today can handle replicating between 
multiple nodes (N-way), the "origin = none" does require setting up 
subscribers between each of the nodes.

For example, if I have 4 nodes A, B, C, D and I want to replicate the 
same table between all of them, I need to set up subscribers between all 
of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node 
can replicate between each other in a way that's convenient (vs. having 
to do something funky with partitions) so this is still a big step forward.

This is a long way of saying that I do think it's fair to say we support 
"N-way" replication so long as you are set up in a mesh (vs. a ring, 
e.g. A=>B=>C=>D=>A).

> Among the above "Replicating changes between primaries" sounds good to
> me or simply "Replication between primaries". As this is a sub-section
> on the Logical Replication page, I feel it is okay to not use Logical
> in the title.

Agreed, I think that's fine.

>> At a minimum, I think we should reference the documentation we have in
>> the logical replication section on conflicts. We may also want to advise
>> that a user is responsible for designing their schemas in a way to
>> minimize the risk of conflicts.
>>
> 
> This sounds reasonable to me.
> 
> One more point about docs, it appears to be added as the last
> sub-section on the Logical Replication page. Is there a reason for
> doing so? I feel this should be third sub-section after describing
> Publication and Subscription.

When I first reviewed, I had not built the docs. Did so on this pass.

I agree with the positioning argument, i.e. it should go after 
"Subscription" in the table of contents -- but it makes me question a 
couple of things:

1. The general ordering of the docs
2. How we describe that section (more on that in a sec)
3. If "row filters" should be part of "subscription" instead of its own 
section.

If you look at the current order, "Quick setup" is the last section; one 
would think the "quick" portion goes first :) Given a lot of this is for 
the current docs, I may start a separate discussion on -docs for this part.

For the time being, I agree it should be moved to the section after 
"Subscription".

I think what this section describes is "Configuring Replication Between 
Nodes" as it covers a few different scenarios.

I do think we need to iterate on these docs -- the examples with the 
commands are generally OK and easy to follow, but a few things I noticed:

1. The general description of the section needs work. We may want to 
refine the description of the use cases, and in the warning, link to 
instructions on how to take backups.
2. We put the "case not supported" in the middle, not at the end.
3. The "generic steps for adding a new node..." section uses a 
convention for steps that is not found in the docs. We also don't 
provide an example for this section, and this is the most complicated 
scenario to set up.

I may be able to propose some suggestions in a few days.

> BTW, do you have any opinion on the idea of the first remaining patch
> where we accomplish two things: a) Checks and throws an error if
> 'copy_data = on' and 'origin = none' but the publication tables were
> also replicated from other publishers. b) Adds 'force' value for
> copy_data parameter to allow copying in such a case. The primary
> reason for this patch is to avoid loops or duplicate data in the
> initial phase. We can't skip copying based on origin as we can do
> while replicating changes from WAL. So, we detect that the publisher
> already has data from some other node and doesn't allow replication
> unless the user uses the 'force' option for copy_data.

In general, I agree with the patch; but I'm not sure why we are calling 
"copy_data = force" in this case and how it varies from "on". I 
understand the goal is to prevent the infinite loop, but is there some 
technical restriction why we can't set "origin = none, copy_data = on" 
and have this work (and apologies if I missed that upthread)?

The other concern I'll note is that we're changing a boolean parameter 
to an enum and I want to be sensitive to folks who are already using 
"copy_data" to be sure we don't break them.

Jonathan

  192. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-25T04:20:58Z

    On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 7/22/22 12:47 AM, Amit Kapila wrote:
> > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> >> 1. I'm concerned by calling this "Bidirectional replication" in the docs
> >> that we are overstating the current capabilities. I think this is
> >> accentuated int he opening paragraph:
> >>
> >> ==snip==
> >>    Bidirectional replication is useful for creating a multi-master database
> >>    environment for replicating read/write operations performed by any of the
> >>    member nodes.
> >> ==snip==
> >>
> >> For one, we're not replicating reads, we're replicating writes. Amongst
> >> the writes, at this point we're only replicating DML. A reader could
> >> think that deploying can work for a full bidirectional solution.
> >>
> >> (Even if we're aspirationally calling this section "Bidirectional
> >> replication", that does make it sound like we're limited to two nodes,
> >> when we can support more than two).
> >>
> >
> > Right, I think the system can support N-Way replication.
>
> I did some more testing of the feature, i.e. doing 3-node and 4-node
> tests. While logical replication today can handle replicating between
> multiple nodes (N-way), the "origin = none" does require setting up
> subscribers between each of the nodes.
>
> For example, if I have 4 nodes A, B, C, D and I want to replicate the
> same table between all of them, I need to set up subscribers between all
> of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node
> can replicate between each other in a way that's convenient (vs. having
> to do something funky with partitions) so this is still a big step forward.
>
> This is a long way of saying that I do think it's fair to say we support
> "N-way" replication so long as you are set up in a mesh (vs. a ring,
> e.g. A=>B=>C=>D=>A).
>

Sorry, but I don't get your point of mesh vs. ring? I think with some
care users can set up replication even in a ring topology.

> > Among the above "Replicating changes between primaries" sounds good to
> > me or simply "Replication between primaries". As this is a sub-section
> > on the Logical Replication page, I feel it is okay to not use Logical
> > in the title.
>
> Agreed, I think that's fine.
>
> >> At a minimum, I think we should reference the documentation we have in
> >> the logical replication section on conflicts. We may also want to advise
> >> that a user is responsible for designing their schemas in a way to
> >> minimize the risk of conflicts.
> >>
> >
> > This sounds reasonable to me.
> >
> > One more point about docs, it appears to be added as the last
> > sub-section on the Logical Replication page. Is there a reason for
> > doing so? I feel this should be third sub-section after describing
> > Publication and Subscription.
>
> When I first reviewed, I had not built the docs. Did so on this pass.
>
> I agree with the positioning argument, i.e. it should go after
> "Subscription" in the table of contents -- but it makes me question a
> couple of things:
>
> 1. The general ordering of the docs
> 2. How we describe that section (more on that in a sec)
> 3. If "row filters" should be part of "subscription" instead of its own
> section.
>

I don't think it is a good idea to keep "row filters" as a part of
subscription because we define those at publisher but there are
certain things like initial sync or combining of row filters that are
related to subscriptions. So, probably having it in a separate
sub-section seems okay to me. I have also thought about keeping it as
a part of Publication or Subscription but left it due to the reasons
mentioned.

> If you look at the current order, "Quick setup" is the last section; one
> would think the "quick" portion goes first :) Given a lot of this is for
> the current docs, I may start a separate discussion on -docs for this part.
>
> For the time being, I agree it should be moved to the section after
> "Subscription".
>

Okay, thanks!

> I think what this section describes is "Configuring Replication Between
> Nodes" as it covers a few different scenarios.
>
> I do think we need to iterate on these docs -- the examples with the
> commands are generally OK and easy to follow, but a few things I noticed:
>
> 1. The general description of the section needs work. We may want to
> refine the description of the use cases, and in the warning, link to
> instructions on how to take backups.
> 2. We put the "case not supported" in the middle, not at the end.
> 3. The "generic steps for adding a new node..." section uses a
> convention for steps that is not found in the docs. We also don't
> provide an example for this section, and this is the most complicated
> scenario to set up.
>

I agree with all these points, especially your point related to a
complicated setup. This won't be easy for users without a very clear
description and examples. However, after this work, it will at least
be possible for users to set up an N-Way replication with some
restrictions and care.  BTW, while working on this we have noticed
that it is normally a lot of work for users to set up N-way
replication among 3, 4, or more nodes and that is why there is another
proposal to make that set up easier by providing a simpler APIs which
will internally do similar to all the manual steps the 0002 patch is
trying to describe. See [1] (please don't be confused with the thread
title but the real intent is to allow users to provide an easier way
to set up an N-Way replication by having all current restrictions of
Logical Replication like it doesn't support DDL replication)

> I may be able to propose some suggestions in a few days.
>

Okay, thanks! The plan is to get the 0001 patch of Vignesh. Then work
on docs to describe how users can set up replication among primary
nodes. After that, if we get consensus on providing simpler APIs for
setting up replication among primary nodes as is being discussed in
the thread [1], then work on it.

> > BTW, do you have any opinion on the idea of the first remaining patch
> > where we accomplish two things: a) Checks and throws an error if
> > 'copy_data = on' and 'origin = none' but the publication tables were
> > also replicated from other publishers. b) Adds 'force' value for
> > copy_data parameter to allow copying in such a case. The primary
> > reason for this patch is to avoid loops or duplicate data in the
> > initial phase. We can't skip copying based on origin as we can do
> > while replicating changes from WAL. So, we detect that the publisher
> > already has data from some other node and doesn't allow replication
> > unless the user uses the 'force' option for copy_data.
>
> In general, I agree with the patch; but I'm not sure why we are calling
> "copy_data = force" in this case and how it varies from "on". I
> understand the goal is to prevent the infinite loop, but is there some
> technical restriction why we can't set "origin = none, copy_data = on"
> and have this work (and apologies if I missed that upthread)?
>

The technical restriction is that we want to throw an error when users
set "origin = none, copy_data = on" and we find that the publication
tables were also replicated from other publishers. Now, we can give
WARNING to users in this case but users won't have any way to avoid
duplicate data which can lead to constraint violations. So, we decided
to throw ERROR and allow users to perform it with a "force" option.

> The other concern I'll note is that we're changing a boolean parameter
> to an enum and I want to be sensitive to folks who are already using
> "copy_data" to be sure we don't break them.
>

Okay, but AFAIU, the patch still allows users to specify on/off, so
won't that be sufficient?

[1] - https://www.postgresql.org/message-id/CAHut%2BPuwRAoWY9pz%3DEubps3ooQCOBFiYPU9Yi%3DVB-U%2ByORU7OA%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  193. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-25T07:28:40Z

    Firstly, I have some (case-sensitivity) questions about the previous
patch which was already pushed [1].

Q1. create_subscription docs

I did not understand why the docs refer to slot_name = NONE, yet the
newly added option says origin = none/any. I think that saying origin
= NONE/ANY would be more consistent with the existing usage of NONE in
this documentation.

~~~

Q2. parse_subscription_options

Similarly, in the code (parse_subscription_options), I did not
understand why the checks for special name values are implemented
differently:

The new 'origin' code is using pg_strcmpcase to check special values
(none/any), and the old 'slot_name' code uses case-sensitive strcmp to
check the special value (none).

FWIW, here I thought the new origin code is the correct one.

======

Now, here are some review comments for the patch v38-0001:

1. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

@@ -1781,6 +1858,121 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
  table_close(rel, RowExclusiveLock);
 }

+/*
+ * Check and throw an error if the publisher has subscribed to the same table
+ * from some other publisher. This check is required only if copydata is ON and
+ * the origin is local for CREATE SUBSCRIPTION and
+ * ALTER SUBSCRIPTION ... REFRESH statements to avoid replicating remote data
+ * from the publisher.
+ *
+ * This check need not be performed on the tables that are already added as
+ * incremental sync for such tables will happen through WAL and the origin of
+ * the data can be identified from the WAL records.
+ *
+ * subrel_local_oids contains the list of relation oids that are already
+ * present on the subscriber.
+ */
+static void
+check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
+    CopyData copydata, char *origin,
+    Oid *subrel_local_oids, int subrel_count)

1a.
"copydata is ON" --> "copy_data = on" (because the comment is talking
about the CREATE/ALTER statements, so it seemed a bit confusing to
refer to the copydata function param instead of the copy_data
subscription parameter)

1b.
"the origin is local" ?? But, "local" was the old special name value.
Now it is "none", so I think this part needs minor rewording.

~~~

2.

+ if (copydata != COPY_DATA_ON || !origin ||
+ (pg_strcasecmp(origin, "none") != 0))
+ return;

Should this be using the constant LOGICALREP_ORIGIN_NONE?

~~~

3.

+ /*
+ * Throw an error if the publisher has subscribed to the same table
+ * from some other publisher. We cannot differentiate between the
+ * origin and non-origin data that is present in the HEAP during the
+ * initial sync. Identification of non-origin data can be done only
+ * from the WAL by using the origin id.
+ *
+ * XXX: For simplicity, we don't check whether the table has any data
+ * or not. If the table doesn't have any data then we don't need to
+ * distinguish between local and non-local data so we can avoid
+ * throwing an error in that case.
+ */

3a.
When the special origin value changed from "local" to "none" this
comment's first part seems to have got a bit lost in translation.

SUGGESTION:
Throw an error if the publisher has subscribed to the same table from
some other publisher. We cannot know the origin of data during the
initial sync. Data origins can be found only from the WAL by looking
at the origin id.

3b.
I think referring to "local and non-local" data in the XXX part of
this comment also needs some minor rewording now that "local" is not a
special origin name anymore.

------
[1] https://github.com/postgres/postgres/commit/366283961ac0ed6d89014444c6090f3fd02fce0a

Kind Regards,
Peter Smith.
Fujitsu Australia

  194. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-25T08:54:00Z

    On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 7/22/22 12:47 AM, Amit Kapila wrote:
> > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> >> 1. I'm concerned by calling this "Bidirectional replication" in the docs
> >> that we are overstating the current capabilities. I think this is
> >> accentuated int he opening paragraph:
> >>
> >> ==snip==
> >>    Bidirectional replication is useful for creating a multi-master database
> >>    environment for replicating read/write operations performed by any of the
> >>    member nodes.
> >> ==snip==
> >>
> >> For one, we're not replicating reads, we're replicating writes. Amongst
> >> the writes, at this point we're only replicating DML. A reader could
> >> think that deploying can work for a full bidirectional solution.
> >>
> >> (Even if we're aspirationally calling this section "Bidirectional
> >> replication", that does make it sound like we're limited to two nodes,
> >> when we can support more than two).
> >>
> >
> > Right, I think the system can support N-Way replication.
>
> I did some more testing of the feature, i.e. doing 3-node and 4-node
> tests. While logical replication today can handle replicating between
> multiple nodes (N-way), the "origin = none" does require setting up
> subscribers between each of the nodes.
>
> For example, if I have 4 nodes A, B, C, D and I want to replicate the
> same table between all of them, I need to set up subscribers between all
> of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node
> can replicate between each other in a way that's convenient (vs. having
> to do something funky with partitions) so this is still a big step forward.
>
> This is a long way of saying that I do think it's fair to say we support
> "N-way" replication so long as you are set up in a mesh (vs. a ring,
> e.g. A=>B=>C=>D=>A).
>
> > Among the above "Replicating changes between primaries" sounds good to
> > me or simply "Replication between primaries". As this is a sub-section
> > on the Logical Replication page, I feel it is okay to not use Logical
> > in the title.
>
> Agreed, I think that's fine.
>
> >> At a minimum, I think we should reference the documentation we have in
> >> the logical replication section on conflicts. We may also want to advise
> >> that a user is responsible for designing their schemas in a way to
> >> minimize the risk of conflicts.
> >>
> >
> > This sounds reasonable to me.
> >
> > One more point about docs, it appears to be added as the last
> > sub-section on the Logical Replication page. Is there a reason for
> > doing so? I feel this should be third sub-section after describing
> > Publication and Subscription.
>
> When I first reviewed, I had not built the docs. Did so on this pass.
>
> I agree with the positioning argument, i.e. it should go after
> "Subscription" in the table of contents -- but it makes me question a
> couple of things:
>
> 1. The general ordering of the docs
> 2. How we describe that section (more on that in a sec)
> 3. If "row filters" should be part of "subscription" instead of its own
> section.
>
> If you look at the current order, "Quick setup" is the last section; one
> would think the "quick" portion goes first :) Given a lot of this is for
> the current docs, I may start a separate discussion on -docs for this part.
>
> For the time being, I agree it should be moved to the section after
> "Subscription".
>
> I think what this section describes is "Configuring Replication Between
> Nodes" as it covers a few different scenarios.
>
> I do think we need to iterate on these docs -- the examples with the
> commands are generally OK and easy to follow, but a few things I noticed:
>
> 1. The general description of the section needs work. We may want to
> refine the description of the use cases, and in the warning, link to
> instructions on how to take backups.
> 2. We put the "case not supported" in the middle, not at the end.
> 3. The "generic steps for adding a new node..." section uses a
> convention for steps that is not found in the docs. We also don't
> provide an example for this section, and this is the most complicated
> scenario to set up.
>
> I may be able to propose some suggestions in a few days.
>
> > BTW, do you have any opinion on the idea of the first remaining patch
> > where we accomplish two things: a) Checks and throws an error if
> > 'copy_data = on' and 'origin = none' but the publication tables were
> > also replicated from other publishers. b) Adds 'force' value for
> > copy_data parameter to allow copying in such a case. The primary
> > reason for this patch is to avoid loops or duplicate data in the
> > initial phase. We can't skip copying based on origin as we can do
> > while replicating changes from WAL. So, we detect that the publisher
> > already has data from some other node and doesn't allow replication
> > unless the user uses the 'force' option for copy_data.
>
> In general, I agree with the patch; but I'm not sure why we are calling
> "copy_data = force" in this case and how it varies from "on". I
> understand the goal is to prevent the infinite loop, but is there some
> technical restriction why we can't set "origin = none, copy_data = on"
> and have this work (and apologies if I missed that upthread)?

Let's take a simple case to understand why copy_data = force is
required to replicate between two primaries for table t1 which has
data as given below:
Node-1:
Table t1 (c1 int) has data
1, 2, 3, 4

Node-2:
Table t1 (c1 int) has data
5, 6, 7, 8

step1 - Node-1
#Publication for t1
Create Publication pub1_2 For Table t1;

step2 - Node-2
#Publication for t1,
Create Publication pub2_1 For Table t1;

step3 - Node-1:
Create Subscription sub1 Connection '<node-2 details>' publication
pub2_1 with (origin = none);

After this the data will be something like this:
Node-1:
1, 2, 3, 4, 5, 6, 7, 8

Node-2:
5, 6, 7, 8

step4 - Node-2:
Create Subscription sub2 Connection '<node-1 details>' Publication
pub1_2 with (origin = none, copy_data=on);
If we had allowed the create subscription to be successful with
copy_data = on. After this the data will be something like this:
Node-1:
1, 2, 3, 4, 5, 6, 7, 8

Node-2:
1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8

So, you can see that data on Node-2 (5, 6, 7, 8) is duplicated. In
case, table t1 has a unique key, it will lead to a unique key
violation and replication won't proceed.

To avoid this we will throw an error:
ERROR:  could not replicate table "public.t1"
DETAIL:  CREATE/ALTER SUBSCRIPTION with origin = none and copy_data =
on is not allowed when the publisher has subscribed same table.
HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force.

Users can then overcome this problem by using the following steps:
step1 to step3 is the same as above.

step4 - Node-2
# Disallow truncates to be published and then truncate the table
Alter Publication pub2_1 Set (publish = 'insert, update, delete');
Truncate t1;

After this the data will be like this:
Node-1:
1, 2, 3, 4, 5, 6, 7, 8

Node-2: no data

step5 - Node-2
Create Subscription sub2 Connection '<node-1 details>' Publication
pub1_2 with (origin = none, copy_data = force);

After this the data will be in sync:
Node-1:
1, 2, 3, 4, 5, 6, 7, 8

Node-2:
1, 2, 3, 4, 5, 6, 7, 8

step6 - Node-1
# Now include truncates to be published
Alter Publication pub1_2 Set (publish = 'insert, update, delete, truncate');

Now the replication setup between two primaries node1 and node2 is
complete. Any incremental changes from node1 will be replicated to
node2, and any incremental changes from node2 will be replicated to
node1.

Regards,
Vignesh

  195. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-25T08:54:43Z

    On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Firstly, I have some (case-sensitivity) questions about the previous
> patch which was already pushed [1].
>
> Q1. create_subscription docs
>
> I did not understand why the docs refer to slot_name = NONE, yet the
> newly added option says origin = none/any. I think that saying origin
> = NONE/ANY would be more consistent with the existing usage of NONE in
> this documentation.
>
> ~~~
>
> Q2. parse_subscription_options
>
> Similarly, in the code (parse_subscription_options), I did not
> understand why the checks for special name values are implemented
> differently:
>
> The new 'origin' code is using pg_strcmpcase to check special values
> (none/any), and the old 'slot_name' code uses case-sensitive strcmp to
> check the special value (none).
>

We have a restriction for slot_names for lower case letters (aka
"Replication slot names may only contain lower case letters, numbers,
and the underscore character.") whereas there is no such restriction
in origin name, that's why the check is different. So, if you try with
slot_name = 'NONE', you will get the error.

-- 
With Regards,
Amit Kapila.

  196. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-25T09:33:25Z

    On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Firstly, I have some (case-sensitivity) questions about the previous
> patch which was already pushed [1].
>
> Q1. create_subscription docs
>
> I did not understand why the docs refer to slot_name = NONE, yet the
> newly added option says origin = none/any. I think that saying origin
> = NONE/ANY would be more consistent with the existing usage of NONE in
> this documentation.

Both NONE and none are ok in the case of origin, if you want I can
change it to NONE/ANY in case of origin to keep it consistent.

Regards,
Vignesh

  197. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-25T14:44:28Z

    On Mon, Jul 25, 2022 at 2:24 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> >
> > On 7/22/22 12:47 AM, Amit Kapila wrote:
> > > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> >
> > >> 1. I'm concerned by calling this "Bidirectional replication" in the docs
> > >> that we are overstating the current capabilities. I think this is
> > >> accentuated int he opening paragraph:
> > >>
> > >> ==snip==
> > >>    Bidirectional replication is useful for creating a multi-master database
> > >>    environment for replicating read/write operations performed by any of the
> > >>    member nodes.
> > >> ==snip==
> > >>
> > >> For one, we're not replicating reads, we're replicating writes. Amongst
> > >> the writes, at this point we're only replicating DML. A reader could
> > >> think that deploying can work for a full bidirectional solution.
> > >>
> > >> (Even if we're aspirationally calling this section "Bidirectional
> > >> replication", that does make it sound like we're limited to two nodes,
> > >> when we can support more than two).
> > >>
> > >
> > > Right, I think the system can support N-Way replication.
> >
> > I did some more testing of the feature, i.e. doing 3-node and 4-node
> > tests. While logical replication today can handle replicating between
> > multiple nodes (N-way), the "origin = none" does require setting up
> > subscribers between each of the nodes.
> >
> > For example, if I have 4 nodes A, B, C, D and I want to replicate the
> > same table between all of them, I need to set up subscribers between all
> > of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node
> > can replicate between each other in a way that's convenient (vs. having
> > to do something funky with partitions) so this is still a big step forward.
> >
> > This is a long way of saying that I do think it's fair to say we support
> > "N-way" replication so long as you are set up in a mesh (vs. a ring,
> > e.g. A=>B=>C=>D=>A).
> >
> > > Among the above "Replicating changes between primaries" sounds good to
> > > me or simply "Replication between primaries". As this is a sub-section
> > > on the Logical Replication page, I feel it is okay to not use Logical
> > > in the title.
> >
> > Agreed, I think that's fine.
> >
> > >> At a minimum, I think we should reference the documentation we have in
> > >> the logical replication section on conflicts. We may also want to advise
> > >> that a user is responsible for designing their schemas in a way to
> > >> minimize the risk of conflicts.
> > >>
> > >
> > > This sounds reasonable to me.
> > >
> > > One more point about docs, it appears to be added as the last
> > > sub-section on the Logical Replication page. Is there a reason for
> > > doing so? I feel this should be third sub-section after describing
> > > Publication and Subscription.
> >
> > When I first reviewed, I had not built the docs. Did so on this pass.
> >
> > I agree with the positioning argument, i.e. it should go after
> > "Subscription" in the table of contents -- but it makes me question a
> > couple of things:
> >
> > 1. The general ordering of the docs
> > 2. How we describe that section (more on that in a sec)
> > 3. If "row filters" should be part of "subscription" instead of its own
> > section.
> >
> > If you look at the current order, "Quick setup" is the last section; one
> > would think the "quick" portion goes first :) Given a lot of this is for
> > the current docs, I may start a separate discussion on -docs for this part.
> >
> > For the time being, I agree it should be moved to the section after
> > "Subscription".
> >
> > I think what this section describes is "Configuring Replication Between
> > Nodes" as it covers a few different scenarios.
> >
> > I do think we need to iterate on these docs -- the examples with the
> > commands are generally OK and easy to follow, but a few things I noticed:
> >
> > 1. The general description of the section needs work. We may want to
> > refine the description of the use cases, and in the warning, link to
> > instructions on how to take backups.
> > 2. We put the "case not supported" in the middle, not at the end.
> > 3. The "generic steps for adding a new node..." section uses a
> > convention for steps that is not found in the docs. We also don't
> > provide an example for this section, and this is the most complicated
> > scenario to set up.
> >
> > I may be able to propose some suggestions in a few days.
> >
> > > BTW, do you have any opinion on the idea of the first remaining patch
> > > where we accomplish two things: a) Checks and throws an error if
> > > 'copy_data = on' and 'origin = none' but the publication tables were
> > > also replicated from other publishers. b) Adds 'force' value for
> > > copy_data parameter to allow copying in such a case. The primary
> > > reason for this patch is to avoid loops or duplicate data in the
> > > initial phase. We can't skip copying based on origin as we can do
> > > while replicating changes from WAL. So, we detect that the publisher
> > > already has data from some other node and doesn't allow replication
> > > unless the user uses the 'force' option for copy_data.
> >
> > In general, I agree with the patch; but I'm not sure why we are calling
> > "copy_data = force" in this case and how it varies from "on". I
> > understand the goal is to prevent the infinite loop, but is there some
> > technical restriction why we can't set "origin = none, copy_data = on"
> > and have this work (and apologies if I missed that upthread)?
>
> Let's take a simple case to understand why copy_data = force is
> required to replicate between two primaries for table t1 which has
> data as given below:
> Node-1:
> Table t1 (c1 int) has data
> 1, 2, 3, 4
>
> Node-2:
> Table t1 (c1 int) has data
> 5, 6, 7, 8
>
> step1 - Node-1
> #Publication for t1
> Create Publication pub1_2 For Table t1;
>
> step2 - Node-2
> #Publication for t1,
> Create Publication pub2_1 For Table t1;
>
> step3 - Node-1:
> Create Subscription sub1 Connection '<node-2 details>' publication
> pub2_1 with (origin = none);
>
> After this the data will be something like this:
> Node-1:
> 1, 2, 3, 4, 5, 6, 7, 8
>
> Node-2:
> 5, 6, 7, 8
>
> step4 - Node-2:
> Create Subscription sub2 Connection '<node-1 details>' Publication
> pub1_2 with (origin = none, copy_data=on);
> If we had allowed the create subscription to be successful with
> copy_data = on. After this the data will be something like this:
> Node-1:
> 1, 2, 3, 4, 5, 6, 7, 8
>
> Node-2:
> 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8
>
> So, you can see that data on Node-2 (5, 6, 7, 8) is duplicated. In
> case, table t1 has a unique key, it will lead to a unique key
> violation and replication won't proceed.
>
> To avoid this we will throw an error:
> ERROR:  could not replicate table "public.t1"
> DETAIL:  CREATE/ALTER SUBSCRIPTION with origin = none and copy_data =
> on is not allowed when the publisher has subscribed same table.
> HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force.
>
> Users can then overcome this problem by using the following steps:
> step1 to step3 is the same as above.
>
> step4 - Node-2
> # Disallow truncates to be published and then truncate the table
> Alter Publication pub2_1 Set (publish = 'insert, update, delete');
> Truncate t1;
>
> After this the data will be like this:
> Node-1:
> 1, 2, 3, 4, 5, 6, 7, 8
>
> Node-2: no data
>
> step5 - Node-2
> Create Subscription sub2 Connection '<node-1 details>' Publication
> pub1_2 with (origin = none, copy_data = force);
>
> After this the data will be in sync:
> Node-1:
> 1, 2, 3, 4, 5, 6, 7, 8
>
> Node-2:
> 1, 2, 3, 4, 5, 6, 7, 8
>
> step6 - Node-1
> # Now include truncates to be published
> Alter Publication pub1_2 Set (publish = 'insert, update, delete, truncate');
>
> Now the replication setup between two primaries node1 and node2 is
> complete. Any incremental changes from node1 will be replicated to
> node2, and any incremental changes from node2 will be replicated to
> node1.

In the above steps, sorry that I mentioned Node-1 instead of Node-2 in
the last step step6.
The below:
 step6 - Node-1
 # Now include truncates to be published
 Alter Publication pub1_2 Set (publish = 'insert, update, delete, truncate');

should be:
step6 - Node-2
# Now include truncates to be published
Alter Publication pub2_1 Set (publish = 'insert, update, delete, truncate');

Regards,
Vignesh

  198. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-25T23:13:17Z

    On Mon, Jul 25, 2022 at 6:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
...
> >
> > Q2. parse_subscription_options
> >
> > Similarly, in the code (parse_subscription_options), I did not
> > understand why the checks for special name values are implemented
> > differently:
> >
> > The new 'origin' code is using pg_strcmpcase to check special values
> > (none/any), and the old 'slot_name' code uses case-sensitive strcmp to
> > check the special value (none).
> >
>
> We have a restriction for slot_names for lower case letters (aka
> "Replication slot names may only contain lower case letters, numbers,
> and the underscore character.") whereas there is no such restriction
> in origin name, that's why the check is different. So, if you try with
> slot_name = 'NONE', you will get the error.
>

2022-07-26 09:06:06.380 AEST [3630] STATEMENT:  create subscription
mysub connection 'host=localhost' publication mypub with
(slot_name='None', enabled=false, create_slot=false);
ERROR:  replication slot name "None" contains invalid character
HINT:  Replication slot names may only contain lower case letters,
numbers, and the underscore character.

You are right. Thanks for the explanation.

(Aside: Probably that error message wording ought to say "contains
invalid characters" instead of "contains invalid character")

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  199. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-25T23:34:11Z

    On Mon, Jul 25, 2022 at 7:33 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Firstly, I have some (case-sensitivity) questions about the previous
> > patch which was already pushed [1].
> >
> > Q1. create_subscription docs
> >
> > I did not understand why the docs refer to slot_name = NONE, yet the
> > newly added option says origin = none/any. I think that saying origin
> > = NONE/ANY would be more consistent with the existing usage of NONE in
> > this documentation.
>
> Both NONE and none are ok in the case of origin, if you want I can
> change it to NONE/ANY in case of origin to keep it consistent.
>

I preferred the special origin values should be documented as NONE/ANY
for better consistency, but let's see what others think about it.

There will also be associated minor changes needed for a few
error/hint messages.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  200. Re: Handle infinite recursion in logical replication setup

    Jonathan S. Katz <jkatz@postgresql.org> — 2022-07-26T01:43:37Z

    On 7/25/22 4:54 AM, vignesh C wrote:
> On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>>
>> On 7/22/22 12:47 AM, Amit Kapila wrote:
>>> On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:

>>> BTW, do you have any opinion on the idea of the first remaining patch
>>> where we accomplish two things: a) Checks and throws an error if
>>> 'copy_data = on' and 'origin = none' but the publication tables were
>>> also replicated from other publishers. b) Adds 'force' value for
>>> copy_data parameter to allow copying in such a case. The primary
>>> reason for this patch is to avoid loops or duplicate data in the
>>> initial phase. We can't skip copying based on origin as we can do
>>> while replicating changes from WAL. So, we detect that the publisher
>>> already has data from some other node and doesn't allow replication
>>> unless the user uses the 'force' option for copy_data.
>>
>> In general, I agree with the patch; but I'm not sure why we are calling
>> "copy_data = force" in this case and how it varies from "on". I
>> understand the goal is to prevent the infinite loop, but is there some
>> technical restriction why we can't set "origin = none, copy_data = on"
>> and have this work (and apologies if I missed that upthread)?
> 
> Let's take a simple case to understand why copy_data = force is
> required to replicate between two primaries for table t1 which has
> data as given below:

> step4 - Node-2:
> Create Subscription sub2 Connection '<node-1 details>' Publication
> pub1_2 with (origin = none, copy_data=on);
> If we had allowed the create subscription to be successful with
> copy_data = on. After this the data will be something like this:
> Node-1:
> 1, 2, 3, 4, 5, 6, 7, 8
> 
> Node-2:
> 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8
> 
> So, you can see that data on Node-2 (5, 6, 7, 8) is duplicated. In
> case, table t1 has a unique key, it will lead to a unique key
> violation and replication won't proceed.
> 
> To avoid this we will throw an error:
> ERROR:  could not replicate table "public.t1"
> DETAIL:  CREATE/ALTER SUBSCRIPTION with origin = none and copy_data =
> on is not allowed when the publisher has subscribed same table.
> HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force.

Thanks for the example. I agree that it is fairly simple to reproduce.

I understand that "copy_data = force" is meant to protect a user from 
hurting themself. I'm not convinced that this is the best way to do so.

For example today I can subscribe to multiple publications that write to 
the same table. If I have a primary key on that table, and two of the 
subscriptions try to write an identical ID, we conflict. We don't have 
any special flags or modes to guard against that from happening, though 
we do have documentation on conflicts and managing them.

AFAICT the same issue with "copy_data" also exists in the above scenario 
too, even without the "origin" attribute. However, I think this case is 
more noticeable for "origin=none" because we currently default 
"copy_data" to "true" and in this case data can be copied in two 
directions.

That said, this introduces a new restriction for this particular 
scenario that doesn't exist on other scenarios. Instead, I would 
advocate we document how to correctly set up the two-way replication 
scenario (which we have a draft!), document the warnings around the 
conflicts, perhaps include Vignesh's instructions on how to remediate a 
conflict on initial sync, and consider throwing a WARNING as you suggested.

Thoughts?

Thanks,

Jonathan

  201. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-26T01:45:52Z

    Here are some review comments for the patch v38-0002:

======

<General> - terminology

There seemed to be an inconsistent alternation of the terms
"primaries" and "nodes"... For example "Setting replication between
two primaries" versus "Adding a new node..." (instead of "Adding a new
primary..."?). I have included suggested minor rewording changes in
the review comments below, but please check in case I miss something.
Because I suggested changes to some titles maybe you will also want to
change the section ids too.

~~~

1. Commit message

The documentation was recently modified to remove the term
"bidirectional replication" and replace it all with "replication
between primaries", so this commit message (and also the patch name
itself) should be similarly modified.

~~~

2.
+   <para>
+    Replication between primaries is useful for creating a multi-master
+    database environment for replicating write operations performed by any of
+    the member nodes. The steps to create replication between primaries in
+    various scenarios are given below. Note: User is responsible for designing
+    their schemas in a way to minimize the risk of conflicts. See
+    <xref linkend="logical-replication-conflicts"/> for the details of logical
+    replication conflicts. The logical replication restrictions applies to
+    the replication between primaries also. See
+    <xref linkend="logical-replication-restrictions"/> for the details of
+    logical replication restrictions.
+   </para>

2a.
"User" -> "The user"

2b.
"The logical replication restrictions applies to..." --> "The logical
replication restrictions apply to..."

2c.
These are important notes. Instead of just being part of the text
blurb, perhaps these should be rendered as SGML <note> (or put them
both in a single <note> if you want)

~~~

3. Setting replication between two primaries

+   <title>Setting replication between two primaries</title>
+   <para>
+    The following steps demonstrate how to setup replication between two
+    primaries when there is no table data present on both nodes
+    <literal>node1</literal> and <literal>node2</literal>:
+   </para>

SUGGESTED
The following steps demonstrate how to set up replication between two
primaries (node1 and node2) when there is no table data present on
both nodes:

~~~

4.
+   <para>
+    Now the replication setup between two primaries <literal>node1</literal>
+    and <literal>node2</literal> is complete. Any incremental changes from
+    <literal>node1</literal> will be replicated to <literal>node2</literal>,
+    and any incremental changes from <literal>node2</literal> will be
+    replicated to <literal>node1</literal>.
+   </para>

"between two primaries" -> "between primaries"

~~~

5. Adding a new node when there is no table data on any of the nodes

SUGGESTION (title)
Adding a new primary when there is no table data on any of the primaries

~~~

6.
+   <para>
+    The following steps demonstrate adding a new node <literal>node3</literal>
+    to the existing <literal>node1</literal> and <literal>node2</literal> when
+    there is no <literal>t1</literal> data on any of the nodes. This requires

SUGGESTION
The following steps demonstrate adding a new primary (node3) to the
existing primaries (node1 and node2) when there is no t1 data on any
of the nodes.

~~~

7. Adding a new node when table data is present on the existing nodes

SUGGESTION (title)
Adding a new primary when table data is present on the existing primaries

~~~

8.
+    <para>
+     The following steps demonstrate adding a new node <literal>node3</literal>
+     which has no <literal>t1</literal> data to the existing
+     <literal>node1</literal> and <literal>node2</literal> where
+     <literal>t1</literal> data is present. This needs similar steps; the only

SUGGESTION
The following steps demonstrate adding a new primary (node3) that has
no t1 data to the existing primaries (node1 and node2) where t1 data
is present.

~~~

9. Adding a new node when table data is present on the new node

SUGGESTION (title)
Adding a new primary that has existing table data

~~~

10.
+   <note>
+    <para>
+     Adding a new node when table data is present on the new node is not
+     supported.
+    </para>
+   </note>

SUGGESTION
Adding a new primary that has existing table data is not supported.

~~~

11. Generic steps for adding a new node to an existing set of primaries

SUGGESTION (title)
Generic steps for adding a new primary to an existing set of primaries

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  202. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-26T02:18:05Z

    On Tue, Jul 26, 2022 at 11:43 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
...
> That said, this introduces a new restriction for this particular
> scenario that doesn't exist on other scenarios. Instead, I would
> advocate we document how to correctly set up the two-way replication
> scenario (which we have a draft!), document the warnings around the
> conflicts, perhaps include Vignesh's instructions on how to remediate a
> conflict on initial sync, and consider throwing a WARNING as you suggested.
>
> Thoughts?

Perhaps a WARNING can be useful if the SUBSCRIPTION was created with
enabled=false because then the user still has a chance to reconsider,
but otherwise, I don't see what good a warning does if the potentially
harmful initial copy is going to proceed anyway; isn't that like
putting a warning sign at the bottom of a cliff?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  203. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-26T03:37:04Z

    On Tue, Jul 26, 2022 at 7:13 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 7/25/22 4:54 AM, vignesh C wrote:
> >
> > Let's take a simple case to understand why copy_data = force is
> > required to replicate between two primaries for table t1 which has
> > data as given below:
>
> > step4 - Node-2:
> > Create Subscription sub2 Connection '<node-1 details>' Publication
> > pub1_2 with (origin = none, copy_data=on);
> > If we had allowed the create subscription to be successful with
> > copy_data = on. After this the data will be something like this:
> > Node-1:
> > 1, 2, 3, 4, 5, 6, 7, 8
> >
> > Node-2:
> > 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8
> >
> > So, you can see that data on Node-2 (5, 6, 7, 8) is duplicated. In
> > case, table t1 has a unique key, it will lead to a unique key
> > violation and replication won't proceed.
> >
> > To avoid this we will throw an error:
> > ERROR:  could not replicate table "public.t1"
> > DETAIL:  CREATE/ALTER SUBSCRIPTION with origin = none and copy_data =
> > on is not allowed when the publisher has subscribed same table.
> > HINT:  Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force.
>
> Thanks for the example. I agree that it is fairly simple to reproduce.
>
> I understand that "copy_data = force" is meant to protect a user from
> hurting themself. I'm not convinced that this is the best way to do so.
>
> For example today I can subscribe to multiple publications that write to
> the same table. If I have a primary key on that table, and two of the
> subscriptions try to write an identical ID, we conflict. We don't have
> any special flags or modes to guard against that from happening, though
> we do have documentation on conflicts and managing them.
>
> AFAICT the same issue with "copy_data" also exists in the above scenario
> too, even without the "origin" attribute.
>

That's true but there is no parameter like origin = NONE which
indicates that constraint violations or duplicate data problems won't
occur due to replication. In the current case, I think the situation
is different because a user has specifically asked not to replicate
any remote data by specifying origin = NONE, which should be dealt
differently. Note that current users or their setup won't see any
difference/change unless they specify the new parameter origin as
NONE.

-- 
With Regards,
Amit Kapila.

  204. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-26T03:42:23Z

    On Tue, Jul 26, 2022 at 7:48 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jul 26, 2022 at 11:43 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> >
> ...
> > That said, this introduces a new restriction for this particular
> > scenario that doesn't exist on other scenarios. Instead, I would
> > advocate we document how to correctly set up the two-way replication
> > scenario (which we have a draft!), document the warnings around the
> > conflicts, perhaps include Vignesh's instructions on how to remediate a
> > conflict on initial sync, and consider throwing a WARNING as you suggested.
> >
> > Thoughts?
>
> Perhaps a WARNING can be useful if the SUBSCRIPTION was created with
> enabled=false because then the user still has a chance to reconsider,
>

Agreed. I think but in that case, when the user enables it, we need to
ensure that we won't allow replicating (during initial sync) remote
data. If this is really required/preferred, it can be done as a
separate enhancement.

-- 
With Regards,
Amit Kapila.

  205. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-26T04:09:01Z

    On Tue, Jul 26, 2022 at 5:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Jul 25, 2022 at 7:33 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Firstly, I have some (case-sensitivity) questions about the previous
> > > patch which was already pushed [1].
> > >
> > > Q1. create_subscription docs
> > >
> > > I did not understand why the docs refer to slot_name = NONE, yet the
> > > newly added option says origin = none/any. I think that saying origin
> > > = NONE/ANY would be more consistent with the existing usage of NONE in
> > > this documentation.
> >
> > Both NONE and none are ok in the case of origin, if you want I can
> > change it to NONE/ANY in case of origin to keep it consistent.
> >
>
> I preferred the special origin values should be documented as NONE/ANY
> for better consistency, but let's see what others think about it.
>
> There will also be associated minor changes needed for a few
> error/hint messages.
>

I am not really sure how much we gain by maintaining consistency with
slot_name because if due to this we have to change the error messages
as well then it can create an inconsistency with reserved origin
names. Consider message: DETAIL:  Origin names "any", "none", and
names starting with "pg_" are reserved. Now, if we change this to
"ANY", "NONE" in the above message, it will look a bit odd as "pg_"
starts with lower case letters.

--
With Regards,
Amit Kapila.

  206. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-26T04:53:05Z

    On Tue, Jul 26, 2022 at 2:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jul 26, 2022 at 5:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Mon, Jul 25, 2022 at 7:33 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Firstly, I have some (case-sensitivity) questions about the previous
> > > > patch which was already pushed [1].
> > > >
> > > > Q1. create_subscription docs
> > > >
> > > > I did not understand why the docs refer to slot_name = NONE, yet the
> > > > newly added option says origin = none/any. I think that saying origin
> > > > = NONE/ANY would be more consistent with the existing usage of NONE in
> > > > this documentation.
> > >
> > > Both NONE and none are ok in the case of origin, if you want I can
> > > change it to NONE/ANY in case of origin to keep it consistent.
> > >
> >
> > I preferred the special origin values should be documented as NONE/ANY
> > for better consistency, but let's see what others think about it.
> >
> > There will also be associated minor changes needed for a few
> > error/hint messages.
> >
>
> I am not really sure how much we gain by maintaining consistency with
> slot_name because if due to this we have to change the error messages
> as well then it can create an inconsistency with reserved origin
> names. Consider message: DETAIL:  Origin names "any", "none", and
> names starting with "pg_" are reserved. Now, if we change this to
> "ANY", "NONE" in the above message, it will look a bit odd as "pg_"
> starts with lower case letters.
>

Sure, the message looks a bit odd with the quotes like you wrote
above, but I would not suggest to change it that way - I was thinking
more like below (which is similar to the style the slot_name messages
use)

CURRENT
DETAIL:  Origin names "any", "none", and names starting with "pg_" are reserved.

SUGGESTED
DETAIL:  Origin names ANY, NONE, and names starting with "pg_" are reserved.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  207. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-07-26T08:05:05Z

    On Sun, Jul 24, 2022 1:28 AM vignesh C <vignesh21@gmail.com> wrote:
> 
> Added a  note for the same and referred it to the conflicts section.
> 
> Thanks for the comments, the attached v38 patch has the changes for the
> same.
> 

Thanks for updating the patch. A comment on the test in 0001 patch.

+# Alter subscription ... refresh publication should fail when a new table is
+# subscribing data from a different publication should fail
+($result, $stdout, $stderr) = $node_A->psql(
+	'postgres', "
+        ALTER SUBSCRIPTION tap_sub_A2 REFRESH PUBLICATION");
+like(
+	$stderr,
+	qr/ERROR: ( [A-Z0-9]+:)? could not replicate table "public.tab_new"/,
+	"Create subscription with origin and copy_data having replicated table in publisher"
+);

The comment says "should fail" twice, the latter one can be removed.

Besides, "Create subscription with origin and copy_data" should be changed to
"Alter subscription with origin and copy_data" I think.

Regards,
Shi yu

  208. RE: Handle infinite recursion in logical replication setup

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2022-07-26T08:42:09Z

    On Sun, Jul 24, 2022 1:28 AM vignesh C <vignesh21@gmail.com> wrote:
> Added a  note for the same and referred it to the conflicts section.
> 
> Thanks for the comments, the attached v38 patch has the changes for the same.

Thanks for your patches.

Two slight comments on the below message in the 0001 patch:

The error message in the function check_pub_table_subscribed().
+		ereport(ERROR,
+				errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+				errmsg("could not replicate table \"%s.%s\"",
+					   nspname, relname),
+				errdetail("CREATE/ALTER SUBSCRIPTION with origin = none and copy_data = on is not allowed when the publisher has subscribed same table."),
+				errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."));

1.
I think it might be better to use true/false here than on/off.
Just for consistency with another error message
(in function parse_subscription_options) and the description of this parameter
in the PG document.

If you agree with this, please also kindly consider the attached
"slight_modification.diff" file.
(This is a slight modification for the second patch, just replace "off" with
"false" in PG document.)

2.
How about replacing "origin = none" and "copy_data = on" in the message with
"%s"? I think this might be better for translation. Just like the following
error message in the function parse_subscription_options:
```
		if (opts->copy_data &&
			IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
			ereport(ERROR,
					(errcode(ERRCODE_SYNTAX_ERROR),
					 errmsg("%s and %s are mutually exclusive options",
							"connect = false", "copy_data = true/force")));
```

Regards,
Wang wei

  209. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-26T17:57:40Z

    On Sun, Jul 24, 2022 at 10:21 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 7/22/22 12:47 AM, Amit Kapila wrote:
> > On Fri, Jul 22, 2022 at 1:39 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> >> 1. I'm concerned by calling this "Bidirectional replication" in the docs
> >> that we are overstating the current capabilities. I think this is
> >> accentuated int he opening paragraph:
> >>
> >> ==snip==
> >>    Bidirectional replication is useful for creating a multi-master database
> >>    environment for replicating read/write operations performed by any of the
> >>    member nodes.
> >> ==snip==
> >>
> >> For one, we're not replicating reads, we're replicating writes. Amongst
> >> the writes, at this point we're only replicating DML. A reader could
> >> think that deploying can work for a full bidirectional solution.
> >>
> >> (Even if we're aspirationally calling this section "Bidirectional
> >> replication", that does make it sound like we're limited to two nodes,
> >> when we can support more than two).
> >>
> >
> > Right, I think the system can support N-Way replication.
>
> I did some more testing of the feature, i.e. doing 3-node and 4-node
> tests. While logical replication today can handle replicating between
> multiple nodes (N-way), the "origin = none" does require setting up
> subscribers between each of the nodes.
>
> For example, if I have 4 nodes A, B, C, D and I want to replicate the
> same table between all of them, I need to set up subscribers between all
> of them (A<=>B, A<=>C, A<=>D, B<=>C, B<=>D, C<=>D). However, each node
> can replicate between each other in a way that's convenient (vs. having
> to do something funky with partitions) so this is still a big step forward.
>
> This is a long way of saying that I do think it's fair to say we support
> "N-way" replication so long as you are set up in a mesh (vs. a ring,
> e.g. A=>B=>C=>D=>A).
>
> > Among the above "Replicating changes between primaries" sounds good to
> > me or simply "Replication between primaries". As this is a sub-section
> > on the Logical Replication page, I feel it is okay to not use Logical
> > in the title.
>
> Agreed, I think that's fine.
>
> >> At a minimum, I think we should reference the documentation we have in
> >> the logical replication section on conflicts. We may also want to advise
> >> that a user is responsible for designing their schemas in a way to
> >> minimize the risk of conflicts.
> >>
> >
> > This sounds reasonable to me.
> >
> > One more point about docs, it appears to be added as the last
> > sub-section on the Logical Replication page. Is there a reason for
> > doing so? I feel this should be third sub-section after describing
> > Publication and Subscription.
>
> When I first reviewed, I had not built the docs. Did so on this pass.
>
> I agree with the positioning argument, i.e. it should go after
> "Subscription" in the table of contents -- but it makes me question a
> couple of things:
>
> 1. The general ordering of the docs
> 2. How we describe that section (more on that in a sec)
> 3. If "row filters" should be part of "subscription" instead of its own
> section.
>
> If you look at the current order, "Quick setup" is the last section; one
> would think the "quick" portion goes first :) Given a lot of this is for
> the current docs, I may start a separate discussion on -docs for this part.
>
> For the time being, I agree it should be moved to the section after
> "Subscription".
>
> I think what this section describes is "Configuring Replication Between
> Nodes" as it covers a few different scenarios.
>
> I do think we need to iterate on these docs -- the examples with the
> commands are generally OK and easy to follow, but a few things I noticed:
>
> 1. The general description of the section needs work. We may want to
> refine the description of the use cases, and in the warning, link to
> instructions on how to take backups.

Modified

> 2. We put the "case not supported" in the middle, not at the end.

Modified

> 3. The "generic steps for adding a new node..." section uses a
> convention for steps that is not found in the docs. We also don't
> provide an example for this section, and this is the most complicated
> scenario to set up.

Modified

Thanks a lot for the suggestions, I have made the changes  for the
same in the v39 patch attached.

Regards,
Vignesh

  210. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-26T18:04:50Z

    On Tue, Jul 26, 2022 at 7:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for the patch v38-0002:
>
> ======
>
> <General> - terminology
>
> There seemed to be an inconsistent alternation of the terms
> "primaries" and "nodes"... For example "Setting replication between
> two primaries" versus "Adding a new node..." (instead of "Adding a new
> primary..."?). I have included suggested minor rewording changes in
> the review comments below, but please check in case I miss something.
> Because I suggested changes to some titles maybe you will also want to
> change the section ids too.
>
> ~~~
>
> 1. Commit message
>
> The documentation was recently modified to remove the term
> "bidirectional replication" and replace it all with "replication
> between primaries", so this commit message (and also the patch name
> itself) should be similarly modified.

Modified

> ~~~
>
> 2.
> +   <para>
> +    Replication between primaries is useful for creating a multi-master
> +    database environment for replicating write operations performed by any of
> +    the member nodes. The steps to create replication between primaries in
> +    various scenarios are given below. Note: User is responsible for designing
> +    their schemas in a way to minimize the risk of conflicts. See
> +    <xref linkend="logical-replication-conflicts"/> for the details of logical
> +    replication conflicts. The logical replication restrictions applies to
> +    the replication between primaries also. See
> +    <xref linkend="logical-replication-restrictions"/> for the details of
> +    logical replication restrictions.
> +   </para>
>
> 2a.
> "User" -> "The user"

Modified

> 2b.
> "The logical replication restrictions applies to..." --> "The logical
> replication restrictions apply to..."

Modified

> 2c.
> These are important notes. Instead of just being part of the text
> blurb, perhaps these should be rendered as SGML <note> (or put them
> both in a single <note> if you want)

Modified

> ~~~
>
> 3. Setting replication between two primaries
>
> +   <title>Setting replication between two primaries</title>
> +   <para>
> +    The following steps demonstrate how to setup replication between two
> +    primaries when there is no table data present on both nodes
> +    <literal>node1</literal> and <literal>node2</literal>:
> +   </para>
>
> SUGGESTED
> The following steps demonstrate how to set up replication between two
> primaries (node1 and node2) when there is no table data present on
> both nodes:.

Modified

> ~~~
>
> 4.
> +   <para>
> +    Now the replication setup between two primaries <literal>node1</literal>
> +    and <literal>node2</literal> is complete. Any incremental changes from
> +    <literal>node1</literal> will be replicated to <literal>node2</literal>,
> +    and any incremental changes from <literal>node2</literal> will be
> +    replicated to <literal>node1</literal>.
> +   </para>
>
> "between two primaries" -> "between primaries"

Modified

> ~~~
>
> 5. Adding a new node when there is no table data on any of the nodes
>
> SUGGESTION (title)
> Adding a new primary when there is no table data on any of the primaries

Modified

> ~~~
>
> 6.
> +   <para>
> +    The following steps demonstrate adding a new node <literal>node3</literal>
> +    to the existing <literal>node1</literal> and <literal>node2</literal> when
> +    there is no <literal>t1</literal> data on any of the nodes. This requires
>
> SUGGESTION
> The following steps demonstrate adding a new primary (node3) to the
> existing primaries (node1 and node2) when there is no t1 data on any
> of the nodes.

Modified

> ~~~
>
> 7. Adding a new node when table data is present on the existing nodes
>
> SUGGESTION (title)
> Adding a new primary when table data is present on the existing primaries

Modified

> ~~~
>
> 8.
> +    <para>
> +     The following steps demonstrate adding a new node <literal>node3</literal>
> +     which has no <literal>t1</literal> data to the existing
> +     <literal>node1</literal> and <literal>node2</literal> where
> +     <literal>t1</literal> data is present. This needs similar steps; the only
>
> SUGGESTION
> The following steps demonstrate adding a new primary (node3) that has
> no t1 data to the existing primaries (node1 and node2) where t1 data
> is present.

Modified

> ~~~
>
> 9. Adding a new node when table data is present on the new node
>
> SUGGESTION (title)
> Adding a new primary that has existing table data

Modified

> ~~~
>
> 10.
> +   <note>
> +    <para>
> +     Adding a new node when table data is present on the new node is not
> +     supported.
> +    </para>
> +   </note>
>
> SUGGESTION
> Adding a new primary that has existing table data is not supported.

Modified

> ~~~
>
> 11. Generic steps for adding a new node to an existing set of primaries
>
> SUGGESTION (title)
> Generic steps for adding a new primary to an existing set of primaries

Modified

Thanks for the comments, the v39 patch shared at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com

Regards,
Vignesh

  211. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-26T18:07:10Z

    On Tue, Jul 26, 2022 at 1:35 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Sun, Jul 24, 2022 1:28 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Added a  note for the same and referred it to the conflicts section.
> >
> > Thanks for the comments, the attached v38 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch. A comment on the test in 0001 patch.
>
> +# Alter subscription ... refresh publication should fail when a new table is
> +# subscribing data from a different publication should fail
> +($result, $stdout, $stderr) = $node_A->psql(
> +       'postgres', "
> +        ALTER SUBSCRIPTION tap_sub_A2 REFRESH PUBLICATION");
> +like(
> +       $stderr,
> +       qr/ERROR: ( [A-Z0-9]+:)? could not replicate table "public.tab_new"/,
> +       "Create subscription with origin and copy_data having replicated table in publisher"
> +);
>
> The comment says "should fail" twice, the latter one can be removed.

Modified

> Besides, "Create subscription with origin and copy_data" should be changed to
> "Alter subscription with origin and copy_data" I think.

Modified to "Refresh publication"

Thanks for the comments, the v39 patch shared at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com

Regards,
Vignesh

  212. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-26T18:08:04Z

    On Tue, Jul 26, 2022 at 2:12 PM wangw.fnst@fujitsu.com
<wangw.fnst@fujitsu.com> wrote:
>
> On Sun, Jul 24, 2022 1:28 AM vignesh C <vignesh21@gmail.com> wrote:
> > Added a  note for the same and referred it to the conflicts section.
> >
> > Thanks for the comments, the attached v38 patch has the changes for the same.
>
> Thanks for your patches.
>
> Two slight comments on the below message in the 0001 patch:
>
> The error message in the function check_pub_table_subscribed().
> +               ereport(ERROR,
> +                               errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> +                               errmsg("could not replicate table \"%s.%s\"",
> +                                          nspname, relname),
> +                               errdetail("CREATE/ALTER SUBSCRIPTION with origin = none and copy_data = on is not allowed when the publisher has subscribed same table."),
> +                               errhint("Use CREATE/ALTER SUBSCRIPTION with copy_data = off/force."));
>
> 1.
> I think it might be better to use true/false here than on/off.
> Just for consistency with another error message
> (in function parse_subscription_options) and the description of this parameter
> in the PG document.

Modified

> If you agree with this, please also kindly consider the attached
> "slight_modification.diff" file.
> (This is a slight modification for the second patch, just replace "off" with
> "false" in PG document.)

I have updated the documentation similarly

> 2.
> How about replacing "origin = none" and "copy_data = on" in the message with
> "%s"? I think this might be better for translation. Just like the following
> error message in the function parse_subscription_options:
> ```
>                 if (opts->copy_data &&
>                         IsSet(opts->specified_opts, SUBOPT_COPY_DATA))
>                         ereport(ERROR,
>                                         (errcode(ERRCODE_SYNTAX_ERROR),
>                                          errmsg("%s and %s are mutually exclusive options",
>                                                         "connect = false", "copy_data = true/force")));
> ```

Modified

Thanks for the comments, the v39 patch shared at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com

Regards,
Vignesh

  213. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-26T18:13:53Z

    On Mon, Jul 25, 2022 at 12:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Firstly, I have some (case-sensitivity) questions about the previous
> patch which was already pushed [1].
>
> Q1. create_subscription docs
>
> I did not understand why the docs refer to slot_name = NONE, yet the
> newly added option says origin = none/any. I think that saying origin
> = NONE/ANY would be more consistent with the existing usage of NONE in
> this documentation.
>
> ~~~
>
> Q2. parse_subscription_options
>
> Similarly, in the code (parse_subscription_options), I did not
> understand why the checks for special name values are implemented
> differently:
>
> The new 'origin' code is using pg_strcmpcase to check special values
> (none/any), and the old 'slot_name' code uses case-sensitive strcmp to
> check the special value (none).
>
> FWIW, here I thought the new origin code is the correct one.
>
> ======
>
> Now, here are some review comments for the patch v38-0001:
>
> 1. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> @@ -1781,6 +1858,121 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
>   table_close(rel, RowExclusiveLock);
>  }
>
> +/*
> + * Check and throw an error if the publisher has subscribed to the same table
> + * from some other publisher. This check is required only if copydata is ON and
> + * the origin is local for CREATE SUBSCRIPTION and
> + * ALTER SUBSCRIPTION ... REFRESH statements to avoid replicating remote data
> + * from the publisher.
> + *
> + * This check need not be performed on the tables that are already added as
> + * incremental sync for such tables will happen through WAL and the origin of
> + * the data can be identified from the WAL records.
> + *
> + * subrel_local_oids contains the list of relation oids that are already
> + * present on the subscriber.
> + */
> +static void
> +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
> +    CopyData copydata, char *origin,
> +    Oid *subrel_local_oids, int subrel_count)
>
> 1a.
> "copydata is ON" --> "copy_data = on" (because the comment is talking
> about the CREATE/ALTER statements, so it seemed a bit confusing to
> refer to the copydata function param instead of the copy_data
> subscription parameter)

Modified

> 1b.
> "the origin is local" ?? But, "local" was the old special name value.
> Now it is "none", so I think this part needs minor rewording.

Modified

>
> ~~~
>
> 2.
>
> + if (copydata != COPY_DATA_ON || !origin ||
> + (pg_strcasecmp(origin, "none") != 0))
> + return;
>
> Should this be using the constant LOGICALREP_ORIGIN_NONE?

Modified

> ~~~
>
> 3.
>
> + /*
> + * Throw an error if the publisher has subscribed to the same table
> + * from some other publisher. We cannot differentiate between the
> + * origin and non-origin data that is present in the HEAP during the
> + * initial sync. Identification of non-origin data can be done only
> + * from the WAL by using the origin id.
> + *
> + * XXX: For simplicity, we don't check whether the table has any data
> + * or not. If the table doesn't have any data then we don't need to
> + * distinguish between local and non-local data so we can avoid
> + * throwing an error in that case.
> + */
>
> 3a.
> When the special origin value changed from "local" to "none" this
> comment's first part seems to have got a bit lost in translation.

Modified

> SUGGESTION:
> Throw an error if the publisher has subscribed to the same table from
> some other publisher. We cannot know the origin of data during the
> initial sync. Data origins can be found only from the WAL by looking
> at the origin id.

Modified

> 3b.
> I think referring to "local and non-local" data in the XXX part of
> this comment also needs some minor rewording now that "local" is not a
> special origin name anymore.

Modified

Thanks for the comments, the v39 patch shared at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2POATc_jwQ-8MBJgGCVZGdUNhnTv8zkBuGzLaY03dM%3DA%40mail.gmail.com

Regards,
Vignesh

  214. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-07-27T02:53:19Z

    On Tue, Jul 26, 2022 at 10:23 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jul 26, 2022 at 2:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > I am not really sure how much we gain by maintaining consistency with
> > slot_name because if due to this we have to change the error messages
> > as well then it can create an inconsistency with reserved origin
> > names. Consider message: DETAIL:  Origin names "any", "none", and
> > names starting with "pg_" are reserved. Now, if we change this to
> > "ANY", "NONE" in the above message, it will look a bit odd as "pg_"
> > starts with lower case letters.
> >
>
> Sure, the message looks a bit odd with the quotes like you wrote
> above, but I would not suggest to change it that way - I was thinking
> more like below (which is similar to the style the slot_name messages
> use)
>
> CURRENT
> DETAIL:  Origin names "any", "none", and names starting with "pg_" are reserved.
>
> SUGGESTED
> DETAIL:  Origin names ANY, NONE, and names starting with "pg_" are reserved.
>

I see your point but not sure if that is an improvement over the
current one, so, let's wait and see if we get some other votes in
favor of your suggestion.

-- 
With Regards,
Amit Kapila.

  215. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-28T05:58:35Z

    Hi Vignesh.

FYI the v39* patch fails to apply [1]. Can you please rebase it?


[1]
=== Applying patches on top of PostgreSQL commit ID
5f858dd3bebd1f3845aef2bff7f4345bfb7b74b3 ===
=== applying patch
./v39-0001-Check-and-throw-an-error-if-publication-tables-w.patch
patching file doc/src/sgml/ref/alter_subscription.sgml
patching file doc/src/sgml/ref/create_subscription.sgml
patching file src/backend/commands/subscriptioncmds.c
Hunk #10 FAILED at 886.
1 out of 14 hunks FAILED -- saving rejects to file
src/backend/commands/subscriptioncmds.c.rej
patching file src/test/regress/expected/subscription.out
patching file src/test/regress/sql/subscription.sql
patching file src/test/subscription/t/030_origin.pl
patching file src/tools/pgindent/typedefs.list

------
[1] http://cfbot.cputube.org/patch_38_3610.log

Kind Regards,
Peter Smith.
Fujitsu Australia

  216. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-28T10:45:42Z

    On Thu, Jul 28, 2022 at 11:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh.
>
> FYI the v39* patch fails to apply [1]. Can you please rebase it?
>
>
> [1]
> === Applying patches on top of PostgreSQL commit ID
> 5f858dd3bebd1f3845aef2bff7f4345bfb7b74b3 ===
> === applying patch
> ./v39-0001-Check-and-throw-an-error-if-publication-tables-w.patch
> patching file doc/src/sgml/ref/alter_subscription.sgml
> patching file doc/src/sgml/ref/create_subscription.sgml
> patching file src/backend/commands/subscriptioncmds.c
> Hunk #10 FAILED at 886.
> 1 out of 14 hunks FAILED -- saving rejects to file
> src/backend/commands/subscriptioncmds.c.rej
> patching file src/test/regress/expected/subscription.out
> patching file src/test/regress/sql/subscription.sql
> patching file src/test/subscription/t/030_origin.pl
> patching file src/tools/pgindent/typedefs.list

Thanks for reporting this, I will post an updated version for this soon.

Regards,
Vignesh

  217. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-28T14:03:31Z

    On Thu, Jul 28, 2022 at 11:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh.
>
> FYI the v39* patch fails to apply [1]. Can you please rebase it?
>
>
> [1]
> === Applying patches on top of PostgreSQL commit ID
> 5f858dd3bebd1f3845aef2bff7f4345bfb7b74b3 ===
> === applying patch
> ./v39-0001-Check-and-throw-an-error-if-publication-tables-w.patch
> patching file doc/src/sgml/ref/alter_subscription.sgml
> patching file doc/src/sgml/ref/create_subscription.sgml
> patching file src/backend/commands/subscriptioncmds.c
> Hunk #10 FAILED at 886.
> 1 out of 14 hunks FAILED -- saving rejects to file
> src/backend/commands/subscriptioncmds.c.rej
> patching file src/test/regress/expected/subscription.out
> patching file src/test/regress/sql/subscription.sql
> patching file src/test/subscription/t/030_origin.pl
> patching file src/tools/pgindent/typedefs.list
>
> ------

Please find the v40 patch attached which is rebased on top of head.

Regards,
Vignesh

  218. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-07-29T03:01:30Z

    Here are some comments for the patch v40-0001:

======

1. Commit message

It might be better to always use 'copy_data = true' in favour of
'copy_data = on' just for consistency with all the docs and the error
messages.

======

2. doc/src/sgml/ref/create_subscription.sgml

@@ -386,6 +401,15 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    can have non-existent publications.
   </para>

+  <para>
+   If the subscription is created with <literal>origin = NONE</literal> and
+   <literal>copy_data = true</literal>, it will check if the publisher has
+   subscribed to the same table from other publishers and, if so, throw an
+   error to prevent possible non-local data from being copied. The user can
+   override this check and continue with the copy operation by specifying
+   <literal>copy_data = force</literal>.
+  </para>

2a.
It is interesting that you changed the note to say origin = NONE.
Personally, I prefer it written as you did, but I think maybe this
change does not belong in this patch. The suggestion for changing from
"none" to NONE is being discussed elsewhere in this thread and
probably all such changes should be done together (if at all) as a
separate patch. Until then I think this patch 0001 should just stay
consistent with whatever is already pushed on HEAD.

2b.
"possible no-local data". Maybe the terminology "local/non-local" is a
hangover from back when the subscription parameter was called local
instead of origin. I'm not sure if you want to change this or not, and
anyway I didn't have any better suggestions – so this comment is just
to bring it to your attention.

======

3. src/backend/commands/subscriptioncmds.c - DefGetCopyData

+ /*
+ * The set of strings accepted here should match up with the
+ * grammar's opt_boolean_or_string production.
+ */
+ if (pg_strcasecmp(sval, "false") == 0 ||
+ pg_strcasecmp(sval, "off") == 0)
+ return COPY_DATA_OFF;
+ if (pg_strcasecmp(sval, "true") == 0 ||
+ pg_strcasecmp(sval, "on") == 0)
+ return COPY_DATA_ON;
+ if (pg_strcasecmp(sval, "force") == 0)
+ return COPY_DATA_FORCE;

I understand the intention of the comment, but it is not strictly
correct to say "should match up" because "force" is a new value.
Perhaps the comment should be as suggested below.

SUGGESTION
The set of strings accepted here must include all those accepted by
the grammar's opt_boolean_or_string production.

~~~

4. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

@@ -1781,6 +1858,122 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
  table_close(rel, RowExclusiveLock);
 }

+/*
+ * Check and throw an error if the publisher has subscribed to the same table
+ * from some other publisher. This check is required only if "copy_data = on"
+ * and "origin = NONE" for CREATE SUBSCRIPTION and
+ * ALTER SUBSCRIPTION ... REFRESH statements to avoid the publisher from
+ * replicating data that has an origin.
+ *
+ * This check need not be performed on the tables that are already added as
+ * incremental sync for such tables will happen through WAL and the origin of
+ * the data can be identified from the WAL records.
+ *
+ * subrel_local_oids contains the list of relation oids that are already
+ * present on the subscriber.
+ */
+static void
+check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
+    CopyData copydata, char *origin,
+    Oid *subrel_local_oids, int subrel_count)

4a.
"copy_data = on" -> "copy_data = true" (for consistency with the docs
and the error messages)

4b.
The same NONE/none review comment from #2a applies here too. Probably
it should be written as none for now unless/until *everything* changes
to NONE.

4c.
"to avoid the publisher from replicating data that has an origin." ->
"to avoid replicating data that has an origin."

4d.
+ * This check need not be performed on the tables that are already added as
+ * incremental sync for such tables will happen through WAL and the origin of
+ * the data can be identified from the WAL records.

SUGGESTION (maybe?)
This check need not be performed on the tables that are already added
because incremental sync for those tables will happen through WAL and
the origin of the data can be identified from the WAL records.

======

5. src/test/subscription/t/030_origin.pl

+ "Refresh publication when the publisher has subscribed for the new table"

SUGGESTION (Just to mention origin = none somehow. Maybe you can
reword it better than this)
Refresh publication when the publisher has subscribed for the new
table, but the subscriber-side wants origin=none

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  219. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-07-29T05:21:40Z

    On Fri, Jul 29, 2022 at 8:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some comments for the patch v40-0001:
>
> ======
>
> 1. Commit message
>
> It might be better to always use 'copy_data = true' in favour of
> 'copy_data = on' just for consistency with all the docs and the error
> messages.
>
> ======

Modified

> 2. doc/src/sgml/ref/create_subscription.sgml
>
> @@ -386,6 +401,15 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     can have non-existent publications.
>    </para>
>
> +  <para>
> +   If the subscription is created with <literal>origin = NONE</literal> and
> +   <literal>copy_data = true</literal>, it will check if the publisher has
> +   subscribed to the same table from other publishers and, if so, throw an
> +   error to prevent possible non-local data from being copied. The user can
> +   override this check and continue with the copy operation by specifying
> +   <literal>copy_data = force</literal>.
> +  </para>
>
> 2a.
> It is interesting that you changed the note to say origin = NONE.
> Personally, I prefer it written as you did, but I think maybe this
> change does not belong in this patch. The suggestion for changing from
> "none" to NONE is being discussed elsewhere in this thread and
> probably all such changes should be done together (if at all) as a
> separate patch. Until then I think this patch 0001 should just stay
> consistent with whatever is already pushed on HEAD.

Modified

> 2b.
> "possible no-local data". Maybe the terminology "local/non-local" is a
> hangover from back when the subscription parameter was called local
> instead of origin. I'm not sure if you want to change this or not, and
> anyway I didn't have any better suggestions – so this comment is just
> to bring it to your attention.
>
> ======

Modified

> 3. src/backend/commands/subscriptioncmds.c - DefGetCopyData
>
> + /*
> + * The set of strings accepted here should match up with the
> + * grammar's opt_boolean_or_string production.
> + */
> + if (pg_strcasecmp(sval, "false") == 0 ||
> + pg_strcasecmp(sval, "off") == 0)
> + return COPY_DATA_OFF;
> + if (pg_strcasecmp(sval, "true") == 0 ||
> + pg_strcasecmp(sval, "on") == 0)
> + return COPY_DATA_ON;
> + if (pg_strcasecmp(sval, "force") == 0)
> + return COPY_DATA_FORCE;
>
> I understand the intention of the comment, but it is not strictly
> correct to say "should match up" because "force" is a new value.
> Perhaps the comment should be as suggested below.
>
> SUGGESTION
> The set of strings accepted here must include all those accepted by
> the grammar's opt_boolean_or_string production.
>
> ~~

Modified

>
> 4. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> @@ -1781,6 +1858,122 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
>   table_close(rel, RowExclusiveLock);
>  }
>
> +/*
> + * Check and throw an error if the publisher has subscribed to the same table
> + * from some other publisher. This check is required only if "copy_data = on"
> + * and "origin = NONE" for CREATE SUBSCRIPTION and
> + * ALTER SUBSCRIPTION ... REFRESH statements to avoid the publisher from
> + * replicating data that has an origin.
> + *
> + * This check need not be performed on the tables that are already added as
> + * incremental sync for such tables will happen through WAL and the origin of
> + * the data can be identified from the WAL records.
> + *
> + * subrel_local_oids contains the list of relation oids that are already
> + * present on the subscriber.
> + */
> +static void
> +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
> +    CopyData copydata, char *origin,
> +    Oid *subrel_local_oids, int subrel_count)
>
> 4a.
> "copy_data = on" -> "copy_data = true" (for consistency with the docs
> and the error messages)

Modified

> 4b.
> The same NONE/none review comment from #2a applies here too. Probably
> it should be written as none for now unless/until *everything* changes
> to NONE.

Modified

> 4c.
> "to avoid the publisher from replicating data that has an origin." ->
> "to avoid replicating data that has an origin."

Modified

> 4d.
> + * This check need not be performed on the tables that are already added as
> + * incremental sync for such tables will happen through WAL and the origin of
> + * the data can be identified from the WAL records.
>
> SUGGESTION (maybe?)
> This check need not be performed on the tables that are already added
> because incremental sync for those tables will happen through WAL and
> the origin of the data can be identified from the WAL records.
>
> ======

Modified

>
> 5. src/test/subscription/t/030_origin.pl
>
> + "Refresh publication when the publisher has subscribed for the new table"
>
> SUGGESTION (Just to mention origin = none somehow. Maybe you can
> reword it better than this)
> Refresh publication when the publisher has subscribed for the new
> table, but the subscriber-side wants origin=none

Modified

Thanks for the comments, the attached v41 patch has the changes for the same.

Regards,
Vignesh

  220. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-08-01T05:27:30Z

    On Fri, Jul 29, 2022 1:22 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> 
> Thanks for the comments, the attached v41 patch has the changes for the
> same.
> 

Thanks for updating the patch.

I wonder in the case that the publisher uses PG15 (or before), subscriber uses
PG16, should we have this check (check if publication tables were also
subscribing from other publishers)? In this case, even if origin=none is
specified, it doesn't work because the publisher doesn't filter the origin. So
maybe we don't need the check for initial sync. Thoughts?

Regards,
Shi yu

  221. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-01T08:02:21Z

    On Mon, Aug 1, 2022 at 3:27 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Fri, Jul 29, 2022 1:22 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> >
> > Thanks for the comments, the attached v41 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch.
>
> I wonder in the case that the publisher uses PG15 (or before), subscriber uses
> PG16, should we have this check (check if publication tables were also
> subscribing from other publishers)? In this case, even if origin=none is
> specified, it doesn't work because the publisher doesn't filter the origin. So
> maybe we don't need the check for initial sync. Thoughts?
>

IIUC for the scenario you've described (subscription origin=none and
publisher < PG16) the subscriber can end up getting extra data they
did not want, right?

So instead of just "don't need the check", maybe this combination
should throw ERROR, or at least a log a WARNING?

------
KInd Regards,
Peter Smith.
Fujitsu Australia

  222. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-01T08:51:56Z

    On Mon, Aug 1, 2022 at 1:32 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Aug 1, 2022 at 3:27 PM shiy.fnst@fujitsu.com
> <shiy.fnst@fujitsu.com> wrote:
> >
> > On Fri, Jul 29, 2022 1:22 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > >
> > > Thanks for the comments, the attached v41 patch has the changes for the
> > > same.
> > >
> >
> > Thanks for updating the patch.
> >
> > I wonder in the case that the publisher uses PG15 (or before), subscriber uses
> > PG16, should we have this check (check if publication tables were also
> > subscribing from other publishers)? In this case, even if origin=none is
> > specified, it doesn't work because the publisher doesn't filter the origin. So
> > maybe we don't need the check for initial sync. Thoughts?
> >
>
> IIUC for the scenario you've described (subscription origin=none and
> publisher < PG16) the subscriber can end up getting extra data they
> did not want, right?
>

Yes, because publishers won't have 'filtering based on origin' functionality.

> So instead of just "don't need the check", maybe this combination
> should throw ERROR, or at least a log a WARNING?
>

I am not sure if doing anything (ERROR or WARNING) would make sense
because anyway later during replication there won't be any filtering.

-- 
With Regards,
Amit Kapila.

  223. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-02T02:01:15Z

    On Mon, Aug 1, 2022 at 6:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Aug 1, 2022 at 1:32 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Mon, Aug 1, 2022 at 3:27 PM shiy.fnst@fujitsu.com
> > <shiy.fnst@fujitsu.com> wrote:
> > >
> > > On Fri, Jul 29, 2022 1:22 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > >
> > > > Thanks for the comments, the attached v41 patch has the changes for the
> > > > same.
> > > >
> > >
> > > Thanks for updating the patch.
> > >
> > > I wonder in the case that the publisher uses PG15 (or before), subscriber uses
> > > PG16, should we have this check (check if publication tables were also
> > > subscribing from other publishers)? In this case, even if origin=none is
> > > specified, it doesn't work because the publisher doesn't filter the origin. So
> > > maybe we don't need the check for initial sync. Thoughts?
> > >
> >
> > IIUC for the scenario you've described (subscription origin=none and
> > publisher < PG16) the subscriber can end up getting extra data they
> > did not want, right?
> >
>
> Yes, because publishers won't have 'filtering based on origin' functionality.
>
> > So instead of just "don't need the check", maybe this combination
> > should throw ERROR, or at least a log a WARNING?
> >
>
> I am not sure if doing anything (ERROR or WARNING) would make sense
> because anyway later during replication there won't be any filtering.
>

I was suggesting stopping that replication from happening at all. If
the user specifically asked for 'origin=none' but the publisher could
not filter that (because < PG16) then I imagined some logic that would
just disable the subscription up-front. Isn't it preferable for the
subscriber to get no data at all then to get data the user
specifically said they did NOT want to get?

e.g. pseudo-code for the worker code something like below:

if (origin != ANY and publisher.server_version < PG16)
{
    set subscription.option.disable_on_error = true;
    throw ERROR ("publisher does not support origin=none - disabling
the subscription");
}

------
Kind Regards,
Peter Smith.
Fujitsu Australia.

  224. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-08-02T09:31:20Z

    On Fri, Jul 29, 2022 1:22 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> On Fri, Jul 29, 2022 at 8:31 AM Peter Smith <smithpb2250@gmail.com>
> wrote:
> >
> 
> Thanks for the comments, the attached v41 patch has the changes for the
> same.
> 

Thanks for updating the patch.

A comment for 0002 patch.

In the example in section 31.11.4 (Generic steps for adding a new primary to an
existing set of primaries), I think there should be a Step-6, corresponding to
the steps mentioned before. And part of Step-5 should actually be part of
Step-6.

Regards,
Shi yu

  225. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-02T11:59:36Z

    On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jul 26, 2022 at 7:13 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> >
> > Thanks for the example. I agree that it is fairly simple to reproduce.
> >
> > I understand that "copy_data = force" is meant to protect a user from
> > hurting themself. I'm not convinced that this is the best way to do so.
> >
> > For example today I can subscribe to multiple publications that write to
> > the same table. If I have a primary key on that table, and two of the
> > subscriptions try to write an identical ID, we conflict. We don't have
> > any special flags or modes to guard against that from happening, though
> > we do have documentation on conflicts and managing them.
> >
> > AFAICT the same issue with "copy_data" also exists in the above scenario
> > too, even without the "origin" attribute.
> >
>
> That's true but there is no parameter like origin = NONE which
> indicates that constraint violations or duplicate data problems won't
> occur due to replication. In the current case, I think the situation
> is different because a user has specifically asked not to replicate
> any remote data by specifying origin = NONE, which should be dealt
> differently. Note that current users or their setup won't see any
> difference/change unless they specify the new parameter origin as
> NONE.
>

Let me try to summarize the discussion so that it is easier for others
to follow. The work in this thread is to avoid loops, and duplicate
data in logical replication when the operations happened on the same
table in multiple nodes. It has been explained in email [1] with an
example of how a logical replication setup can lead to duplicate or
inconsistent data.

The idea to solve this problem is that we don't replicate data that is
not generated locally which we can normally identify based on origin
of data in WAL. The commit 366283961a achieves that for replication
but still the problem can happen during initial sync which is
performed internally via copy. We can't differentiate the data in heap
based on origin. So, we decided to prohibit the subscription
operations that can perform initial sync (ex. Create Subscription,
Alter Subscription ... Refresh) by detecting that the publisher has
subscribed to the same table from some other publisher.

To prohibit the subscription operations, the currently proposed patch
throws an error.  Then, it also provides a new copy_data option
'force' under which the user will still be able to perform the
operation. This could be useful when the user intentionally wants to
replicate the initial data even if it contains data from multiple
nodes (for example, when in a multi-node setup, one decides to get the
initial data from just one node and then allow replication of data to
proceed from each of respective nodes).

The other alternative discussed was to just give a warning for
subscription operations and probably document the steps for users to
avoid it. But the problem with that is once the user sees this
warning, it won't be able to do anything except recreate the setup, so
why not give an error in the first place?

Thoughts?

[1] - https://www.postgresql.org/message-id/CALDaNm1eJr6qXT9esVPzgc5Qvy4uMhV4kCCTSmxARKjf%2BMwcnw%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  226. Re: Handle infinite recursion in logical replication setup

    Masahiko Sawada <sawada.mshk@gmail.com> — 2022-08-04T13:00:53Z

    On Tue, Aug 2, 2022 at 8:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jul 26, 2022 at 7:13 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> > >
> > > Thanks for the example. I agree that it is fairly simple to reproduce.
> > >
> > > I understand that "copy_data = force" is meant to protect a user from
> > > hurting themself. I'm not convinced that this is the best way to do so.
> > >
> > > For example today I can subscribe to multiple publications that write to
> > > the same table. If I have a primary key on that table, and two of the
> > > subscriptions try to write an identical ID, we conflict. We don't have
> > > any special flags or modes to guard against that from happening, though
> > > we do have documentation on conflicts and managing them.
> > >
> > > AFAICT the same issue with "copy_data" also exists in the above scenario
> > > too, even without the "origin" attribute.
> > >
> >
> > That's true but there is no parameter like origin = NONE which
> > indicates that constraint violations or duplicate data problems won't
> > occur due to replication. In the current case, I think the situation
> > is different because a user has specifically asked not to replicate
> > any remote data by specifying origin = NONE, which should be dealt
> > differently. Note that current users or their setup won't see any
> > difference/change unless they specify the new parameter origin as
> > NONE.
> >
>
> Let me try to summarize the discussion so that it is easier for others
> to follow. The work in this thread is to avoid loops, and duplicate
> data in logical replication when the operations happened on the same
> table in multiple nodes. It has been explained in email [1] with an
> example of how a logical replication setup can lead to duplicate or
> inconsistent data.
>
> The idea to solve this problem is that we don't replicate data that is
> not generated locally which we can normally identify based on origin
> of data in WAL. The commit 366283961a achieves that for replication
> but still the problem can happen during initial sync which is
> performed internally via copy. We can't differentiate the data in heap
> based on origin. So, we decided to prohibit the subscription
> operations that can perform initial sync (ex. Create Subscription,
> Alter Subscription ... Refresh) by detecting that the publisher has
> subscribed to the same table from some other publisher.
>
> To prohibit the subscription operations, the currently proposed patch
> throws an error.  Then, it also provides a new copy_data option
> 'force' under which the user will still be able to perform the
> operation. This could be useful when the user intentionally wants to
> replicate the initial data even if it contains data from multiple
> nodes (for example, when in a multi-node setup, one decides to get the
> initial data from just one node and then allow replication of data to
> proceed from each of respective nodes).
>
> The other alternative discussed was to just give a warning for
> subscription operations and probably document the steps for users to
> avoid it. But the problem with that is once the user sees this
> warning, it won't be able to do anything except recreate the setup, so
> why not give an error in the first place?
>
> Thoughts?

Thank you for the summary!

I understand that this feature could help some cases, but I'm really
not sure adding new value 'force' for them is worthwhile, and
concerned it could reduce the usability.

IIUC this feature would work only when origin = 'none' and copy_data =
'on', and the purpose is to prevent the data from being
duplicated/conflicted by the initial table sync. But there are cases
where duplication/conflict doesn't happen even if origin = 'none' and
copy_data = 'on'. For instance, the table on the publisher might be
empty. Also, even with origin = 'any', copy_data = 'on', there is a
possibility of data duplication/conflict. Why do we need to address
only the case where origin = 'none'? I think that using origin =
'none' doesn't necessarily mean using bi-directional (or N-way)
replication. Even when using uni-directional logical replication with
two nodes, they may use origin = 'none'. Therefore, it seems to me
that this feature works only for a narrow situation and has false
positives.

Since it has been the user's responsibility not to try to make the
data inconsistent by the initial table sync, I think that it might be
sufficient if we note the risk in the documentation.

Regards,

--
Masahiko Sawada
EDB:  https://www.enterprisedb.com/

  227. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-05T04:10:02Z

    On Thu, Aug 4, 2022 at 6:31 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Aug 2, 2022 at 8:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Let me try to summarize the discussion so that it is easier for others
> > to follow. The work in this thread is to avoid loops, and duplicate
> > data in logical replication when the operations happened on the same
> > table in multiple nodes. It has been explained in email [1] with an
> > example of how a logical replication setup can lead to duplicate or
> > inconsistent data.
> >
> > The idea to solve this problem is that we don't replicate data that is
> > not generated locally which we can normally identify based on origin
> > of data in WAL. The commit 366283961a achieves that for replication
> > but still the problem can happen during initial sync which is
> > performed internally via copy. We can't differentiate the data in heap
> > based on origin. So, we decided to prohibit the subscription
> > operations that can perform initial sync (ex. Create Subscription,
> > Alter Subscription ... Refresh) by detecting that the publisher has
> > subscribed to the same table from some other publisher.
> >
> > To prohibit the subscription operations, the currently proposed patch
> > throws an error.  Then, it also provides a new copy_data option
> > 'force' under which the user will still be able to perform the
> > operation. This could be useful when the user intentionally wants to
> > replicate the initial data even if it contains data from multiple
> > nodes (for example, when in a multi-node setup, one decides to get the
> > initial data from just one node and then allow replication of data to
> > proceed from each of respective nodes).
> >
> > The other alternative discussed was to just give a warning for
> > subscription operations and probably document the steps for users to
> > avoid it. But the problem with that is once the user sees this
> > warning, it won't be able to do anything except recreate the setup, so
> > why not give an error in the first place?
> >
> > Thoughts?
>
> Thank you for the summary!
>
> I understand that this feature could help some cases, but I'm really
> not sure adding new value 'force' for them is worthwhile, and
> concerned it could reduce the usability.
>
> IIUC this feature would work only when origin = 'none' and copy_data =
> 'on', and the purpose is to prevent the data from being
> duplicated/conflicted by the initial table sync. But there are cases
> where duplication/conflict doesn't happen even if origin = 'none' and
> copy_data = 'on'. For instance, the table on the publisher might be
> empty.
>

Right, but if we want we can check the tables on publishers to ensure
that. Now, another case could be where the corresponding subscription
was disabled on publisher during create subscription but got enabled
just before copy, we can even catch that situation as we are doing for
column lists in fetch_remote_table_info(). Are there other cases of
false positives you can think of?  I see your point that we can
document that users should be careful with certain configurations to
avoid data inconsistency but not able completely convince myself about
the same. I think the main thing to decide here is how much we want to
ask users to be careful by referring them to docs. Now, if you are not
convinced with giving an ERROR here then we can probably show a
WARNING (that we might copy data for multiple origins during initial
sync in spite of the user having specified origin as NONE)?

>
 Also, even with origin = 'any', copy_data = 'on', there is a
> possibility of data duplication/conflict. Why do we need to address
> only the case where origin = 'none'?
>

Because the user has specifically asked not to replicate any remote
data by specifying origin = NONE, which should be dealt with
differently whereas 'any' doesn't have such a requirement. Now,
tomorrow, if we want to support replication based on specific origin
names say origin = 'node-1' then also we won't be able to identify the
data during initial sync but I think 'none' is a special case where
giving some intimation to user won't be a bad idea especially because
we can identify the same.

> I think that using origin =
> 'none' doesn't necessarily mean using bi-directional (or N-way)
> replication. Even when using uni-directional logical replication with
> two nodes, they may use origin = 'none'.
>

It is possible but still, I think it is a must for bi-directional (or
N-way) replication, otherwise, there is a risk of loops.

-- 
With Regards,
Amit Kapila.

  228. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-08-08T04:51:24Z

    On Fri, Jul 29, 2022 at 10:51 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, Jul 29, 2022 at 8:31 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are some comments for the patch v40-0001:
> >
> > ======
> >
> > 1. Commit message
> >
> > It might be better to always use 'copy_data = true' in favour of
> > 'copy_data = on' just for consistency with all the docs and the error
> > messages.
> >
> > ======
>
> Modified
>
> > 2. doc/src/sgml/ref/create_subscription.sgml
> >
> > @@ -386,6 +401,15 @@ CREATE SUBSCRIPTION <replaceable
> > class="parameter">subscription_name</replaceabl
> >     can have non-existent publications.
> >    </para>
> >
> > +  <para>
> > +   If the subscription is created with <literal>origin = NONE</literal> and
> > +   <literal>copy_data = true</literal>, it will check if the publisher has
> > +   subscribed to the same table from other publishers and, if so, throw an
> > +   error to prevent possible non-local data from being copied. The user can
> > +   override this check and continue with the copy operation by specifying
> > +   <literal>copy_data = force</literal>.
> > +  </para>
> >
> > 2a.
> > It is interesting that you changed the note to say origin = NONE.
> > Personally, I prefer it written as you did, but I think maybe this
> > change does not belong in this patch. The suggestion for changing from
> > "none" to NONE is being discussed elsewhere in this thread and
> > probably all such changes should be done together (if at all) as a
> > separate patch. Until then I think this patch 0001 should just stay
> > consistent with whatever is already pushed on HEAD.
>
> Modified
>
> > 2b.
> > "possible no-local data". Maybe the terminology "local/non-local" is a
> > hangover from back when the subscription parameter was called local
> > instead of origin. I'm not sure if you want to change this or not, and
> > anyway I didn't have any better suggestions – so this comment is just
> > to bring it to your attention.
> >
> > ======
>
> Modified
>
> > 3. src/backend/commands/subscriptioncmds.c - DefGetCopyData
> >
> > + /*
> > + * The set of strings accepted here should match up with the
> > + * grammar's opt_boolean_or_string production.
> > + */
> > + if (pg_strcasecmp(sval, "false") == 0 ||
> > + pg_strcasecmp(sval, "off") == 0)
> > + return COPY_DATA_OFF;
> > + if (pg_strcasecmp(sval, "true") == 0 ||
> > + pg_strcasecmp(sval, "on") == 0)
> > + return COPY_DATA_ON;
> > + if (pg_strcasecmp(sval, "force") == 0)
> > + return COPY_DATA_FORCE;
> >
> > I understand the intention of the comment, but it is not strictly
> > correct to say "should match up" because "force" is a new value.
> > Perhaps the comment should be as suggested below.
> >
> > SUGGESTION
> > The set of strings accepted here must include all those accepted by
> > the grammar's opt_boolean_or_string production.
> >
> > ~~
>
> Modified
>
> >
> > 4. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
> >
> > @@ -1781,6 +1858,122 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
> >   table_close(rel, RowExclusiveLock);
> >  }
> >
> > +/*
> > + * Check and throw an error if the publisher has subscribed to the same table
> > + * from some other publisher. This check is required only if "copy_data = on"
> > + * and "origin = NONE" for CREATE SUBSCRIPTION and
> > + * ALTER SUBSCRIPTION ... REFRESH statements to avoid the publisher from
> > + * replicating data that has an origin.
> > + *
> > + * This check need not be performed on the tables that are already added as
> > + * incremental sync for such tables will happen through WAL and the origin of
> > + * the data can be identified from the WAL records.
> > + *
> > + * subrel_local_oids contains the list of relation oids that are already
> > + * present on the subscriber.
> > + */
> > +static void
> > +check_pub_table_subscribed(WalReceiverConn *wrconn, List *publications,
> > +    CopyData copydata, char *origin,
> > +    Oid *subrel_local_oids, int subrel_count)
> >
> > 4a.
> > "copy_data = on" -> "copy_data = true" (for consistency with the docs
> > and the error messages)
>
> Modified
>
> > 4b.
> > The same NONE/none review comment from #2a applies here too. Probably
> > it should be written as none for now unless/until *everything* changes
> > to NONE.
>
> Modified
>
> > 4c.
> > "to avoid the publisher from replicating data that has an origin." ->
> > "to avoid replicating data that has an origin."
>
> Modified
>
> > 4d.
> > + * This check need not be performed on the tables that are already added as
> > + * incremental sync for such tables will happen through WAL and the origin of
> > + * the data can be identified from the WAL records.
> >
> > SUGGESTION (maybe?)
> > This check need not be performed on the tables that are already added
> > because incremental sync for those tables will happen through WAL and
> > the origin of the data can be identified from the WAL records.
> >
> > ======
>
> Modified
>
> >
> > 5. src/test/subscription/t/030_origin.pl
> >
> > + "Refresh publication when the publisher has subscribed for the new table"
> >
> > SUGGESTION (Just to mention origin = none somehow. Maybe you can
> > reword it better than this)
> > Refresh publication when the publisher has subscribed for the new
> > table, but the subscriber-side wants origin=none
>
> Modified
>
> Thanks for the comments, the attached v41 patch has the changes for the same.

The patch does not apply on Head because of the commit
"0c20dd33db1607d6a85ffce24238c1e55e384b49",  I have attached a rebased
v41 patch on top of the HEAD. I have not yet done the changes to
change the error to warning in the first patch, I will change it in
the next version once it is concluded.

Regards,
Vignesh

  229. RE: Handle infinite recursion in logical replication setup

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2022-08-17T03:18:08Z

    On Tuesday, August 2, 2022 8:00 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jul 26, 2022 at 7:13 AM Jonathan S. Katz <jkatz@postgresql.org>
> wrote:
> > >
> > > Thanks for the example. I agree that it is fairly simple to reproduce.
> > >
> > > I understand that "copy_data = force" is meant to protect a user
> > > from hurting themself. I'm not convinced that this is the best way to do so.
> > >
> > > For example today I can subscribe to multiple publications that
> > > write to the same table. If I have a primary key on that table, and
> > > two of the subscriptions try to write an identical ID, we conflict.
> > > We don't have any special flags or modes to guard against that from
> > > happening, though we do have documentation on conflicts and managing
> them.
> > >
> > > AFAICT the same issue with "copy_data" also exists in the above
> > > scenario too, even without the "origin" attribute.
> > >
> >
> > That's true but there is no parameter like origin = NONE which
> > indicates that constraint violations or duplicate data problems won't
> > occur due to replication. In the current case, I think the situation
> > is different because a user has specifically asked not to replicate
> > any remote data by specifying origin = NONE, which should be dealt
> > differently. Note that current users or their setup won't see any
> > difference/change unless they specify the new parameter origin as
> > NONE.
> >
> 
> Let me try to summarize the discussion so that it is easier for others to follow.
> The work in this thread is to avoid loops, and duplicate data in logical
> replication when the operations happened on the same table in multiple nodes.
> It has been explained in email [1] with an example of how a logical replication
> setup can lead to duplicate or inconsistent data.
> 
> The idea to solve this problem is that we don't replicate data that is not
> generated locally which we can normally identify based on origin of data in
> WAL. The commit 366283961a achieves that for replication but still the
> problem can happen during initial sync which is performed internally via copy.
> We can't differentiate the data in heap based on origin. So, we decided to
> prohibit the subscription operations that can perform initial sync (ex. Create
> Subscription, Alter Subscription ... Refresh) by detecting that the publisher has
> subscribed to the same table from some other publisher.
> 
> To prohibit the subscription operations, the currently proposed patch throws
> an error.  Then, it also provides a new copy_data option 'force' under which
> the user will still be able to perform the operation. This could be useful when
> the user intentionally wants to replicate the initial data even if it contains data
> from multiple nodes (for example, when in a multi-node setup, one decides to
> get the initial data from just one node and then allow replication of data to
> proceed from each of respective nodes).
> 
> The other alternative discussed was to just give a warning for subscription
> operations and probably document the steps for users to avoid it. But the
> problem with that is once the user sees this warning, it won't be able to do
> anything except recreate the setup, so why not give an error in the first place?
> 
> Thoughts?

Thanks for the summary.

I think it's fine to make the user use the copy_data option more carefully to
prevent duplicate copies by reporting an ERROR.

But I also have similar concern with Sawada-san as it's possible for user to
receive an ERROR in some unexpected cases.

For example I want to build bi-directional setup between two nodes:

Node A: TABLE test (has actual data)
Node B: TABLE test (empty)

Step 1:
CREATE PUBLICATION on both Node A and B.

Step 2:
CREATE SUBSCRIPTION on Node A with (copy_data = on)
-- this is fine as there is no data on Node B

Step 3:
CREATE SUBSCRIPTION on Node B with (copy_data = on)
-- this should be fine as user needs to copy data from Node A to Node B,
-- but we still report an error for this case.

It looks a bit strict to report an ERROR in this case and it seems not easy to
avoid this. So, personally, I think it might be better to document the correct
steps to build the bi-directional replication and probably also docuemnt the
steps to recover if user accidently did duplicate initial copy if not
documented yet.

In addition, we could also LOG some additional information about the ORIGIN and
initial copy which might help user to analyze if needed.


----- Some other thoughts about the duplicate initial copy problem.

Actually, I feel the better way to address the possible duplicate copy problem
is to provide a command like "bi_setup" which can help user build the
bi-directional replication in all nodes and can handle the initial copy
automtically. But that might be too far.

Another naive idea I once thought is that maybe we can add a publication option
like: data_source_in_bi_group. If data_source_in_bi_group is true, we silently
let it succeed when we subscribe this publication
with(origin='NONE',copy_data=on). If data_source_in_bi_group is false, we
report an ERROR when we subscribe this publication
with(origin='NONE',copy_data=on). The slight difference in this approach is
that we don't do any additional check for the publisher and subscriber, we
trust the user and give the responsibility to choose the node with actual data
to user. Having said that this approach seems not a good approach as it doesn't
solve the actual problem and can only handle the case when ORIGIN='NONE' and
cannot handle the future case when we support ORIGIN='NODE1'.

So, I personally feel that adding document and LOG might be sufficient at this
stage.

Best regards,
Hou zj

  230. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-17T06:33:43Z

    On Wed, Aug 17, 2022 at 8:48 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
>
> On Tuesday, August 2, 2022 8:00 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Thanks for the summary.
>
> I think it's fine to make the user use the copy_data option more carefully to
> prevent duplicate copies by reporting an ERROR.
>
> But I also have similar concern with Sawada-san as it's possible for user to
> receive an ERROR in some unexpected cases.
>
> For example I want to build bi-directional setup between two nodes:
>
> Node A: TABLE test (has actual data)
> Node B: TABLE test (empty)
>
> Step 1:
> CREATE PUBLICATION on both Node A and B.
>
> Step 2:
> CREATE SUBSCRIPTION on Node A with (copy_data = on)
> -- this is fine as there is no data on Node B
>
> Step 3:
> CREATE SUBSCRIPTION on Node B with (copy_data = on)
> -- this should be fine as user needs to copy data from Node A to Node B,
> -- but we still report an error for this case.
>
> It looks a bit strict to report an ERROR in this case and it seems not easy to
> avoid this. So, personally, I think it might be better to document the correct
> steps to build the bi-directional replication and probably also docuemnt the
> steps to recover if user accidently did duplicate initial copy if not
> documented yet.
>
> In addition, we could also LOG some additional information about the ORIGIN and
> initial copy which might help user to analyze if needed.
>

But why LOG instead of WARNING? I feel in this case there is a chance
of inconsistent data so a WARNING like "publication "pub1" could have
data from multiple origins" can be given when the user has specified
options: "copy_data = on, origin = NONE" while creating a
subscription. We give a WARNING during subscription creation when the
corresponding publication doesn't exist, eg.

postgres=# create subscription sub1 connection 'dbname = postgres'
publication pub1;
WARNING:  publication "pub1" does not exist in the publisher

Then, we can explain in docs how users can avoid data inconsistencies
while setting up replication.

-- 
With Regards,
Amit Kapila.

  231. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-17T07:03:50Z

    On Wed, Aug 17, 2022 at 4:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 17, 2022 at 8:48 AM houzj.fnst@fujitsu.com
> <houzj.fnst@fujitsu.com> wrote:
> >
> > On Tuesday, August 2, 2022 8:00 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Thanks for the summary.
> >
> > I think it's fine to make the user use the copy_data option more carefully to
> > prevent duplicate copies by reporting an ERROR.
> >
> > But I also have similar concern with Sawada-san as it's possible for user to
> > receive an ERROR in some unexpected cases.
> >
> > For example I want to build bi-directional setup between two nodes:
> >
> > Node A: TABLE test (has actual data)
> > Node B: TABLE test (empty)
> >
> > Step 1:
> > CREATE PUBLICATION on both Node A and B.
> >
> > Step 2:
> > CREATE SUBSCRIPTION on Node A with (copy_data = on)
> > -- this is fine as there is no data on Node B
> >
> > Step 3:
> > CREATE SUBSCRIPTION on Node B with (copy_data = on)
> > -- this should be fine as user needs to copy data from Node A to Node B,
> > -- but we still report an error for this case.
> >
> > It looks a bit strict to report an ERROR in this case and it seems not easy to
> > avoid this. So, personally, I think it might be better to document the correct
> > steps to build the bi-directional replication and probably also docuemnt the
> > steps to recover if user accidently did duplicate initial copy if not
> > documented yet.
> >
> > In addition, we could also LOG some additional information about the ORIGIN and
> > initial copy which might help user to analyze if needed.
> >
>
> But why LOG instead of WARNING? I feel in this case there is a chance
> of inconsistent data so a WARNING like "publication "pub1" could have
> data from multiple origins" can be given when the user has specified
> options: "copy_data = on, origin = NONE" while creating a
> subscription. We give a WARNING during subscription creation when the
> corresponding publication doesn't exist, eg.
>
> postgres=# create subscription sub1 connection 'dbname = postgres'
> publication pub1;
> WARNING:  publication "pub1" does not exist in the publisher
>
> Then, we can explain in docs how users can avoid data inconsistencies
> while setting up replication.
>

I was wondering if this copy/origin case really should be a NOTICE.

See table [1]. It says WARNING is meant for "warnings of likey
problems". But this is not exactly a "likely" problem - IIUC we really
don't know if there is even any problem at all .... we only know there
is the *potential* for a problem, but the user has to then judge it
for themselves, Perhaps WARNING may be a bit overkill for this
situation -  it might be unnecessarily scary to give false warnings.

OTOH, NOTICE [1] says it is for "information that might be helpful to
users" which seems more like what is needed here.

------
[1] https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-SEVERITY-LEVELS

Kind Regards,
Peter Smith.
Fujitsu Australia.

  232. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-18T03:13:08Z

    On Wed, Aug 17, 2022 at 12:34 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Wed, Aug 17, 2022 at 4:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Aug 17, 2022 at 8:48 AM houzj.fnst@fujitsu.com
> > <houzj.fnst@fujitsu.com> wrote:
> > >
> > > On Tuesday, August 2, 2022 8:00 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Thanks for the summary.
> > >
> > > I think it's fine to make the user use the copy_data option more carefully to
> > > prevent duplicate copies by reporting an ERROR.
> > >
> > > But I also have similar concern with Sawada-san as it's possible for user to
> > > receive an ERROR in some unexpected cases.
> > >
> > > For example I want to build bi-directional setup between two nodes:
> > >
> > > Node A: TABLE test (has actual data)
> > > Node B: TABLE test (empty)
> > >
> > > Step 1:
> > > CREATE PUBLICATION on both Node A and B.
> > >
> > > Step 2:
> > > CREATE SUBSCRIPTION on Node A with (copy_data = on)
> > > -- this is fine as there is no data on Node B
> > >
> > > Step 3:
> > > CREATE SUBSCRIPTION on Node B with (copy_data = on)
> > > -- this should be fine as user needs to copy data from Node A to Node B,
> > > -- but we still report an error for this case.
> > >
> > > It looks a bit strict to report an ERROR in this case and it seems not easy to
> > > avoid this. So, personally, I think it might be better to document the correct
> > > steps to build the bi-directional replication and probably also docuemnt the
> > > steps to recover if user accidently did duplicate initial copy if not
> > > documented yet.
> > >
> > > In addition, we could also LOG some additional information about the ORIGIN and
> > > initial copy which might help user to analyze if needed.
> > >
> >
> > But why LOG instead of WARNING? I feel in this case there is a chance
> > of inconsistent data so a WARNING like "publication "pub1" could have
> > data from multiple origins" can be given when the user has specified
> > options: "copy_data = on, origin = NONE" while creating a
> > subscription. We give a WARNING during subscription creation when the
> > corresponding publication doesn't exist, eg.
> >
> > postgres=# create subscription sub1 connection 'dbname = postgres'
> > publication pub1;
> > WARNING:  publication "pub1" does not exist in the publisher
> >
> > Then, we can explain in docs how users can avoid data inconsistencies
> > while setting up replication.
> >
>
> I was wondering if this copy/origin case really should be a NOTICE.
>

We usually give NOTICE for some sort of additional implicit
information, e.g., when we create a slot during CREATE SUBSCRIPTION
command: "NOTICE: created replication slot "sub1" on publisher". IMO,
this is likely to be a problem of data inconsistency so I think here
we can choose between WARNING and LOG. I prefer WARNING but okay with
LOG as well if others feel so. I think we can change this later as
well if required. We do have an option to not do anything and just
document it but I feel it is better to give user some indication of
problem here because not everyone reads each update of documentation.

Jonathan, Sawada-San, Hou-San, and others, what do you think is the
best way to move forward here?

-- 
With Regards,
Amit Kapila.

  233. RE: Handle infinite recursion in logical replication setup

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2022-08-22T03:49:15Z

    On Thursday, August 18, 2022 11:13 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> 
> On Wed, Aug 17, 2022 at 12:34 PM Peter Smith <smithpb2250@gmail.com>
> wrote:
> >
> > On Wed, Aug 17, 2022 at 4:33 PM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > >
> > > On Wed, Aug 17, 2022 at 8:48 AM houzj.fnst@fujitsu.com
> > > <houzj.fnst@fujitsu.com> wrote:
> > > >
> > > > On Tuesday, August 2, 2022 8:00 PM Amit Kapila
> <amit.kapila16@gmail.com> wrote:
> > > > > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila
> <amit.kapila16@gmail.com> wrote:
> > > >
> > > > Thanks for the summary.
> > > >
> > > > I think it's fine to make the user use the copy_data option more
> > > > carefully to prevent duplicate copies by reporting an ERROR.
> > > >
> > > > But I also have similar concern with Sawada-san as it's possible
> > > > for user to receive an ERROR in some unexpected cases.
> > > >
> > > > For example I want to build bi-directional setup between two nodes:
> > > >
> > > > Node A: TABLE test (has actual data) Node B: TABLE test (empty)
> > > >
> > > > Step 1:
> > > > CREATE PUBLICATION on both Node A and B.
> > > >
> > > > Step 2:
> > > > CREATE SUBSCRIPTION on Node A with (copy_data = on)
> > > > -- this is fine as there is no data on Node B
> > > >
> > > > Step 3:
> > > > CREATE SUBSCRIPTION on Node B with (copy_data = on)
> > > > -- this should be fine as user needs to copy data from Node A to
> > > > Node B,
> > > > -- but we still report an error for this case.
> > > >
> > > > It looks a bit strict to report an ERROR in this case and it seems
> > > > not easy to avoid this. So, personally, I think it might be better
> > > > to document the correct steps to build the bi-directional
> > > > replication and probably also docuemnt the steps to recover if
> > > > user accidently did duplicate initial copy if not documented yet.
> > > >
> > > > In addition, we could also LOG some additional information about
> > > > the ORIGIN and initial copy which might help user to analyze if needed.
> > > >
> > >
> > > But why LOG instead of WARNING? I feel in this case there is a
> > > chance of inconsistent data so a WARNING like "publication "pub1"
> > > could have data from multiple origins" can be given when the user
> > > has specified
> > > options: "copy_data = on, origin = NONE" while creating a
> > > subscription. We give a WARNING during subscription creation when
> > > the corresponding publication doesn't exist, eg.
> > >
> > > postgres=# create subscription sub1 connection 'dbname = postgres'
> > > publication pub1;
> > > WARNING:  publication "pub1" does not exist in the publisher
> > >
> > > Then, we can explain in docs how users can avoid data
> > > inconsistencies while setting up replication.
> > >
> >
> > I was wondering if this copy/origin case really should be a NOTICE.
> >
> 
> We usually give NOTICE for some sort of additional implicit information, e.g.,
> when we create a slot during CREATE SUBSCRIPTION
> command: "NOTICE: created replication slot "sub1" on publisher". IMO, this is
> likely to be a problem of data inconsistency so I think here we can choose
> between WARNING and LOG. I prefer WARNING but okay with LOG as well if
> others feel so. I think we can change this later as well if required. We do have an
> option to not do anything and just document it but I feel it is better to give user
> some indication of problem here because not everyone reads each update of
> documentation.
> 
> Jonathan, Sawada-San, Hou-San, and others, what do you think is the best way
> to move forward here?

I think it's fine to throw a WARNING in this case given that there is a
chance of inconsistent data.

Best regards,
Hou zj

  234. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-08-24T13:57:14Z

    On Mon, Aug 22, 2022 at 9:19 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
>
> On Thursday, August 18, 2022 11:13 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Aug 17, 2022 at 12:34 PM Peter Smith <smithpb2250@gmail.com>
> > wrote:
> > >
> > > On Wed, Aug 17, 2022 at 4:33 PM Amit Kapila <amit.kapila16@gmail.com>
> > wrote:
> > > >
> > > > On Wed, Aug 17, 2022 at 8:48 AM houzj.fnst@fujitsu.com
> > > > <houzj.fnst@fujitsu.com> wrote:
> > > > >
> > > > > On Tuesday, August 2, 2022 8:00 PM Amit Kapila
> > <amit.kapila16@gmail.com> wrote:
> > > > > > On Tue, Jul 26, 2022 at 9:07 AM Amit Kapila
> > <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > Thanks for the summary.
> > > > >
> > > > > I think it's fine to make the user use the copy_data option more
> > > > > carefully to prevent duplicate copies by reporting an ERROR.
> > > > >
> > > > > But I also have similar concern with Sawada-san as it's possible
> > > > > for user to receive an ERROR in some unexpected cases.
> > > > >
> > > > > For example I want to build bi-directional setup between two nodes:
> > > > >
> > > > > Node A: TABLE test (has actual data) Node B: TABLE test (empty)
> > > > >
> > > > > Step 1:
> > > > > CREATE PUBLICATION on both Node A and B.
> > > > >
> > > > > Step 2:
> > > > > CREATE SUBSCRIPTION on Node A with (copy_data = on)
> > > > > -- this is fine as there is no data on Node B
> > > > >
> > > > > Step 3:
> > > > > CREATE SUBSCRIPTION on Node B with (copy_data = on)
> > > > > -- this should be fine as user needs to copy data from Node A to
> > > > > Node B,
> > > > > -- but we still report an error for this case.
> > > > >
> > > > > It looks a bit strict to report an ERROR in this case and it seems
> > > > > not easy to avoid this. So, personally, I think it might be better
> > > > > to document the correct steps to build the bi-directional
> > > > > replication and probably also docuemnt the steps to recover if
> > > > > user accidently did duplicate initial copy if not documented yet.
> > > > >
> > > > > In addition, we could also LOG some additional information about
> > > > > the ORIGIN and initial copy which might help user to analyze if needed.
> > > > >
> > > >
> > > > But why LOG instead of WARNING? I feel in this case there is a
> > > > chance of inconsistent data so a WARNING like "publication "pub1"
> > > > could have data from multiple origins" can be given when the user
> > > > has specified
> > > > options: "copy_data = on, origin = NONE" while creating a
> > > > subscription. We give a WARNING during subscription creation when
> > > > the corresponding publication doesn't exist, eg.
> > > >
> > > > postgres=# create subscription sub1 connection 'dbname = postgres'
> > > > publication pub1;
> > > > WARNING:  publication "pub1" does not exist in the publisher
> > > >
> > > > Then, we can explain in docs how users can avoid data
> > > > inconsistencies while setting up replication.
> > > >
> > >
> > > I was wondering if this copy/origin case really should be a NOTICE.
> > >
> >
> > We usually give NOTICE for some sort of additional implicit information, e.g.,
> > when we create a slot during CREATE SUBSCRIPTION
> > command: "NOTICE: created replication slot "sub1" on publisher". IMO, this is
> > likely to be a problem of data inconsistency so I think here we can choose
> > between WARNING and LOG. I prefer WARNING but okay with LOG as well if
> > others feel so. I think we can change this later as well if required. We do have an
> > option to not do anything and just document it but I feel it is better to give user
> > some indication of problem here because not everyone reads each update of
> > documentation.
> >
> > Jonathan, Sawada-San, Hou-San, and others, what do you think is the best way
> > to move forward here?
>
> I think it's fine to throw a WARNING in this case given that there is a
> chance of inconsistent data.

Since there was no objections to change it to throw a warning, I have
made the changes for the same.

I have made one change for the documentation patch.
The current steps states that:
1. Create a publication on primary3.
2. Lock table t1 on primary2 and primary3 in EXCLUSIVE mode until the
setup is completed. There is no need to lock table t1 on primary1
because any data changes made will be synchronized while creating the
subscription with copy_data = true.
3. Create a subscription on primary1 to subscribe to primary3.
4. Create a subscription on primary2 to subscribe to primary3.
5. Create a subscription on primary3 to subscribe to primary1. Use
copy_data = true so that the existing table data is copied during
initial sync.
6. Create a subscription on primary3 to subscribe to primary2. Use
copy_data = false because the initial table data would have been
already copied in the previous step.
7. Now the replication setup between primaries primary1, primary2 and
primary3 is complete. Incremental changes made on any primary will be
replicated to the other two primaries.

We found a problem with the proposed steps. Here the problem is that
we lock table t1 on primary2/primary3 in step2, then we create a
subscription on primary3 in step5. While we create subscription in
step5, tablesync worker will try to lock the table in primary3 during
step5 before copying, as we have already taken a lock, this process
will wait.
Even if the lock is released in this step just before creating
subscription, there is a chance that some other transaction can
acquire the lock and change the data which will result in inconsistent
data. Here the locking solution cannot handle adding a new primary in
a consistent way.

I have changed the lock step to mention that user need to ensure no
data changes happen until the setup is completed.
The attached v42 patch has the changes for the same.

Regards,
Vignesh

  235. Re: Handle infinite recursion in logical replication setup

    Dilip Kumar <dilipbalaut@gmail.com> — 2022-08-26T04:22:27Z

    On Mon, Aug 22, 2022 at 9:19 AM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
>
> > Jonathan, Sawada-San, Hou-San, and others, what do you think is the best way
> > to move forward here?
>
> I think it's fine to throw a WARNING in this case given that there is a
> chance of inconsistent data.

IMHO, since the user has specifically asked for origin=NONE but we do
not have any way to detect the origin during initial sync so I think
this could be documented and we can also issue the WARNING.  So that
users notice that part and carefully set up the replication.  OTOH, I
do not think that giving an error is very inconvenient because we are
already providing a new option "origin=NONE" so along with that lets
force the user to choose between copy_data=off or copy_data=force and
with that, there is no scope for mistakes.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

  236. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-08-29T02:54:06Z

    On Fri, Aug 26, 2022 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Aug 22, 2022 at 9:19 AM houzj.fnst@fujitsu.com
> <houzj.fnst@fujitsu.com> wrote:
> >
> > > Jonathan, Sawada-San, Hou-San, and others, what do you think is the best way
> > > to move forward here?
> >
> > I think it's fine to throw a WARNING in this case given that there is a
> > chance of inconsistent data.
>
> IMHO, since the user has specifically asked for origin=NONE but we do
> not have any way to detect the origin during initial sync so I think
> this could be documented and we can also issue the WARNING.  So that
> users notice that part and carefully set up the replication.  OTOH, I
> do not think that giving an error is very inconvenient because we are
> already providing a new option "origin=NONE" so along with that lets
> force the user to choose between copy_data=off or copy_data=force and
> with that, there is no scope for mistakes.

Since Jonathan also had suggested to throw a warning as in [1] and
Hou-san also had suggested to throw a warning as in [2], I have made
changes to throw a warning and also documented the following contents
for the same:
+  <para>
+   If the subscription is created with <literal>origin = none</literal> and
+   <literal>copy_data = true</literal>, it will check if the publisher has
+   subscribed to the same table from other publishers and, if so, throw a
+   warning to notify user to check the publisher tables. The user can ensure
+   that publisher tables do not have data which has an origin associated before
+   continuing with any other operations to prevent inconsistent data being
+   replicated.
+  </para>

The changes for the same are available in the v42 patch at [3].
[1] - https://www.postgresql.org/message-id/afb653ba-e2b1-33a3-a54c-849f4466e1b4%40postgresql.org
[2] - https://www.postgresql.org/message-id/OS0PR01MB5716C383623ADAD64CE4841194719%40OS0PR01MB5716.jpnprd01.prod.outlook.com
[3] - https://www.postgresql.org/message-id/CALDaNm2oLWsSYtOFLdOkbrpC94%3DJZzKMCYDJoiZaqAX_Hn%2BU9Q%40mail.gmail.com

Regards,
Vignesh

  237. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-29T03:10:29Z

    On Mon, Aug 29, 2022 at 8:24 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, Aug 26, 2022 at 9:52 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > IMHO, since the user has specifically asked for origin=NONE but we do
> > not have any way to detect the origin during initial sync so I think
> > this could be documented and we can also issue the WARNING.  So that
> > users notice that part and carefully set up the replication.  OTOH, I
> > do not think that giving an error is very inconvenient because we are
> > already providing a new option "origin=NONE" so along with that lets
> > force the user to choose between copy_data=off or copy_data=force and
> > with that, there is no scope for mistakes.
>

Initially, I also thought that giving an error should be okay in this
case but later multiple people voted against that idea primarily
because we won't be able to detect whether the initial sync data
contains data from multiple origins. Consider that even though the
publisher is subscribed to some other publisher but it may not have
gotten any data from it by the time we try to subscribe from it. Also,
one might have removed the subscription that led to data from multiple
origins on the publisher, in this case, may be one can say that we can
consider that the data is now owned by the publisher but I am not
sure. So, considering this, I think giving a WARNING and documenting
how to set up replication correctly sounds like a reasonable way
forward.

> Since Jonathan also had suggested to throw a warning as in [1] and
> Hou-san also had suggested to throw a warning as in [2],
>

Agreed that multiple seems to be in favor of that approach. We can
always promote it to ERROR later if others and or users felt so.

-- 
With Regards,
Amit Kapila.

  238. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-29T06:29:27Z

    Here are some review comments for patch v42-0001:

======

1. Commit message

A later review comment below suggests some changes to the WARNING
message so if those changes are made then the example in this commit
message also needs to be modified.

======

2. doc/src/sgml/ref/alter_subscription.sgml - copy_data

Refer to the Notes for the usage of true for copy_data parameter and
its interaction with the origin parameter.

I think saying "usage of true" sounded a bit strange.

SUGGESTION
Refer to the Notes about how copy_data = true can interact with the
origin parameter.

======

3. doc/src/sgml/ref/create_subscription.sgml - copy data

Refer to the Notes for the usage of true for copy_data parameter and
its interaction with the origin parameter.

SUGGESTION
(same as #2)

~~

4. doc/src/sgml/ref/create_subscription.sgml - origin

Refer to the Notes for the usage of true for copy_data parameter and
its interaction with the origin parameter.

SUGGESTION
(same as #2)

~~~

5. doc/src/sgml/ref/create_subscription.sgml - Notes

+  <para>
+   If the subscription is created with <literal>origin = none</literal> and
+   <literal>copy_data = true</literal>, it will check if the publisher has
+   subscribed to the same table from other publishers and, if so, throw a
+   warning to notify user to check the publisher tables. The user can ensure

"throw a warning to notify user" -> "log a warning to notify the user"

~~

6.

+   warning to notify user to check the publisher tables. The user can ensure
+   that publisher tables do not have data which has an origin associated before
+   continuing with any other operations to prevent inconsistent data being
+   replicated.
+  </para>

6a.

I'm not so sure about this. IMO the warning is not about the
replication – really it is about the COPY which has already happened
anyway. So the user can't really prevent anything from going wrong;
instead, they have to take some action to clean up if anything did go
wrong.

SUGGESTION
Before continuing with other operations the user should check that
publisher tables did not have data with different origins, otherwise
inconsistent data may have been copied.

~

6b.

I am also wondering what can the user do now. Assuming there was bad
COPY then the subscriber table has already got unwanted stuff copied
into it. Is there any advice we can give to help users fix this mess?

======

7. src/backend/commands/subscriptioncmds.c - AlterSubscription_refresh

+ /* Check whether we can allow copy of newly added relations. */
+ check_pub_table_subscribed(wrconn, sub->publications, copy_data,
+    sub->origin, subrel_local_oids,
+    subrel_count);

"whether we can allow" seems not quite the right wording here anymore,
because now there is no ERROR stopping this - so if there was unwanted
data the COPY will proceed to copy it anyhow...

~~~

8. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

@@ -1781,6 +1794,121 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
  table_close(rel, RowExclusiveLock);
 }

+/*
+ * Check and throw a warning if the publisher has subscribed to the same table
+ * from some other publisher. This check is required only if "copy_data = true"
+ * and "origin = none" for CREATE SUBSCRIPTION and
+ * ALTER SUBSCRIPTION ... REFRESH statements to notify user that data having
+ * origin might have been copied.

"throw a warning" -> "log a warning"

"to notify user" -> "to notify the user" ?

~~~

9.

+ if (copydata == false || !origin ||
+ (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) != 0))
+ return;

SUGGESTION
if (!copydata || !origin ||
(pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) != 0))

~~~

10. (Question)

I can't tell just by reading the code if FOR ALL TABLES case is
handled – e.g. will this recognise the case were the publisher might
have table data from other origins because it has a subscription on
some other node that was publishing "FOR ALL TABLES"?

~~~

11.

+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("publisher has subscribed table \"%s.%s\" from some other publisher",
+    nspname, relname),
+ errdetail("Publisher might have subscribed one or more tables from
some other publisher."),
+ errhint("Verify that these publisher tables do not have data that
has an origin associated before proceeding to avoid inconsistency."));
+ break;

11a.
I'm not sure having that "break" code is correct logic. Won't that
mean the user will only get a warning for the first potential problem
encountered but then other potential problems tables will have no
warnings at all so the user may not be made aware of them? Perhaps you
need to first gather the list of all the suspicious tables before
logging a single warning which shows that list? Or perhaps you need to
log multiple warnings – one warning per suspicious table?

~

11b.
The WARNING message seems a bit inside-out because the errmsg is
giving more details than the errdetail.

SUGGESTIONS (or something similar)
errmsg - "subscription XXX requested origin=NONE but may have copied
data that had a different origin."
errdetail – Publisher YYY has subscribed table \"%s.%s\" from some
other publisher"

~

11c.
The errhint sentence is unusual.

BEFORE
"Verify that these publisher tables do not have data that has an
origin associated before proceeding to avoid inconsistency."

SUGGESTION (or something similar)
Before proceeding, verify that initial data copied from the publisher
tables did not come from other origins.

======

12. src/test/subscription/t/030_origin.pl

+# have remotely originated data from node_A. We throw a warning, in this case,
+# to draw attention to there being possible remote data.

"throw a warning" -> "log a warning" (this occurs 2x)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  239. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-29T06:31:07Z

    Here are some review comments for patch v42-0002:

======

1. doc/src/sgml/logical-replication.sgml

<literal>copy_data = true </literal>

There are a couple of these tags where there is a trailing space
before the </literal>. I guess it is doing no harm, but it is doing no
good either, so IMO better to get rid of the space.

~~

2. doc/src/sgml/logical-replication.sgml - 31.3.1. Setting replication
between two primaries

User need to ensure that no data changes happen on table t1 on
primary1 and primary2 until the setup is completed.

SUGGESTION
The user must ensure that no data changes happen on table t1 of
primary1 and primary2 until the setup is completed.

~~~

3. doc/src/sgml/logical-replication.sgml - 31.3.2. Adding a new
primary when there is no table data on any of the primaries

User need to ensure that no data changes happen on table t1 on all the
primaries primary1, primary2 and primary3 until the setup is
completed.

SUGGESTION
The user must ensure that no data changes happen on table t1 of all
the primaries primary1, primary2 and primary3 until the setup is
completed.

~~~

4. doc/src/sgml/logical-replication.sgml - 31.3.3. Adding a new
primary when table data is present on the existing primaries

User need to ensure that no operations should be performed on table t1
on primary2 and primary3 until the setup is completed.

SUGGESTION
The user must ensure that no operations are performed on table t1 of
primary2 and primary3 until the setup is completed.

Here also you changed the wording - "no data changes happen" versus
"no operations should be performed". Was that a deliberate difference?
Maybe they should all be consistent wording? If they are
interchangeable IMO the "no operations are performed..." wording was
the better of the two.

~~~

5. doc/src/sgml/logical-replication.sgml - 31.3.4. Generic steps for
adding a new primary to an existing set of primaries

Step-2: User need to ensure that no changes happen on the required
tables of the new primary until the setup is complete.

SUGGESTION
Step-2: The user must ensure that no changes happen on the required
tables of the new primary until the setup is complete.

~~~

6.

Step-4. User need to ensure that no changes happen on the required
tables of the existing primaries except the first primary until the
setup is complete.

SUGGESTION
Step-4. The user must ensure that no changes happen on the required
tables of the existing primaries except the first primary until the
setup is complete.

~~~

7. (the example)

7a.
Step-2. User need to ensure that no changes happen on table t1 on
primary4 until the setup is completed.

SUGGESTION
Step-2. The user must ensure that no changes happen on table t1 of
primary4 until the setup is completed.

~

7b.
Step-4. User need to ensure that no changes happen on table t1 on
primary2 and primary3 until the setup is completed.

SUGGESTION
Step-4. The user must ensure that no changes happen on table t1 of
primary2 and primary3 until the setup is completed.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  240. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-29T11:04:51Z

    On Wed, Aug 24, 2022 at 7:27 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Since there was no objections to change it to throw a warning, I have
> made the changes for the same.
>

Review comments for v42-0001*
==========================
1. Can we improve the query in check_pub_table_subscribed() so that it
doesn't fetch any table that is already part of the subscription on
the subscriber? This may be better than what the patch is doing which
is to first fetch such information and then skip it. If forming this
query turns out to be too complex then we can retain your method as
well but I feel it is worth trying to optimize the query used in the
patch.

2. I thought that it may be better to fetch all the tables that have
the possibility to have data from more than one origin but maybe the
list will be too long especially for FOR ALL TABLE types of cases. Is
this the reason you have decided to give a WARNING as soon as you see
any such table? If so, probably adding a comment for it can be good.

3.
+ $node_publisher->wait_for_catchup($sub_name);
+
+ # also wait for initial table sync to finish
+ $node_subscriber->poll_query_until('postgres', $synced_query)
+   or die "Timed out while waiting for subscriber to synchronize data";
....
....
....
+$node_B->safe_psql(
+ 'postgres', "
+        ALTER SUBSCRIPTION $subname_BA REFRESH PUBLICATION");
+
+$node_B->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for subscriber to synchronize data";

You can replace this and any similar code in the patch with a method
used in commit 0c20dd33db.

4.
+###############################################################################
+# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B) bidirectional
+# replication setup when the existing nodes (node_A & node_B) has pre-existing
+# data and the new node (node_C) does not have any data.
+###############################################################################
+$result = $node_A->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
+is($result, qq(), 'Check existing data');
+
+$result = $node_B->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
+is($result, qq(), 'Check existing data');
+
+$result = $node_C->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
+is($result, qq(), 'Check existing data');

The comments say that for this test, two of the nodes have some
pre-existing data but I don't see the same in the test results. The
test following this one will also have a similar effect. BTW, I am not
sure all the new three node tests added by this patch are required
because I don't see if they have additional code coverage or test
anything which is not tested without those. This has doubled the
amount of test timing for this test file which is okay if these tests
make any useful contribution but otherwise, it may be better to remove
these. Am, I missing something?

-- 
With Regards,
Amit Kapila.

  241. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-29T11:51:44Z

    On Mon, Aug 29, 2022 at 11:59 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v42-0001:
>
> ~~
>
> 6.
>
> +   warning to notify user to check the publisher tables. The user can ensure
> +   that publisher tables do not have data which has an origin associated before
> +   continuing with any other operations to prevent inconsistent data being
> +   replicated.
> +  </para>
>
> 6a.
>
> I'm not so sure about this. IMO the warning is not about the
> replication – really it is about the COPY which has already happened
> anyway. So the user can't really prevent anything from going wrong;
> instead, they have to take some action to clean up if anything did go
> wrong.
>
> SUGGESTION
> Before continuing with other operations the user should check that
> publisher tables did not have data with different origins, otherwise
> inconsistent data may have been copied.
>
> ~

I would like to avoid the use of 'may' here. So, let's change it to
something like: "Before continuing with other operations the user
should check that publisher tables did not have data with different
origins to prevent data inconsistency issues on the subscriber."
>
> 6b.
>
> I am also wondering what can the user do now. Assuming there was bad
> COPY then the subscriber table has already got unwanted stuff copied
> into it. Is there any advice we can give to help users fix this mess?
>

I don't think the user can do much in that case. She will probably
need to re-create the replication.

>
> 11.
>
> + ereport(WARNING,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("publisher has subscribed table \"%s.%s\" from some other publisher",
> +    nspname, relname),
> + errdetail("Publisher might have subscribed one or more tables from
> some other publisher."),
> + errhint("Verify that these publisher tables do not have data that
> has an origin associated before proceeding to avoid inconsistency."));
> + break;
>
> 11a.
> I'm not sure having that "break" code is correct logic. Won't that
> mean the user will only get a warning for the first potential problem
> encountered but then other potential problems tables will have no
> warnings at all so the user may not be made aware of them? Perhaps you
> need to first gather the list of all the suspicious tables before
> logging a single warning which shows that list? Or perhaps you need to
> log multiple warnings – one warning per suspicious table?
>
> ~
>
> 11b.
> The WARNING message seems a bit inside-out because the errmsg is
> giving more details than the errdetail.
>
> SUGGESTIONS (or something similar)
> errmsg - "subscription XXX requested origin=NONE but may have copied
> data that had a different origin."
> errdetail – Publisher YYY has subscribed table \"%s.%s\" from some
> other publisher"
>

Your suggestion is better but we can't say 'copied' because the copy
may start afterwards by tablesync worker. Also, using 'may' is not
advisable in error messages. How about : "subscription XXX requested
origin=NONE but might copy data that had a different origin."?

-- 
With Regards,
Amit Kapila.

  242. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-08-30T17:05:30Z

    On Mon, 29 Aug 2022 at 11:59, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v42-0001:
>
> ======
>
> 1. Commit message
>
> A later review comment below suggests some changes to the WARNING
> message so if those changes are made then the example in this commit
> message also needs to be modified.

Modified

> ======
>
> 2. doc/src/sgml/ref/alter_subscription.sgml - copy_data
>
> Refer to the Notes for the usage of true for copy_data parameter and
> its interaction with the origin parameter.
>
> I think saying "usage of true" sounded a bit strange.
>
> SUGGESTION
> Refer to the Notes about how copy_data = true can interact with the
> origin parameter.

Modified

> ======
>
> 3. doc/src/sgml/ref/create_subscription.sgml - copy data
>
> Refer to the Notes for the usage of true for copy_data parameter and
> its interaction with the origin parameter.
>
> SUGGESTION
> (same as #2)

Modified

> ~~
>
> 4. doc/src/sgml/ref/create_subscription.sgml - origin
>
> Refer to the Notes for the usage of true for copy_data parameter and
> its interaction with the origin parameter.
>
> SUGGESTION
> (same as #2)

Modified

> ~~~
>
> 5. doc/src/sgml/ref/create_subscription.sgml - Notes
>
> +  <para>
> +   If the subscription is created with <literal>origin = none</literal> and
> +   <literal>copy_data = true</literal>, it will check if the publisher has
> +   subscribed to the same table from other publishers and, if so, throw a
> +   warning to notify user to check the publisher tables. The user can ensure
>
> "throw a warning to notify user" -> "log a warning to notify the user"

Modified

> ~~
>
> 6.
>
> +   warning to notify user to check the publisher tables. The user can ensure
> +   that publisher tables do not have data which has an origin associated before
> +   continuing with any other operations to prevent inconsistent data being
> +   replicated.
> +  </para>
>
> 6a.
>
> I'm not so sure about this. IMO the warning is not about the
> replication – really it is about the COPY which has already happened
> anyway. So the user can't really prevent anything from going wrong;
> instead, they have to take some action to clean up if anything did go
> wrong.
>
> SUGGESTION
> Before continuing with other operations the user should check that
> publisher tables did not have data with different origins, otherwise
> inconsistent data may have been copied.

Modified based on the suggestion provided by Amit.

> ~
>
> 6b.
>
> I am also wondering what can the user do now. Assuming there was bad
> COPY then the subscriber table has already got unwanted stuff copied
> into it. Is there any advice we can give to help users fix this mess?

There is nothing much a user can do in this case. Only option would be
to take a backup before the operation and restore it and then recreate
the replication setup. I was not sure if we should document these
steps as similar inconsistent data could get created without the
origin option . Thoughts?

> ======
>
> 7. src/backend/commands/subscriptioncmds.c - AlterSubscription_refresh
>
> + /* Check whether we can allow copy of newly added relations. */
> + check_pub_table_subscribed(wrconn, sub->publications, copy_data,
> +    sub->origin, subrel_local_oids,
> +    subrel_count);
>
> "whether we can allow" seems not quite the right wording here anymore,
> because now there is no ERROR stopping this - so if there was unwanted
> data the COPY will proceed to copy it anyhow...

Removed the comment as the function details the necessary things.

> ~~~
>
> 8. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> @@ -1781,6 +1794,121 @@ AlterSubscriptionOwner_oid(Oid subid, Oid newOwnerId)
>   table_close(rel, RowExclusiveLock);
>  }
>
> +/*
> + * Check and throw a warning if the publisher has subscribed to the same table
> + * from some other publisher. This check is required only if "copy_data = true"
> + * and "origin = none" for CREATE SUBSCRIPTION and
> + * ALTER SUBSCRIPTION ... REFRESH statements to notify user that data having
> + * origin might have been copied.
>
> "throw a warning" -> "log a warning"
>
> "to notify user" -> "to notify the user" ?

Modified

> ~~~
>
> 9.
>
> + if (copydata == false || !origin ||
> + (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) != 0))
> + return;
>
> SUGGESTION
> if (!copydata || !origin ||
> (pg_strcasecmp(origin, LOGICALREP_ORIGIN_NONE) != 0))

Modified

> ~~~
>
> 10. (Question)
>
> I can't tell just by reading the code if FOR ALL TABLES case is
> handled – e.g. will this recognise the case were the publisher might
> have table data from other origins because it has a subscription on
> some other node that was publishing "FOR ALL TABLES"?

Yes it handles it.

> ~~~
>
> 11.
>
> + ereport(WARNING,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("publisher has subscribed table \"%s.%s\" from some other publisher",
> +    nspname, relname),
> + errdetail("Publisher might have subscribed one or more tables from
> some other publisher."),
> + errhint("Verify that these publisher tables do not have data that
> has an origin associated before proceeding to avoid inconsistency."));
> + break;
>
> 11a.
> I'm not sure having that "break" code is correct logic. Won't that
> mean the user will only get a warning for the first potential problem
> encountered but then other potential problems tables will have no
> warnings at all so the user may not be made aware of them? Perhaps you
> need to first gather the list of all the suspicious tables before
> logging a single warning which shows that list? Or perhaps you need to
> log multiple warnings – one warning per suspicious table?

Modified to get the list of all the tables, I have limited it to
display 100 tables followed by ... to indicate there might be more
tables. I did not want to loga warning message with many 1000's of
tables which will not help the readability of the message.

> ~
>
> 11b.
> The WARNING message seems a bit inside-out because the errmsg is
> giving more details than the errdetail.
>
> SUGGESTIONS (or something similar)
> errmsg - "subscription XXX requested origin=NONE but may have copied
> data that had a different origin."
> errdetail – Publisher YYY has subscribed table \"%s.%s\" from some
> other publisher"

We don't have the tables based on each publisher, it is for all the
publishers. I have changed the errdetail message slightly.

> ~
>
> 11c.
> The errhint sentence is unusual.
>
> BEFORE
> "Verify that these publisher tables do not have data that has an
> origin associated before proceeding to avoid inconsistency."
>
> SUGGESTION (or something similar)
> Before proceeding, verify that initial data copied from the publisher
> tables did not come from other origins.

Modified

> ======
>
> 12. src/test/subscription/t/030_origin.pl
>
> +# have remotely originated data from node_A. We throw a warning, in this case,
> +# to draw attention to there being possible remote data.
>
> "throw a warning" -> "log a warning" (this occurs 2x)

Modified

The attached v43 patch has the changes for the same.

Regards,
Vignesh

  243. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-08-30T17:11:02Z

    On Mon, 29 Aug 2022 at 12:01, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v42-0002:
>
> ======
>
> 1. doc/src/sgml/logical-replication.sgml
>
> <literal>copy_data = true </literal>
>
> There are a couple of these tags where there is a trailing space
> before the </literal>. I guess it is doing no harm, but it is doing no
> good either, so IMO better to get rid of the space.

Modified

> ~~
>
> 2. doc/src/sgml/logical-replication.sgml - 31.3.1. Setting replication
> between two primaries
>
> User need to ensure that no data changes happen on table t1 on
> primary1 and primary2 until the setup is completed.
>
> SUGGESTION
> The user must ensure that no data changes happen on table t1 of
> primary1 and primary2 until the setup is completed.

Modified

> ~~~
>
> 3. doc/src/sgml/logical-replication.sgml - 31.3.2. Adding a new
> primary when there is no table data on any of the primaries
>
> User need to ensure that no data changes happen on table t1 on all the
> primaries primary1, primary2 and primary3 until the setup is
> completed.
>
> SUGGESTION
> The user must ensure that no data changes happen on table t1 of all
> the primaries primary1, primary2 and primary3 until the setup is
> completed.

Modified

> ~~~
>
> 4. doc/src/sgml/logical-replication.sgml - 31.3.3. Adding a new
> primary when table data is present on the existing primaries
>
> User need to ensure that no operations should be performed on table t1
> on primary2 and primary3 until the setup is completed.
>
> SUGGESTION
> The user must ensure that no operations are performed on table t1 of
> primary2 and primary3 until the setup is completed.
>
> Here also you changed the wording - "no data changes happen" versus
> "no operations should be performed". Was that a deliberate difference?
> Maybe they should all be consistent wording? If they are
> interchangeable IMO the "no operations are performed..." wording was
> the better of the two.

Modified, it was not a deliberate change. I have changed it to "no
operations are performed" in other places too.

> ~~~
>
> 5. doc/src/sgml/logical-replication.sgml - 31.3.4. Generic steps for
> adding a new primary to an existing set of primaries
>
> Step-2: User need to ensure that no changes happen on the required
> tables of the new primary until the setup is complete.
>
> SUGGESTION
> Step-2: The user must ensure that no changes happen on the required
> tables of the new primary until the setup is complete.

Modified

> ~~~
>
> 6.
>
> Step-4. User need to ensure that no changes happen on the required
> tables of the existing primaries except the first primary until the
> setup is complete.
>
> SUGGESTION
> Step-4. The user must ensure that no changes happen on the required
> tables of the existing primaries except the first primary until the
> setup is complete.

Modified

> ~~~
>
> 7. (the example)
>
> 7a.
> Step-2. User need to ensure that no changes happen on table t1 on
> primary4 until the setup is completed.
>
> SUGGESTION
> Step-2. The user must ensure that no changes happen on table t1 of
> primary4 until the setup is completed.

Modified

> ~
>
> 7b.
> Step-4. User need to ensure that no changes happen on table t1 on
> primary2 and primary3 until the setup is completed.
>
> SUGGESTION
> Step-4. The user must ensure that no changes happen on table t1 of
> primary2 and primary3 until the setup is completed.

Modified

Thanks for the comments, the changes for the same are available in the
v43 patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm1V%2BAvzPsUcq%3DmNYG%2BJfAyovTWBe4vWKTknOgH_ko1E0Q%40mail.gmail.com

Regards,
Vignesh

  244. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-31T06:05:01Z

    Here are my review comments for v43-0001.

======

1. Commit message

The initial copy phase has no way to know the origin of the row data,
so if 'copy_data = true' in the step 4 below, log a warning to notify user
that potentially non-local data might have been copied.

1a.
"in the step 4" -> "in step 4"

~

1b.
"notify user" -> "notify the user"

======

2. doc/src/sgml/ref/create_subscription.sgml

+  <para>
+   If the subscription is created with <literal>origin = none</literal> and
+   <literal>copy_data = true</literal>, it will check if the publisher has
+   subscribed to the same table from other publishers and, if so, log a
+   warning to notify the user to check the publisher tables. Before continuing
+   with other operations the user should check that publisher tables did not
+   have data with different origins to prevent data inconsistency issues on the
+   subscriber.
+  </para>

I am not sure about that wording saying "to prevent data inconsistency
issues" because I thought maybe is already too late because any
unwanted origin data is already copied during the initial sync. I
think the user can do it try to clean up any problems caused before
things become even worse (??)

~~~

You asked for my thoughts (see [1] 6b):

> There is nothing much a user can do in this case. Only option would be
> to take a backup before the operation and restore it and then recreate
> the replication setup. I was not sure if we should document these
> steps as similar inconsistent data could get created without the
> origin option . Thoughts?

I think those cases are a bit different:

- For a normal scenario (i.e. origin=ANY) then inconsistent data just
refers to problems like maybe PK violations so I think you just get
what you get and have to deal with the errors...

- But when the user said origin=NONE they are specifically saying they
do NOT want any non-local data, yet they still might end up getting
data contrary to their wishes. So I think maybe there ought to be
something documented to warn about this potentially happening. My
first impression is that all this seems like a kind of terrible
problem because if it happens then cleanup could be difficult - if
that subscriber in turn was also a publisher then this bad data might
have propagated all over the place already! Anyway, I guess all this
could be mentioned in some (new ?) section of the
logical-replication.sgml file (i.e. patch 0002).

IDEA: I was wondering if the documentation should recommend (or maybe
we can even enforce this) that when using origin=NONE the user should
always create the subscription with enabled=FALSE. That way if there
are some warnings about potential bad origin data then there it might
be possible to take some action or change their mind *before* any
unwanted data pollutes everything.

======

3. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed

+ if (count == 0)
+ tablenames = makeStringInfo();
+ else
+ appendStringInfoString(tablenames, ", ");
+
+ appendStringInfo(tablenames, "%s.%s", nspname, relname);
+ count++;

I think the quotes are not correct - each separate table name should be quoted.

~~~

4.

+ /*
+ * There might be many tables present in this case, we will display a
+ * maximum of 100 tables followed by "..." to indicate there might be
+ * more tables.
+ */
+ if (count == 100)
+ {
+ appendStringInfoString(tablenames, ", ...");
+ break;
+ }

4a.
IMO this code should be part of the previous if/else

e.g.
if (count == 0) ... makeStringInfo()
else if (count > 100) ... append ellipsis "..." and break out of the loop
else ... append table name to the message

Doing it in this way means "..." definitely means there are more than
100 bad tables.

~

4b.
Changing like above (#4a) simplifies the comment because it removes
doubts like "might be…" since you already know you found the 101st
table.

SUGGESTION (comment)
The message displays a maximum of 100 tables having potentially
non-local data. Any more than that will be indicated by "...".

~

4c.
It still seems a bit disconcerting to me that there can be cases where
bad data was possibly copied but there is no record of what tables
have been polluted. Perhaps when the maximum bad tables are exceeded
then there can be some additional message to tell users what SQL they
can use to discover what all the other bad tables were.

~~~

5.

+ * XXX: For simplicity, we don't check whether the table has any data
+ * or not. If the table doesn't have any data then we don't need to
+ * distinguish between data having origin and data not having origin so
+ * we can avoid throwing a warning in that case.

"throwing a warning" -> "logging a warning"

~~~

6.

+ errdetail("Subscribed table(s) \"%s\" has been subscribed from some
other publisher.",
+   tablenames->data),

I don't think the table name list should be quoted in the errdetail,
because each of the individual table names should have been quoted.
See previous review comment #3.

======

7. src/test/subscription/t/030_origin.pl

+###############################################################################
+# Specify origin as 'none' which indicates that the publisher should only
+# replicate the changes that are generated locally from node_B, but in
+# this case since the node_B is also subscribing data from node_A, node_B can
+# have remotely originated data from node_A. We log a warning, in this case,
+# to draw attention to there being possible remote data.
+###############################################################################

This comment wording makes it seem like this is using different
subscription parameters from the other tests that preceded it, but in
fact the prior tests also been using origin=none. (??)

Maybe it just needs a trivial rewording.

SUGGESTION
Specifying origin=NONE indicates that...

~~~

8.

+like(
+ $stderr,
+ qr/WARNING: ( [A-Z0-9]+:)? subscription tap_sub_a_b_2 requested
origin=NONE but might copy data that had a different origin/,
+ "Create subscription with origin = none and copy_data when the
publisher has subscribed same table"
+);

Is it possible for these checks of the log file to get the errdetail
message too? Then the test will be more readable for the expected
result and you will also have the bonus of testing the table-name list
in the warning more thoroughly.

------
[1] https://www.postgresql.org/message-id/CALDaNm1V%2BAvzPsUcq%3DmNYG%2BJfAyovTWBe4vWKTknOgH_ko1E0Q%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  245. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-08-31T06:15:29Z

    Here are some review comments for patch v43-0002:

======

1. doc/src/sgml/ref/create_subscription.sgml

@@ -403,7 +403,9 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
    warning to notify the user to check the publisher tables. Before continuing
    with other operations the user should check that publisher tables did not
    have data with different origins to prevent data inconsistency issues on the
-   subscriber.
+   subscriber. Refer to <xref linkend="replication-between-primaries"/> for
+   how <literal>copy_data</literal> and <literal>origin</literal> can be used
+   to set up replication between primaries.
   </para>

Regarding my earlier v43-0001 review (see [1] comment #2) perhaps
another pg docs section should be added in the
logical-replication.sgml (e.g. "Specifying origins during CREATE
SUBSCRIPTION"), so then this Notes text also should have more added to
it.

SUGGESTION
Refer to <XXX_REF> for details about potential initialization
inconsistency warnings using origin=NONE.
Refer to <YYY_REF> for how copy_data and origin can be used to set up
replication between primaries.

------
[1] https://www.postgresql.org/message-id/CAHut%2BPvonTd423-cWqoxh0w8Bd_Po3OToqqyxuR1iMNmxSLr_Q%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  246. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-31T10:36:40Z

    On Wed, Aug 31, 2022 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v43-0001.
>
> ======
>
> 1. Commit message
>
> The initial copy phase has no way to know the origin of the row data,
> so if 'copy_data = true' in the step 4 below, log a warning to notify user
> that potentially non-local data might have been copied.
>
> 1a.
> "in the step 4" -> "in step 4"
>
> ~
>
> 1b.
> "notify user" -> "notify the user"
>
> ======
>
> 2. doc/src/sgml/ref/create_subscription.sgml
>
> +  <para>
> +   If the subscription is created with <literal>origin = none</literal> and
> +   <literal>copy_data = true</literal>, it will check if the publisher has
> +   subscribed to the same table from other publishers and, if so, log a
> +   warning to notify the user to check the publisher tables. Before continuing
> +   with other operations the user should check that publisher tables did not
> +   have data with different origins to prevent data inconsistency issues on the
> +   subscriber.
> +  </para>
>
> I am not sure about that wording saying "to prevent data inconsistency
> issues" because I thought maybe is already too late because any
> unwanted origin data is already copied during the initial sync. I
> think the user can do it try to clean up any problems caused before
> things become even worse (??)
>

Oh no, it is not too late in all cases. The problem can only occur if
the user will try to subscribe from all the different publications
with copy_data = true. We are anyway trying to clarify in the second
patch the right way to accomplish this. So, I am not sure what better
we can do here. The only bulletproof way is to provide some APIs where
users don't need to bother about all these cases, basically something
similar to what Kuroda-San is working on in the thread [1].

> ~~~
>
> You asked for my thoughts (see [1] 6b):
>
> > There is nothing much a user can do in this case. Only option would be
> > to take a backup before the operation and restore it and then recreate
> > the replication setup. I was not sure if we should document these
> > steps as similar inconsistent data could get created without the
> > origin option . Thoughts?
>
> I think those cases are a bit different:
>
> - For a normal scenario (i.e. origin=ANY) then inconsistent data just
> refers to problems like maybe PK violations so I think you just get
> what you get and have to deal with the errors...
>
> - But when the user said origin=NONE they are specifically saying they
> do NOT want any non-local data, yet they still might end up getting
> data contrary to their wishes. So I think maybe there ought to be
> something documented to warn about this potentially happening. My
> first impression is that all this seems like a kind of terrible
> problem because if it happens then cleanup could be difficult - if
> that subscriber in turn was also a publisher then this bad data might
> have propagated all over the place already! Anyway, I guess all this
> could be mentioned in some (new ?) section of the
> logical-replication.sgml file (i.e. patch 0002).
>
> IDEA: I was wondering if the documentation should recommend (or maybe
> we can even enforce this) that when using origin=NONE the user should
> always create the subscription with enabled=FALSE. That way if there
> are some warnings about potential bad origin data then there it might
> be possible to take some action or change their mind *before* any
> unwanted data pollutes everything.
>

If we want to give this suggestion, then we need to also say that this
is only valid when copy_data is true. The other drawback is if users
always need to do this to use origin=NONE then this can be a burden to
the user. I am not against having such a NOTE but the tone should not
be to enforce users to do this.

[1] - https://www.postgresql.org/message-id/CAHut%2BPuwRAoWY9pz%3DEubps3ooQCOBFiYPU9Yi%3DVB-U%2ByORU7OA%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  247. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-09-01T00:19:19Z

    On Wed, Aug 31, 2022 at 8:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 31, 2022 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my review comments for v43-0001.
> >
...
> > ======
> >
> > 2. doc/src/sgml/ref/create_subscription.sgml
> >
> > +  <para>
> > +   If the subscription is created with <literal>origin = none</literal> and
> > +   <literal>copy_data = true</literal>, it will check if the publisher has
> > +   subscribed to the same table from other publishers and, if so, log a
> > +   warning to notify the user to check the publisher tables. Before continuing
> > +   with other operations the user should check that publisher tables did not
> > +   have data with different origins to prevent data inconsistency issues on the
> > +   subscriber.
> > +  </para>
> >
> > I am not sure about that wording saying "to prevent data inconsistency
> > issues" because I thought maybe is already too late because any
> > unwanted origin data is already copied during the initial sync. I
> > think the user can do it try to clean up any problems caused before
> > things become even worse (??)
> >
>
> Oh no, it is not too late in all cases. The problem can only occur if
> the user will try to subscribe from all the different publications
> with copy_data = true. We are anyway trying to clarify in the second
> patch the right way to accomplish this. So, I am not sure what better
> we can do here. The only bulletproof way is to provide some APIs where
> users don't need to bother about all these cases, basically something
> similar to what Kuroda-San is working on in the thread [1].
>

The point of my review comment was only about the wording of the note
- specifically, you cannot "prevent" something (e,g, data
inconsistency) if it has already happened.

Maybe modify the wording like below?

BEFORE
Before continuing with other operations the user should check that
publisher tables did not have data with different origins to prevent
data inconsistency issues on the subscriber.

AFTER
If a publisher table with data from different origins was found to
have been copied to the subscriber, then some corrective action may be
necessary before continuing with other operations.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  248. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-09-01T02:31:37Z

    On Wed, Aug 31, 2022 1:06 AM vignesh C <vignesh21@gmail.com> wrote:
> 
> The attached v43 patch has the changes for the same.
> 

Thanks for updating the patch.

Here is a comment on the 0001 patch.

+		if (!isnewtable)
+		{
+			pfree(nspname);
+			pfree(relname);
+			continue;
+		}

If it is a new table, in which case it would log a warning, should we also call
pfree()?

Regards,
Shi yu

  249. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-01T03:49:20Z

    On Wed, Aug 31, 2022 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v43-0001.
>
> ======
>
> 1. Commit message
>
> The initial copy phase has no way to know the origin of the row data,
> so if 'copy_data = true' in the step 4 below, log a warning to notify user
> that potentially non-local data might have been copied.
>
> 1a.
> "in the step 4" -> "in step 4"
>
> ~
>
> 1b.
> "notify user" -> "notify the user"
>
> ======
>
> 2. doc/src/sgml/ref/create_subscription.sgml
>
> +  <para>
> +   If the subscription is created with <literal>origin = none</literal> and
> +   <literal>copy_data = true</literal>, it will check if the publisher has
> +   subscribed to the same table from other publishers and, if so, log a
> +   warning to notify the user to check the publisher tables. Before continuing
> +   with other operations the user should check that publisher tables did not
> +   have data with different origins to prevent data inconsistency issues on the
> +   subscriber.
> +  </para>
>
> I am not sure about that wording saying "to prevent data inconsistency
> issues" because I thought maybe is already too late because any
> unwanted origin data is already copied during the initial sync. I
> think the user can do it try to clean up any problems caused before
> things become even worse (??)
>
> ~~~
>
> You asked for my thoughts (see [1] 6b):
>
> > There is nothing much a user can do in this case. Only option would be
> > to take a backup before the operation and restore it and then recreate
> > the replication setup. I was not sure if we should document these
> > steps as similar inconsistent data could get created without the
> > origin option . Thoughts?
>
> I think those cases are a bit different:
>
> - For a normal scenario (i.e. origin=ANY) then inconsistent data just
> refers to problems like maybe PK violations so I think you just get
> what you get and have to deal with the errors...
>
> - But when the user said origin=NONE they are specifically saying they
> do NOT want any non-local data, yet they still might end up getting
> data contrary to their wishes. So I think maybe there ought to be
> something documented to warn about this potentially happening. My
> first impression is that all this seems like a kind of terrible
> problem because if it happens then cleanup could be difficult - if
> that subscriber in turn was also a publisher then this bad data might
> have propagated all over the place already! Anyway, I guess all this
> could be mentioned in some (new ?) section of the
> logical-replication.sgml file (i.e. patch 0002).
>
> IDEA: I was wondering if the documentation should recommend (or maybe
> we can even enforce this) that when using origin=NONE the user should
> always create the subscription with enabled=FALSE. That way if there
> are some warnings about potential bad origin data then there it might
> be possible to take some action or change their mind *before* any
> unwanted data pollutes everything.
>
> ======
>
> 3. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + if (count == 0)
> + tablenames = makeStringInfo();
> + else
> + appendStringInfoString(tablenames, ", ");
> +
> + appendStringInfo(tablenames, "%s.%s", nspname, relname);
> + count++;
>
> I think the quotes are not correct - each separate table name should be quoted.
>
> ~~~
>
> 4.
>
> + /*
> + * There might be many tables present in this case, we will display a
> + * maximum of 100 tables followed by "..." to indicate there might be
> + * more tables.
> + */
> + if (count == 100)
> + {
> + appendStringInfoString(tablenames, ", ...");
> + break;
> + }
>
> 4a.
> IMO this code should be part of the previous if/else
>
> e.g.
> if (count == 0) ... makeStringInfo()
> else if (count > 100) ... append ellipsis "..." and break out of the loop
> else ... append table name to the message
>
> Doing it in this way means "..." definitely means there are more than
> 100 bad tables.
>
> ~
>
> 4b.
> Changing like above (#4a) simplifies the comment because it removes
> doubts like "might be…" since you already know you found the 101st
> table.
>
> SUGGESTION (comment)
> The message displays a maximum of 100 tables having potentially
> non-local data. Any more than that will be indicated by "...".
>
> ~
>

Are we using this style of an error message anywhere else in the code?
If not, then I think we should avoid it here as well as it appears a
bit adhoc to me in the sense that why restrict at 100 tables. Can we
instead think of displaying the publications list in the message? So
errdetail would be something like: Subscribed publications \"%s\" has
been subscribed from some other publisher. Then we can probably give a
SQL query for a user to find tables that may data from multiple
origins especially if that is not complicated. Also, then we can
change the function name to check_publications_origin().

Few other minor comments on v43-0001*
1.
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("subscription %s requested origin=NONE but might copy data
that had a different origin.",
+ subname),

In error message, we don't use full stop at the end.

2.
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("subscription %s requested origin=NONE but might copy data
that had a different origin.",
+ subname),
+ errdetail("Subscribed table(s) \"%s\" has been subscribed from some
other publisher.",
+   tablenames->data),

It is better to use errdetail_plural here as there could be one or
more objects in this message?

-- 
With Regards,
Amit Kapila.

  250. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-01T18:39:11Z

    On Wed, 31 Aug 2022 at 11:35, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v43-0001.
>
> ======
>
> 1. Commit message
>
> The initial copy phase has no way to know the origin of the row data,
> so if 'copy_data = true' in the step 4 below, log a warning to notify user
> that potentially non-local data might have been copied.
>
> 1a.
> "in the step 4" -> "in step 4"

Modified

> ~
>
> 1b.
> "notify user" -> "notify the user"

Modified

> ======
>
> 2. doc/src/sgml/ref/create_subscription.sgml
>
> +  <para>
> +   If the subscription is created with <literal>origin = none</literal> and
> +   <literal>copy_data = true</literal>, it will check if the publisher has
> +   subscribed to the same table from other publishers and, if so, log a
> +   warning to notify the user to check the publisher tables. Before continuing
> +   with other operations the user should check that publisher tables did not
> +   have data with different origins to prevent data inconsistency issues on the
> +   subscriber.
> +  </para>
>
> I am not sure about that wording saying "to prevent data inconsistency
> issues" because I thought maybe is already too late because any
> unwanted origin data is already copied during the initial sync. I
> think the user can do it try to clean up any problems caused before
> things become even worse (??)
>
> ~~~
>
> You asked for my thoughts (see [1] 6b):
>
> > There is nothing much a user can do in this case. Only option would be
> > to take a backup before the operation and restore it and then recreate
> > the replication setup. I was not sure if we should document these
> > steps as similar inconsistent data could get created without the
> > origin option . Thoughts?
>
> I think those cases are a bit different:
>
> - For a normal scenario (i.e. origin=ANY) then inconsistent data just
> refers to problems like maybe PK violations so I think you just get
> what you get and have to deal with the errors...
>
> - But when the user said origin=NONE they are specifically saying they
> do NOT want any non-local data, yet they still might end up getting
> data contrary to their wishes. So I think maybe there ought to be
> something documented to warn about this potentially happening. My
> first impression is that all this seems like a kind of terrible
> problem because if it happens then cleanup could be difficult - if
> that subscriber in turn was also a publisher then this bad data might
> have propagated all over the place already! Anyway, I guess all this
> could be mentioned in some (new ?) section of the
> logical-replication.sgml file (i.e. patch 0002).
>
> IDEA: I was wondering if the documentation should recommend (or maybe
> we can even enforce this) that when using origin=NONE the user should
> always create the subscription with enabled=FALSE. That way if there
> are some warnings about potential bad origin data then there it might
> be possible to take some action or change their mind *before* any
> unwanted data pollutes everything.

Updated document based on your suggestion.

> ======
>
> 3. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
>
> + if (count == 0)
> + tablenames = makeStringInfo();
> + else
> + appendStringInfoString(tablenames, ", ");
> +
> + appendStringInfo(tablenames, "%s.%s", nspname, relname);
> + count++;
>
> I think the quotes are not correct - each separate table name should be quoted.

We will not be displaying tables anymore, the comment does not apply anymore

> ~~~
>
> 4.
>
> + /*
> + * There might be many tables present in this case, we will display a
> + * maximum of 100 tables followed by "..." to indicate there might be
> + * more tables.
> + */
> + if (count == 100)
> + {
> + appendStringInfoString(tablenames, ", ...");
> + break;
> + }
>
> 4a.
> IMO this code should be part of the previous if/else
>
> e.g.
> if (count == 0) ... makeStringInfo()
> else if (count > 100) ... append ellipsis "..." and break out of the loop
> else ... append table name to the message
>
> Doing it in this way means "..." definitely means there are more than
> 100 bad tables.

This code is removed, the comment is no more applicable.

> ~
>
> 4b.
> Changing like above (#4a) simplifies the comment because it removes
> doubts like "might be…" since you already know you found the 101st
> table.
>
> SUGGESTION (comment)
> The message displays a maximum of 100 tables having potentially
> non-local data. Any more than that will be indicated by "...".

This code is removed, the comment is no more applicable.

> ~
>
> 4c.
> It still seems a bit disconcerting to me that there can be cases where
> bad data was possibly copied but there is no record of what tables
> have been polluted. Perhaps when the maximum bad tables are exceeded
> then there can be some additional message to tell users what SQL they
> can use to discover what all the other bad tables were.

We will not be displaying the tablename anymore, we will be displaying
the publication names which has been subscribed from other
publications.

> ~~~
>
> 5.
>
> + * XXX: For simplicity, we don't check whether the table has any data
> + * or not. If the table doesn't have any data then we don't need to
> + * distinguish between data having origin and data not having origin so
> + * we can avoid throwing a warning in that case.
>
> "throwing a warning" -> "logging a warning"

Modified

> ~~~
>
> 6.
>
> + errdetail("Subscribed table(s) \"%s\" has been subscribed from some
> other publisher.",
> +   tablenames->data),
>
> I don't think the table name list should be quoted in the errdetail,
> because each of the individual table names should have been quoted.
> See previous review comment #3.

Modified

> ======
>
> 7. src/test/subscription/t/030_origin.pl
>
> +###############################################################################
> +# Specify origin as 'none' which indicates that the publisher should only
> +# replicate the changes that are generated locally from node_B, but in
> +# this case since the node_B is also subscribing data from node_A, node_B can
> +# have remotely originated data from node_A. We log a warning, in this case,
> +# to draw attention to there being possible remote data.
> +###############################################################################
>
> This comment wording makes it seem like this is using different
> subscription parameters from the other tests that preceded it, but in
> fact the prior tests also been using origin=none. (??)
>
> Maybe it just needs a trivial rewording.
>
> SUGGESTION
> Specifying origin=NONE indicates that...

Modified

> ~~~
>
> 8.
>
> +like(
> + $stderr,
> + qr/WARNING: ( [A-Z0-9]+:)? subscription tap_sub_a_b_2 requested
> origin=NONE but might copy data that had a different origin/,
> + "Create subscription with origin = none and copy_data when the
> publisher has subscribed same table"
> +);
>
> Is it possible for these checks of the log file to get the errdetail
> message too? Then the test will be more readable for the expected
> result and you will also have the bonus of testing the table-name list
> in the warning more thoroughly.

I did not see any tap test doing this, I did not want to make some
change which would fail in buildfarm machines. I have not done any
change for this.

Thanks for the comments, the attached v44 patch has the changes for the same.

Regards,
Vignesh

  251. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-01T18:44:09Z

    On Mon, 29 Aug 2022 at 16:35, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 24, 2022 at 7:27 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Since there was no objections to change it to throw a warning, I have
> > made the changes for the same.
> >
>
> Review comments for v42-0001*
> ==========================
> 1. Can we improve the query in check_pub_table_subscribed() so that it
> doesn't fetch any table that is already part of the subscription on
> the subscriber? This may be better than what the patch is doing which
> is to first fetch such information and then skip it. If forming this
> query turns out to be too complex then we can retain your method as
> well but I feel it is worth trying to optimize the query used in the
> patch.

I'm analysing this, I will post a reply for this soon.

> 2. I thought that it may be better to fetch all the tables that have
> the possibility to have data from more than one origin but maybe the
> list will be too long especially for FOR ALL TABLE types of cases. Is
> this the reason you have decided to give a WARNING as soon as you see
> any such table? If so, probably adding a comment for it can be good.

Yes that was the reason, now I have changed it to display the
publications based on your recent comment whose count will be
significantly very much lesser compared to the number of tables.

> 3.
> + $node_publisher->wait_for_catchup($sub_name);
> +
> + # also wait for initial table sync to finish
> + $node_subscriber->poll_query_until('postgres', $synced_query)
> +   or die "Timed out while waiting for subscriber to synchronize data";
> ....
> ....
> ....
> +$node_B->safe_psql(
> + 'postgres', "
> +        ALTER SUBSCRIPTION $subname_BA REFRESH PUBLICATION");
> +
> +$node_B->poll_query_until('postgres', $synced_query)
> +  or die "Timed out while waiting for subscriber to synchronize data";
>
> You can replace this and any similar code in the patch with a method
> used in commit 0c20dd33db.

Modified

> 4.
> +###############################################################################
> +# Join 3rd node (node_C) to the existing 2 nodes(node_A & node_B) bidirectional
> +# replication setup when the existing nodes (node_A & node_B) has pre-existing
> +# data and the new node (node_C) does not have any data.
> +###############################################################################
> +$result = $node_A->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
> +is($result, qq(), 'Check existing data');
> +
> +$result = $node_B->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
> +is($result, qq(), 'Check existing data');
> +
> +$result = $node_C->safe_psql('postgres', "SELECT * FROM tab ORDER BY 1;");
> +is($result, qq(), 'Check existing data');
>
> The comments say that for this test, two of the nodes have some
> pre-existing data but I don't see the same in the test results. The
> test following this one will also have a similar effect. BTW, I am not
> sure all the new three node tests added by this patch are required
> because I don't see if they have additional code coverage or test
> anything which is not tested without those. This has doubled the
> amount of test timing for this test file which is okay if these tests
> make any useful contribution but otherwise, it may be better to remove
> these. Am, I missing something?

Earlier we felt with the origin patch we could support joining of
nodes to existing n-wary replication setup in various scenarios, I had
added the tests for the same scenarios. But as the patch evolved, I
could understand that this patch cannot handle the add node scenarios.
I have removed these tests.

Thanks for the comments, the v44 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com

Regards,
Vignesh

  252. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-01T18:48:47Z

    On Wed, 31 Aug 2022 at 11:45, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v43-0002:
>
> ======
>
> 1. doc/src/sgml/ref/create_subscription.sgml
>
> @@ -403,7 +403,9 @@ CREATE SUBSCRIPTION <replaceable
> class="parameter">subscription_name</replaceabl
>     warning to notify the user to check the publisher tables. Before continuing
>     with other operations the user should check that publisher tables did not
>     have data with different origins to prevent data inconsistency issues on the
> -   subscriber.
> +   subscriber. Refer to <xref linkend="replication-between-primaries"/> for
> +   how <literal>copy_data</literal> and <literal>origin</literal> can be used
> +   to set up replication between primaries.
>    </para>
>
> Regarding my earlier v43-0001 review (see [1] comment #2) perhaps
> another pg docs section should be added in the
> logical-replication.sgml (e.g. "Specifying origins during CREATE
> SUBSCRIPTION"), so then this Notes text also should have more added to
> it.
>
> SUGGESTION
> Refer to <XXX_REF> for details about potential initialization
> inconsistency warnings using origin=NONE.
> Refer to <YYY_REF> for how copy_data and origin can be used to set up
> replication between primaries.

I have moved all these contents to a separate section in the
logical-replication page. I have referred to this link from the
documentation of origin and copy_data parameter. I have also referred
to "setting up replication between primaries" in the newly added
section. Since this new section is referred to from other places, I
felt we need not provide a link from create_subscription notes. The
changes for the same are available at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com

Regards,
Vignesh

  253. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-01T18:49:54Z

    On Thu, 1 Sept 2022 at 08:01, shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Wed, Aug 31, 2022 1:06 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v43 patch has the changes for the same.
> >
>
> Thanks for updating the patch.
>
> Here is a comment on the 0001 patch.
>
> +               if (!isnewtable)
> +               {
> +                       pfree(nspname);
> +                       pfree(relname);
> +                       continue;
> +               }
>
> If it is a new table, in which case it would log a warning, should we also call
> pfree()?

Yes it should be added, I have fixed the same in the v44 patch attached at [1].
[1] -  https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com

Regards,
Vignesh

  254. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-01T18:53:16Z

    On Thu, 1 Sept 2022 at 09:19, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 31, 2022 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my review comments for v43-0001.
> >
> > ======
> >
> > 1. Commit message
> >
> > The initial copy phase has no way to know the origin of the row data,
> > so if 'copy_data = true' in the step 4 below, log a warning to notify user
> > that potentially non-local data might have been copied.
> >
> > 1a.
> > "in the step 4" -> "in step 4"
> >
> > ~
> >
> > 1b.
> > "notify user" -> "notify the user"
> >
> > ======
> >
> > 2. doc/src/sgml/ref/create_subscription.sgml
> >
> > +  <para>
> > +   If the subscription is created with <literal>origin = none</literal> and
> > +   <literal>copy_data = true</literal>, it will check if the publisher has
> > +   subscribed to the same table from other publishers and, if so, log a
> > +   warning to notify the user to check the publisher tables. Before continuing
> > +   with other operations the user should check that publisher tables did not
> > +   have data with different origins to prevent data inconsistency issues on the
> > +   subscriber.
> > +  </para>
> >
> > I am not sure about that wording saying "to prevent data inconsistency
> > issues" because I thought maybe is already too late because any
> > unwanted origin data is already copied during the initial sync. I
> > think the user can do it try to clean up any problems caused before
> > things become even worse (??)
> >
> > ~~~
> >
> > You asked for my thoughts (see [1] 6b):
> >
> > > There is nothing much a user can do in this case. Only option would be
> > > to take a backup before the operation and restore it and then recreate
> > > the replication setup. I was not sure if we should document these
> > > steps as similar inconsistent data could get created without the
> > > origin option . Thoughts?
> >
> > I think those cases are a bit different:
> >
> > - For a normal scenario (i.e. origin=ANY) then inconsistent data just
> > refers to problems like maybe PK violations so I think you just get
> > what you get and have to deal with the errors...
> >
> > - But when the user said origin=NONE they are specifically saying they
> > do NOT want any non-local data, yet they still might end up getting
> > data contrary to their wishes. So I think maybe there ought to be
> > something documented to warn about this potentially happening. My
> > first impression is that all this seems like a kind of terrible
> > problem because if it happens then cleanup could be difficult - if
> > that subscriber in turn was also a publisher then this bad data might
> > have propagated all over the place already! Anyway, I guess all this
> > could be mentioned in some (new ?) section of the
> > logical-replication.sgml file (i.e. patch 0002).
> >
> > IDEA: I was wondering if the documentation should recommend (or maybe
> > we can even enforce this) that when using origin=NONE the user should
> > always create the subscription with enabled=FALSE. That way if there
> > are some warnings about potential bad origin data then there it might
> > be possible to take some action or change their mind *before* any
> > unwanted data pollutes everything.
> >
> > ======
> >
> > 3. src/backend/commands/subscriptioncmds.c - check_pub_table_subscribed
> >
> > + if (count == 0)
> > + tablenames = makeStringInfo();
> > + else
> > + appendStringInfoString(tablenames, ", ");
> > +
> > + appendStringInfo(tablenames, "%s.%s", nspname, relname);
> > + count++;
> >
> > I think the quotes are not correct - each separate table name should be quoted.
> >
> > ~~~
> >
> > 4.
> >
> > + /*
> > + * There might be many tables present in this case, we will display a
> > + * maximum of 100 tables followed by "..." to indicate there might be
> > + * more tables.
> > + */
> > + if (count == 100)
> > + {
> > + appendStringInfoString(tablenames, ", ...");
> > + break;
> > + }
> >
> > 4a.
> > IMO this code should be part of the previous if/else
> >
> > e.g.
> > if (count == 0) ... makeStringInfo()
> > else if (count > 100) ... append ellipsis "..." and break out of the loop
> > else ... append table name to the message
> >
> > Doing it in this way means "..." definitely means there are more than
> > 100 bad tables.
> >
> > ~
> >
> > 4b.
> > Changing like above (#4a) simplifies the comment because it removes
> > doubts like "might be…" since you already know you found the 101st
> > table.
> >
> > SUGGESTION (comment)
> > The message displays a maximum of 100 tables having potentially
> > non-local data. Any more than that will be indicated by "...".
> >
> > ~
> >
>
> Are we using this style of an error message anywhere else in the code?
> If not, then I think we should avoid it here as well as it appears a
> bit adhoc to me in the sense that why restrict at 100 tables. Can we
> instead think of displaying the publications list in the message? So
> errdetail would be something like: Subscribed publications \"%s\" has
> been subscribed from some other publisher. Then we can probably give a
> SQL query for a user to find tables that may data from multiple
> origins especially if that is not complicated. Also, then we can
> change the function name to check_publications_origin().

Modified as suggested.

> Few other minor comments on v43-0001*
> 1.
> + ereport(WARNING,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("subscription %s requested origin=NONE but might copy data
> that had a different origin.",
> + subname),
>
> In error message, we don't use full stop at the end.

Modified

> 2.
> + ereport(WARNING,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("subscription %s requested origin=NONE but might copy data
> that had a different origin.",
> + subname),
> + errdetail("Subscribed table(s) \"%s\" has been subscribed from some
> other publisher.",
> +   tablenames->data),
>
> It is better to use errdetail_plural here as there could be one or
> more objects in this message?

Modified the new errdetail message to errdetail_plural.

Thanks for the comments, the v44 version attached at [1] has the
changes for the same.
[1] -  https://www.postgresql.org/message-id/CALDaNm0NRJ1O1cYcZD%3Df7NgynozFprb7zpJSayFN5rcaS44G6Q%40mail.gmail.com

Regards,
Vignesh

  255. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-09-02T08:21:20Z

    Here are my review comments for the latest patch v44-0001.

======

1. doc/src/sgml/logical-replication.sgml

+ <sect1 id="specifying-origins-for-subscription">
+  <title>Specifying origins for subscription</title>

I thought the title is OK, but maybe can be better. OTOH, I am not
sure if my suggestions below are improvements or not. Anyway, even if
the title says the same, the convention is to use uppercase words.

Something like:
"Specifying Origins for Subscriptions"
"Specifying Data Origins for Subscriptions"
"Specifying Data Origins in CREATE SUBSCRIPTION"
"Subscription Data Origins"

~~~

2.

Currently, this new section in this patch is only discussing the
*problems* users might encounter for using copy_data,origin=NONE, but
I think a section with a generic title about "subscription origins"
should be able to stand alone. For example, it should give some brief
mention of what is the meaning/purpose of origin=ANY and origin=NONE.
And it should xref back to CREATE SUBSCRIPTION page.

IMO all this content currently in the patch maybe belongs in a
sub-section of this new section (e.g. call it something like
"Preventing Data Inconsistencies for copy_data,origin=NONE")

~~~

3.

+   To find which tables might potentially include non-local origins (due to
+   other subscriptions created on the publisher) try this SQL query by
+   specifying the publications in IN condition:
+<programlisting>
+SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
+FROM pg_publication P,
+     LATERAL pg_get_publication_tables(P.pubname) GPT
+     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
+     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
+WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
+      P.pubname IN (...);
+</programlisting>
+  </para>

3a.
I think the "To find..." sentence should be a new paragraph.

~

3b.
Instead of saying "specifying the publications in IN condition" I
think it might be simpler if those instructions are part of the SQL

SUGGESTION
+<programlisting>
+# substitute <pub-names> below with your publication name(s) to be queried
+SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
+FROM pg_publication P,
+     LATERAL pg_get_publication_tables(P.pubname) GPT
+     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
+     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
+WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
+      P.pubname IN (<pub-names>);
+</programlisting>

~

3c.
</programlisting>
  </para>

I recall there was a commit or hackers post sometime earlier this year
that reported it is better for these particular closing tags to be
done together like </programlisting></para> because otherwise there is
some case where the whitespace might not render correctly.
Unfortunately, I can't seem to find the evidence of that post/commit,
but I am sure I saw something about this because I have been using
that practice ever since I saw it.

======

4. doc/src/sgml/ref/alter_subscription.sgml

+         <para>
+          Refer to the <xref
linkend="specifying-origins-for-subscription"/> about how
+          <literal>copy_data = true</literal> can interact with the
+          <literal>origin</literal> parameter.
+         </para>

Previously when this xref pointed to the "Note" section the sentence
looked OK (it said, "Refer to the Note about how...") but now the word
is not "Note" anymore, (e.g. "Refer to the Section 31.3 about how
copy_data = true can interact with the origin parameter. "), so I
think this should be tweaked...

"Refer to the XXX about how..." -> "See XXX for details of how..."

======

5. doc/src/sgml/ref/create_subscription.sgml

Save as comment #4 (2 times)

======

6. src/backend/commands/subscriptioncmds.c

+ if (bsearch(&relid, subrel_local_oids,
+ subrel_count, sizeof(Oid), oid_cmp))
+ isnewtable = false;

SUGGESTION (search PG source - there are examples like this)
newtable = bsearch(&relid, subrel_local_oids, subrel_count,
sizeof(Oid), oid_cmp);

~~~

7.

+ if (list_length(publist))
+ {

I think the proper way to test for non-empty List is

if (publist != NIL) { ... }

or simply

if (publist) { ... }

~~~

8.

+ errmsg("subscription \"%s\" requested origin=NONE but might copy
data that had a different origin",

Do you think this should say "requested copy_data with origin=NONE",
instead of just "requested origin=NONE"?


------
Kind Regards,
Peter Smith.
Fujitsu Australia

  256. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-09-02T08:26:08Z

    On Fri, Sep 2, 2022 at 6:21 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the latest patch v44-0001.
>
...
>
> 6. src/backend/commands/subscriptioncmds.c
>
> + if (bsearch(&relid, subrel_local_oids,
> + subrel_count, sizeof(Oid), oid_cmp))
> + isnewtable = false;
>
> SUGGESTION (search PG source - there are examples like this)
> newtable = bsearch(&relid, subrel_local_oids, subrel_count,
> sizeof(Oid), oid_cmp);
>

Oops. Of course, I meant !bsearch

SUGGESTION
newtable = !bsearch(&relid, subrel_local_oids, subrel_count,
sizeof(Oid), oid_cmp);

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  257. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-02T10:29:01Z

    On Mon, 29 Aug 2022 at 16:35, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 24, 2022 at 7:27 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Since there was no objections to change it to throw a warning, I have
> > made the changes for the same.
> >
>
> Review comments for v42-0001*
> ==========================
> 1. Can we improve the query in check_pub_table_subscribed() so that it
> doesn't fetch any table that is already part of the subscription on
> the subscriber? This may be better than what the patch is doing which
> is to first fetch such information and then skip it. If forming this
> query turns out to be too complex then we can retain your method as
> well but I feel it is worth trying to optimize the query used in the
> patch.

I have modified the query to skip the tables as part of the query. The
attached v45 patch has the changes for the same.

Regards,
Vignesh

  258. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-02T10:33:44Z

    On Fri, 2 Sept 2022 at 13:51, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the latest patch v44-0001.
>
> ======
>
> 1. doc/src/sgml/logical-replication.sgml
>
> + <sect1 id="specifying-origins-for-subscription">
> +  <title>Specifying origins for subscription</title>
>
> I thought the title is OK, but maybe can be better. OTOH, I am not
> sure if my suggestions below are improvements or not. Anyway, even if
> the title says the same, the convention is to use uppercase words.
>
> Something like:
> "Specifying Origins for Subscriptions"
> "Specifying Data Origins for Subscriptions"
> "Specifying Data Origins in CREATE SUBSCRIPTION"
> "Subscription Data Origins"

Modified to "Preventing Data Inconsistencies for copy_data,
origin=NONE" to avoid confusions

> ~~~
>
> 2.
>
> Currently, this new section in this patch is only discussing the
> *problems* users might encounter for using copy_data,origin=NONE, but
> I think a section with a generic title about "subscription origins"
> should be able to stand alone. For example, it should give some brief
> mention of what is the meaning/purpose of origin=ANY and origin=NONE.
> And it should xref back to CREATE SUBSCRIPTION page.
>
> IMO all this content currently in the patch maybe belongs in a
> sub-section of this new section (e.g. call it something like
> "Preventing Data Inconsistencies for copy_data,origin=NONE")

I wanted to just document the inconsistency issue when copy_data and
origin is used. I felt the origin information was already present in
create subscription page. I have not documented the origin again here.
I have renamed the section to "Preventing Data Inconsistencies for
copy_data, origin=NONE" to avoid any confusions.

> ~~~
>
> 3.
>
> +   To find which tables might potentially include non-local origins (due to
> +   other subscriptions created on the publisher) try this SQL query by
> +   specifying the publications in IN condition:
> +<programlisting>
> +SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
> +FROM pg_publication P,
> +     LATERAL pg_get_publication_tables(P.pubname) GPT
> +     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
> +     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> +WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
> +      P.pubname IN (...);
> +</programlisting>
> +  </para>
>
> 3a.
> I think the "To find..." sentence should be a new paragraph.

modified

> ~
>
> 3b.
> Instead of saying "specifying the publications in IN condition" I
> think it might be simpler if those instructions are part of the SQL
>
> SUGGESTION
> +<programlisting>
> +# substitute <pub-names> below with your publication name(s) to be queried
> +SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
> +FROM pg_publication P,
> +     LATERAL pg_get_publication_tables(P.pubname) GPT
> +     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
> +     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> +WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
> +      P.pubname IN (<pub-names>);
> +</programlisting>

 modified, I could not use <pub-name> since it was considering it as a
tag, instead I have changed it to pub-names

> ~
>
> 3c.
> </programlisting>
>   </para>
>
> I recall there was a commit or hackers post sometime earlier this year
> that reported it is better for these particular closing tags to be
> done together like </programlisting></para> because otherwise there is
> some case where the whitespace might not render correctly.
> Unfortunately, I can't seem to find the evidence of that post/commit,
> but I am sure I saw something about this because I have been using
> that practice ever since I saw it.

Modified

> ======
>
> 4. doc/src/sgml/ref/alter_subscription.sgml
>
> +         <para>
> +          Refer to the <xref
> linkend="specifying-origins-for-subscription"/> about how
> +          <literal>copy_data = true</literal> can interact with the
> +          <literal>origin</literal> parameter.
> +         </para>
>
> Previously when this xref pointed to the "Note" section the sentence
> looked OK (it said, "Refer to the Note about how...") but now the word
> is not "Note" anymore, (e.g. "Refer to the Section 31.3 about how
> copy_data = true can interact with the origin parameter. "), so I
> think this should be tweaked...
>
> "Refer to the XXX about how..." -> "See XXX for details of how..."

Modified

> ======
>
> 5. doc/src/sgml/ref/create_subscription.sgml
>
> Save as comment #4 (2 times)

Modified

> ======
>
> 6. src/backend/commands/subscriptioncmds.c
>
> + if (bsearch(&relid, subrel_local_oids,
> + subrel_count, sizeof(Oid), oid_cmp))
> + isnewtable = false;
>
> SUGGESTION (search PG source - there are examples like this)
> newtable = bsearch(&relid, subrel_local_oids, subrel_count,
> sizeof(Oid), oid_cmp);

This code has been removed as part of another comment, the comment no
more applies.

> ~~~
>
> 7.
>
> + if (list_length(publist))
> + {
>
> I think the proper way to test for non-empty List is
>
> if (publist != NIL) { ... }
>
> or simply
>
> if (publist) { ... }

Modified

> ~~~
>
> 8.
>
> + errmsg("subscription \"%s\" requested origin=NONE but might copy
> data that had a different origin",
>
> Do you think this should say "requested copy_data with origin=NONE",
> instead of just "requested origin=NONE"?

Modified

Thanks for the comments, the v45 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm3fwW4mHGfpZfVLaHe_tSavOSPqntD5XPwO%2B_jwScrj_g%40mail.gmail.com

Regards,
Vignesh

  259. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-09-05T04:17:05Z

    Here are my review comments for v45-0001:

======

1. doc/src/sgml/logical-replication.sgml

  <para>
   To find which tables might potentially include non-local origins (due to
   other subscriptions created on the publisher) try this SQL query:
<programlisting>
SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
FROM pg_publication P,
     LATERAL pg_get_publication_tables(P.pubname) GPT
     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
      P.pubname IN (pub-names);
</programlisting></para>

1a.
To use "<pub-names>" with the <> then simply put meta characters in the SGML.
e.g.
&lt;pub-names&gt;

~

1b.
The patch forgot to add the SQL "#" instruction as suggested by my
previous comment (see [1] #3b)

~~~

2.

 <sect1 id="preventing-inconsistencies-for-copy_data-origin">
  <title>Preventing Data Inconsistencies for copy_data, origin=NONE</title>

The title is OK, but I think this should all be a <sect2> sub-section
of "31.2 Subscription"

======

3. src/backend/commands/subscriptioncmds.c - check_publications_origin

+ initStringInfo(&cmd);
+ appendStringInfoString(&cmd,
+    "SELECT DISTINCT P.pubname AS pubname\n"
+    "FROM pg_publication P,\n"
+    " LATERAL pg_get_publication_tables(P.pubname) GPT\n"
+    " LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
+    " pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
+    "WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN (");
+ get_publications_str(publications, &cmd, true);
+ appendStringInfoChar(&cmd, ')');
+ get_skip_tables_str(subrel_local_oids, subrel_count, &cmd);

(see from get_skip_tables_str)

+ appendStringInfoString(dest, " AND C.oid NOT IN (SELECT C.oid FROM
pg_class C JOIN pg_namespace N on N.oid = C.relnamespace where ");


IMO the way you are using get_skip_tables_str should be modified. I
will show by way of example below.
- "where" -> "WHERE"
- put the SELECT at the caller instead of inside the function
- handle the ")" at the caller

All this will also make the body of the 'get_skip_tables_str' function
much simpler (see the next review comments)

SUGGESTION
if (subrel_count > 0)
{
/* TODO - put some explanatory comment here about skipping the tables */
appendStringInfo(&cmd,
" AND C.oid NOT IN (\n"
"SELECT C.oid FROM pg_class C\n"
"JOIN pg_namespace N on N.oid = C.relnamespace WHERE ");
get_skip_tables_str(subrel_local_oids, subrel_count, &cmd);
appendStringInf(&cmd, “)”);
}

~~~

4. src/backend/commands/subscriptioncmds.c - get_skip_tables_str

+/*
+ * Add the table names that should be skipped.
+ */

This comment does not have enough detail to know really what the
function is for. Perhaps you only need to say that this is a helper
function for 'check_publications_origin' and then where it is called
you can give a comment (e.g. see my review comment #3)

~~

5. get_skip_tables_str (body)

5a. Variable 'count' is not a very good name; IMO just say 'i' for
index, and it can be declared C99 style.

~

5b. Variable 'first' is not necessary - it's same as (i == 0)

~

5c. The whole "SELECT" part and the ")" parts are more simply done at
the caller (see the review comment #3)

~

5d.

+ appendStringInfo(dest, "(C.relname = '%s' and N.nspname = '%s')",
+ tablename, schemaname);

It makes no difference but I thought it would feel more natural if the
SQL would compare the schema name *before* the table name.

~

5e.
"and" -> "AND"

~

Doing all 5a,b,c,d, and e means overall having a much simpler function body:

SUGGESTION
+ for (int i = 0; i < subrel_count; i++)
+ {
+ Oid relid = subrel_local_oids[i];
+ char    *schemaname = get_namespace_name(get_rel_namespace(relid));
+ char    *tablename = get_rel_name(relid);
+
+ if (i > 0)
+ appendStringInfoString(dest, " OR ");
+
+ appendStringInfo(dest, "(N.nspname = '%s' AND C.relname = '%s')",
+ schemaname, tablename);
+ }

------
[1] https://www.postgresql.org/message-id/CAHut%2BPsku25%2BVjz7HiohWxc2WU07O_ZV4voFG%2BU7WzwKhUzpGQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  260. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-05T09:40:41Z

    On Mon, Sep 5, 2022 at 9:47 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v45-0001:
>
> ======
>
> 1. doc/src/sgml/logical-replication.sgml
>
>   <para>
>    To find which tables might potentially include non-local origins (due to
>    other subscriptions created on the publisher) try this SQL query:
> <programlisting>
> SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
> FROM pg_publication P,
>      LATERAL pg_get_publication_tables(P.pubname) GPT
>      LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
>      pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
>       P.pubname IN (pub-names);
> </programlisting></para>
>
> 1a.
> To use "<pub-names>" with the <> then simply put meta characters in the SGML.
> e.g.
> &lt;pub-names&gt;
>
> ~
>
> 1b.
> The patch forgot to add the SQL "#" instruction as suggested by my
> previous comment (see [1] #3b)
>
> ~~~
>
> 2.
>
>  <sect1 id="preventing-inconsistencies-for-copy_data-origin">
>   <title>Preventing Data Inconsistencies for copy_data, origin=NONE</title>
>
> The title is OK, but I think this should all be a <sect2> sub-section
> of "31.2 Subscription"
>
> ======
>

It is recommended to create the subscription
+   using <literal>enabled=false</literal>, so that if the origin WARNING occurs
+   no copy has happened yet. Otherwise some corrective steps might be needed to
+   remove any unwanted data that got copied.

I am not completely sure of this part of the docs as this can add
additional steps for users while working on subscriptions even when
the same is not required. I suggest for now we can remove this part.
Later based on some feedback on this feature, we can extend the docs
if required.

Also, instead of having it as a separate section, let's keep this as
part of create_subscription.sgml

*
+ errhint("Verify that initial data copied from the publisher tables
did not come from other origins. Some corrective action may be
necessary."));

The second sentence in this message doesn't seem to be required.


-- 
With Regards,
Amit Kapila.

  261. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-06T03:13:38Z

    On Mon, 5 Sept 2022 at 09:47, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v45-0001:
>
> ======
>
> 1. doc/src/sgml/logical-replication.sgml
>
>   <para>
>    To find which tables might potentially include non-local origins (due to
>    other subscriptions created on the publisher) try this SQL query:
> <programlisting>
> SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
> FROM pg_publication P,
>      LATERAL pg_get_publication_tables(P.pubname) GPT
>      LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
>      pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
>       P.pubname IN (pub-names);
> </programlisting></para>
>
> 1a.
> To use "<pub-names>" with the <> then simply put meta characters in the SGML.
> e.g.
> &lt;pub-names&gt;

Modified

> ~
>
> 1b.
> The patch forgot to add the SQL "#" instruction as suggested by my
> previous comment (see [1] #3b)

Modified

> ~~~
>
> 2.
>
>  <sect1 id="preventing-inconsistencies-for-copy_data-origin">
>   <title>Preventing Data Inconsistencies for copy_data, origin=NONE</title>
>
> The title is OK, but I think this should all be a <sect2> sub-section
> of "31.2 Subscription"

I have moved it to create subscription notes based on a recent comment
from Amit.

> ======
>
> 3. src/backend/commands/subscriptioncmds.c - check_publications_origin
>
> + initStringInfo(&cmd);
> + appendStringInfoString(&cmd,
> +    "SELECT DISTINCT P.pubname AS pubname\n"
> +    "FROM pg_publication P,\n"
> +    " LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> +    " LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> +    " pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> +    "WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN (");
> + get_publications_str(publications, &cmd, true);
> + appendStringInfoChar(&cmd, ')');
> + get_skip_tables_str(subrel_local_oids, subrel_count, &cmd);
>
> (see from get_skip_tables_str)
>
> + appendStringInfoString(dest, " AND C.oid NOT IN (SELECT C.oid FROM
> pg_class C JOIN pg_namespace N on N.oid = C.relnamespace where ");
>
>
> IMO the way you are using get_skip_tables_str should be modified. I
> will show by way of example below.
> - "where" -> "WHERE"
> - put the SELECT at the caller instead of inside the function
> - handle the ")" at the caller
>
> All this will also make the body of the 'get_skip_tables_str' function
> much simpler (see the next review comments)
>
> SUGGESTION
> if (subrel_count > 0)
> {
> /* TODO - put some explanatory comment here about skipping the tables */
> appendStringInfo(&cmd,
> " AND C.oid NOT IN (\n"
> "SELECT C.oid FROM pg_class C\n"
> "JOIN pg_namespace N on N.oid = C.relnamespace WHERE ");
> get_skip_tables_str(subrel_local_oids, subrel_count, &cmd);
> appendStringInf(&cmd, “)”);
> }

Modified

> ~~~
>
> 4. src/backend/commands/subscriptioncmds.c - get_skip_tables_str
>
> +/*
> + * Add the table names that should be skipped.
> + */
>
> This comment does not have enough detail to know really what the
> function is for. Perhaps you only need to say that this is a helper
> function for 'check_publications_origin' and then where it is called
> you can give a comment (e.g. see my review comment #3)

Modified

> ~~
>
> 5. get_skip_tables_str (body)
>
> 5a. Variable 'count' is not a very good name; IMO just say 'i' for
> index, and it can be declared C99 style.

Modified

> ~
>
> 5b. Variable 'first' is not necessary - it's same as (i == 0)

Modified

> ~
>
> 5c. The whole "SELECT" part and the ")" parts are more simply done at
> the caller (see the review comment #3)

Modified

> ~
>
> 5d.
>
> + appendStringInfo(dest, "(C.relname = '%s' and N.nspname = '%s')",
> + tablename, schemaname);
>
> It makes no difference but I thought it would feel more natural if the
> SQL would compare the schema name *before* the table name.

Modified

> ~
>
> 5e.
> "and" -> "AND"

Modified

> ~
>
> Doing all 5a,b,c,d, and e means overall having a much simpler function body:
>
> SUGGESTION
> + for (int i = 0; i < subrel_count; i++)
> + {
> + Oid relid = subrel_local_oids[i];
> + char    *schemaname = get_namespace_name(get_rel_namespace(relid));
> + char    *tablename = get_rel_name(relid);
> +
> + if (i > 0)
> + appendStringInfoString(dest, " OR ");
> +
> + appendStringInfo(dest, "(N.nspname = '%s' AND C.relname = '%s')",
> + schemaname, tablename);
> + }

Modified

Thanks for the comments, the attached patch has the changes for the same.

Regards,
Vignesh

  262. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-06T03:15:45Z

    On Mon, 5 Sept 2022 at 15:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Sep 5, 2022 at 9:47 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my review comments for v45-0001:
> >
> > ======
> >
> > 1. doc/src/sgml/logical-replication.sgml
> >
> >   <para>
> >    To find which tables might potentially include non-local origins (due to
> >    other subscriptions created on the publisher) try this SQL query:
> > <programlisting>
> > SELECT DISTINCT N.nspname AS schemaname, C.relname AS tablename
> > FROM pg_publication P,
> >      LATERAL pg_get_publication_tables(P.pubname) GPT
> >      LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
> >      pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> > WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND
> >       P.pubname IN (pub-names);
> > </programlisting></para>
> >
> > 1a.
> > To use "<pub-names>" with the <> then simply put meta characters in the SGML.
> > e.g.
> > &lt;pub-names&gt;
> >
> > ~
> >
> > 1b.
> > The patch forgot to add the SQL "#" instruction as suggested by my
> > previous comment (see [1] #3b)
> >
> > ~~~
> >
> > 2.
> >
> >  <sect1 id="preventing-inconsistencies-for-copy_data-origin">
> >   <title>Preventing Data Inconsistencies for copy_data, origin=NONE</title>
> >
> > The title is OK, but I think this should all be a <sect2> sub-section
> > of "31.2 Subscription"
> >
> > ======
> >
>
> It is recommended to create the subscription
> +   using <literal>enabled=false</literal>, so that if the origin WARNING occurs
> +   no copy has happened yet. Otherwise some corrective steps might be needed to
> +   remove any unwanted data that got copied.
>
> I am not completely sure of this part of the docs as this can add
> additional steps for users while working on subscriptions even when
> the same is not required. I suggest for now we can remove this part.
> Later based on some feedback on this feature, we can extend the docs
> if required.

Modified

> Also, instead of having it as a separate section, let's keep this as
> part of create_subscription.sgml

Modified

> *
> + errhint("Verify that initial data copied from the publisher tables
> did not come from other origins. Some corrective action may be
> necessary."));
>
> The second sentence in this message doesn't seem to be required.

Modified

Thanks for the comments, the v46 patch at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm3TTGdCCkeDsN8hqtF_2z-8%2B%3D3tc9Gh5xOKAQ_BRMCkMA%40mail.gmail.com

Regards,
Vignesh

  263. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-09-06T04:00:47Z

    On Tue, Sep 6, 2022 11:14 AM vignesh C <vignesh21@gmail.com> wrote:
> 
> Thanks for the comments, the attached patch has the changes for the same.
> 

Thanks for updating the patch. Here are some comments.

1.
+	if (subrel_count)
+	{
+		/*
+		 * In case of ALTER SUBSCRIPTION ... REFRESH, subrel_local_oids
+		 * contains the list of relation oids that are already present on the
+		 * subscriber. This check should be skipped for these tables.
+		 */
+		appendStringInfo(&cmd, "AND C.oid NOT IN (\n"
+						 "SELECT C.oid \n"
+						 "FROM pg_class C\n"
+						 "     JOIN pg_namespace N ON N.oid = C.relnamespace\n"
+						 "WHERE ");
+		get_skip_tables_str(subrel_local_oids, subrel_count, &cmd);
+		appendStringInfo(&cmd, ")");
+	}

I think we can skip the tables without using a subquery. See the SQL below.
Would it be better?

SELECT DISTINCT P.pubname AS pubname
FROM pg_publication P,
         LATERAL pg_get_publication_tables(P.pubname) GPT
         LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
         pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN ('p1', 'p3')
AND NOT (N.nspname = 'public' AND C.relname = 'tbl')
AND NOT (N.nspname = 'public' AND C.relname = 't1');


2.
+   When using a subscription parameter combination of
+   <literal>copy_data=true</literal> and <literal>origin=NONE</literal>, the

Could we use "copy_data = true" and "origin = NONE" (add two spaces around the
equals sign)? I think it looks clearer that way. And it is consistent with other
places in this file.

Regards,
Shi yu

  264. RE: Handle infinite recursion in logical replication setup

    Wei Wang (Fujitsu) <wangw.fnst@fujitsu.com> — 2022-09-06T04:01:04Z

    On Tues, 6 Sept 2022 at 11:14, vignesh C <vignesh21@gmail.com> wrote:
> Thanks for the comments, the attached patch has the changes for the same.

Thanks for updating the patch.

Here is one comment for 0001 patch.
1. The query in function check_publications_origin.
+	appendStringInfoString(&cmd,
+						   "SELECT DISTINCT P.pubname AS pubname\n"
+						   "FROM pg_publication P,\n"
+						   "	 LATERAL pg_get_publication_tables(P.pubname) GPT\n"
+						   "	 LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
+						   "	 pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
+						   "WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN (");

Since I found that we only use "PS.srrelid" in the WHERE statement by
specifying "PS.srrelid IS NOT NULL", could we just use "[INNER] JOIN" to join
the table pg_subscription_rel?

Regards,
Wang wei

  265. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2022-09-06T05:11:16Z

    Thanks for addressing my previous review comments. I have no more
comments for v46*.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  266. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-06T09:55:10Z

    On Tue, Sep 6, 2022 at 9:31 AM wangw.fnst@fujitsu.com
<wangw.fnst@fujitsu.com> wrote:
>
> On Tues, 6 Sept 2022 at 11:14, vignesh C <vignesh21@gmail.com> wrote:
> > Thanks for the comments, the attached patch has the changes for the same.
>
> Thanks for updating the patch.
>
> Here is one comment for 0001 patch.
> 1. The query in function check_publications_origin.
> +       appendStringInfoString(&cmd,
> +                                                  "SELECT DISTINCT P.pubname AS pubname\n"
> +                                                  "FROM pg_publication P,\n"
> +                                                  "     LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> +                                                  "     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> +                                                  "     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> +                                                  "WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN (");
>
> Since I found that we only use "PS.srrelid" in the WHERE statement by
> specifying "PS.srrelid IS NOT NULL", could we just use "[INNER] JOIN" to join
> the table pg_subscription_rel?
>

I also think we can use INNER JOIN here but maybe there is a reason
why that is not used in the first place. If we change it in the code
then also let's change the same in the docs section as well.

Few minor points:
===============
1.
This scenario is detected and a WARNING is logged to the
+   user, but the warning is only an indication of a potential problem; it is
+   the user responsibility to make the necessary checks to ensure the copied
+   data origins are really as wanted or not.
+  </para>

/user/user's

2. How about a commit message like:
Raise a warning if there is a possibility of data from multiple origins.

This commit raises a warning message for a combination of options
('copy_data = true' and 'origin = none') during CREATE/ALTER subscription
operations if the publication tables were also replicated from other
publishers.

During replication, we can skip the data from other origins as we have that
information in WAL but that is not possible during initial sync so we raise
a warning if there is such a possibility.

-- 
With Regards,
Amit Kapila.

  267. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-07T04:22:53Z

    On Tue, 6 Sept 2022 at 09:31, shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Tue, Sep 6, 2022 11:14 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached patch has the changes for the same.
> >
>
> Thanks for updating the patch. Here are some comments.
>
> 1.
> +       if (subrel_count)
> +       {
> +               /*
> +                * In case of ALTER SUBSCRIPTION ... REFRESH, subrel_local_oids
> +                * contains the list of relation oids that are already present on the
> +                * subscriber. This check should be skipped for these tables.
> +                */
> +               appendStringInfo(&cmd, "AND C.oid NOT IN (\n"
> +                                                "SELECT C.oid \n"
> +                                                "FROM pg_class C\n"
> +                                                "     JOIN pg_namespace N ON N.oid = C.relnamespace\n"
> +                                                "WHERE ");
> +               get_skip_tables_str(subrel_local_oids, subrel_count, &cmd);
> +               appendStringInfo(&cmd, ")");
> +       }
>
> I think we can skip the tables without using a subquery. See the SQL below.
> Would it be better?
>
> SELECT DISTINCT P.pubname AS pubname
> FROM pg_publication P,
>          LATERAL pg_get_publication_tables(P.pubname) GPT
>          LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),
>          pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
> WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN ('p1', 'p3')
> AND NOT (N.nspname = 'public' AND C.relname = 'tbl')
> AND NOT (N.nspname = 'public' AND C.relname = 't1');

Modified

> 2.
> +   When using a subscription parameter combination of
> +   <literal>copy_data=true</literal> and <literal>origin=NONE</literal>, the
>
> Could we use "copy_data = true" and "origin = NONE" (add two spaces around the
> equals sign)? I think it looks clearer that way. And it is consistent with other
> places in this file.

Modified

Thanks for the comments, the attached v47 patch has the changes for the same.

Regards,
Vignesh

  268. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-07T04:24:09Z

    On Tue, 6 Sept 2022 at 09:31, wangw.fnst@fujitsu.com
<wangw.fnst@fujitsu.com> wrote:
>
> On Tues, 6 Sept 2022 at 11:14, vignesh C <vignesh21@gmail.com> wrote:
> > Thanks for the comments, the attached patch has the changes for the same.
>
> Thanks for updating the patch.
>
> Here is one comment for 0001 patch.
> 1. The query in function check_publications_origin.
> +       appendStringInfoString(&cmd,
> +                                                  "SELECT DISTINCT P.pubname AS pubname\n"
> +                                                  "FROM pg_publication P,\n"
> +                                                  "     LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> +                                                  "     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> +                                                  "     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> +                                                  "WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN (");
>
> Since I found that we only use "PS.srrelid" in the WHERE statement by
> specifying "PS.srrelid IS NOT NULL", could we just use "[INNER] JOIN" to join
> the table pg_subscription_rel?

Thanks for the comment, the v47 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm33T%3D23P-GWvy3O7cT1BfHuGV8dosAw1AVLf40MPvg2bg%40mail.gmail.com

Regards,
Vignesh

  269. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-07T04:25:16Z

    On Tue, 6 Sept 2022 at 15:25, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Sep 6, 2022 at 9:31 AM wangw.fnst@fujitsu.com
> <wangw.fnst@fujitsu.com> wrote:
> >
> > On Tues, 6 Sept 2022 at 11:14, vignesh C <vignesh21@gmail.com> wrote:
> > > Thanks for the comments, the attached patch has the changes for the same.
> >
> > Thanks for updating the patch.
> >
> > Here is one comment for 0001 patch.
> > 1. The query in function check_publications_origin.
> > +       appendStringInfoString(&cmd,
> > +                                                  "SELECT DISTINCT P.pubname AS pubname\n"
> > +                                                  "FROM pg_publication P,\n"
> > +                                                  "     LATERAL pg_get_publication_tables(P.pubname) GPT\n"
> > +                                                  "     LEFT JOIN pg_subscription_rel PS ON (GPT.relid = PS.srrelid),\n"
> > +                                                  "     pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)\n"
> > +                                                  "WHERE C.oid = GPT.relid AND PS.srrelid IS NOT NULL AND P.pubname IN (");
> >
> > Since I found that we only use "PS.srrelid" in the WHERE statement by
> > specifying "PS.srrelid IS NOT NULL", could we just use "[INNER] JOIN" to join
> > the table pg_subscription_rel?
> >
>
> I also think we can use INNER JOIN here but maybe there is a reason
> why that is not used in the first place. If we change it in the code
> then also let's change the same in the docs section as well.

Modified

> Few minor points:
> ===============
> 1.
> This scenario is detected and a WARNING is logged to the
> +   user, but the warning is only an indication of a potential problem; it is
> +   the user responsibility to make the necessary checks to ensure the copied
> +   data origins are really as wanted or not.
> +  </para>
>
> /user/user's

Modified

> 2. How about a commit message like:
> Raise a warning if there is a possibility of data from multiple origins.
>
> This commit raises a warning message for a combination of options
> ('copy_data = true' and 'origin = none') during CREATE/ALTER subscription
> operations if the publication tables were also replicated from other
> publishers.
>
> During replication, we can skip the data from other origins as we have that
> information in WAL but that is not possible during initial sync so we raise
> a warning if there is such a possibility.

Yes, this looks good. Modified

Thanks for the comment, the v47 patch attached at [1] has the changes
for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm33T%3D23P-GWvy3O7cT1BfHuGV8dosAw1AVLf40MPvg2bg%40mail.gmail.com

Regards,
Vignesh

  270. RE: Handle infinite recursion in logical replication setup

    Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com> — 2022-09-07T07:39:23Z

    On Wed, Sep 7, 2022 12:23 PM vignesh C <vignesh21@gmail.com> wrote:
> 
> 
> Thanks for the comments, the attached v47 patch has the changes for the
> same.
> 

Thanks for updating the patch.

Here is a comment.

+	for (i = 0; i < subrel_count; i++)
+	{
+		Oid			relid = subrel_local_oids[i];
+		char	   *schemaname = get_namespace_name(get_rel_namespace(relid));
+		char	   *tablename = get_rel_name(relid);
+
+		appendStringInfo(&cmd, "AND NOT (N.nspname = '%s' AND C.relname = '%s')\n",
+						 schemaname, tablename);
+	}

Would it be better to add "pfree(schemaname)" and "pfree(tablename)" after
calling appendStringInfo()?

Regards,
Shi yu

  271. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-07T08:09:14Z

    On Wed, Sep 7, 2022 at 1:09 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
>
> On Wed, Sep 7, 2022 12:23 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> >
> > Thanks for the comments, the attached v47 patch has the changes for the
> > same.
> >
>
> Thanks for updating the patch.
>
> Here is a comment.
>
> +       for (i = 0; i < subrel_count; i++)
> +       {
> +               Oid                     relid = subrel_local_oids[i];
> +               char       *schemaname = get_namespace_name(get_rel_namespace(relid));
> +               char       *tablename = get_rel_name(relid);
> +
> +               appendStringInfo(&cmd, "AND NOT (N.nspname = '%s' AND C.relname = '%s')\n",
> +                                                schemaname, tablename);
> +       }
>
> Would it be better to add "pfree(schemaname)" and "pfree(tablename)" after
> calling appendStringInfo()?
>

No, I don't think we need to do retail pfree of each and every
allocation as these allocations are made in the portal context which
will be freed by the command end.

-- 
With Regards,
Amit Kapila.

  272. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-07T09:04:02Z

    On Wed, Sep 7, 2022 at 9:53 AM vignesh C <vignesh21@gmail.com> wrote:
>
> Thanks for the comments, the attached v47 patch has the changes for the same.
>

V47-0001* looks good to me apart from below minor things. I would like
to commit this tomorrow unless there are more comments on it.

Few minor suggestions:
==================
1.
+ list_free_deep(publist);
+ pfree(pubnames->data);
+ pfree(pubnames);

I don't think we need to free this memory as it will automatically be
released at end of the command (this memory is allocated in portal
context). I understand there is no harm in freeing it as well but
better to leave it as there doesn't appear to be any danger of leaking
memory for a longer time.

2.
+ res = walrcv_exec(wrconn, cmd.data, 1, tableRow);
+ pfree(cmd.data);

Don't you need to free cmd as well here? I think it is better to avoid
freeing it due to reasons mentioned in the previous point.

-- 
With Regards,
Amit Kapila.

  273. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-07T11:40:14Z

    On Wed, 7 Sept 2022 at 14:34, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Sep 7, 2022 at 9:53 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for the comments, the attached v47 patch has the changes for the same.
> >
>
> V47-0001* looks good to me apart from below minor things. I would like
> to commit this tomorrow unless there are more comments on it.
>
> Few minor suggestions:
> ==================
> 1.
> + list_free_deep(publist);
> + pfree(pubnames->data);
> + pfree(pubnames);
>
> I don't think we need to free this memory as it will automatically be
> released at end of the command (this memory is allocated in portal
> context). I understand there is no harm in freeing it as well but
> better to leave it as there doesn't appear to be any danger of leaking
> memory for a longer time.

Modified

> 2.
> + res = walrcv_exec(wrconn, cmd.data, 1, tableRow);
> + pfree(cmd.data);
>
> Don't you need to free cmd as well here? I think it is better to avoid
> freeing it due to reasons mentioned in the previous point.

cmd need not be freed here as we use initStringInfo for initialization
and initStringInfo allocates memory for data member only.

The attached patch has the changes for the same.

Regards,
Vignesh

  274. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-08T02:26:45Z

    Hi,

There is a buildfarm failure on mylodon at [1] because of the new test
added. I will analyse and share the findings for the same.

[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mylodon&dt=2022-09-08%2001%3A40%3A27

Regards,
Vignesh

On Wed, 7 Sept 2022 at 17:10, vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 7 Sept 2022 at 14:34, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Sep 7, 2022 at 9:53 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Thanks for the comments, the attached v47 patch has the changes for the same.
> > >
> >
> > V47-0001* looks good to me apart from below minor things. I would like
> > to commit this tomorrow unless there are more comments on it.
> >
> > Few minor suggestions:
> > ==================
> > 1.
> > + list_free_deep(publist);
> > + pfree(pubnames->data);
> > + pfree(pubnames);
> >
> > I don't think we need to free this memory as it will automatically be
> > released at end of the command (this memory is allocated in portal
> > context). I understand there is no harm in freeing it as well but
> > better to leave it as there doesn't appear to be any danger of leaking
> > memory for a longer time.
>
> Modified
>
> > 2.
> > + res = walrcv_exec(wrconn, cmd.data, 1, tableRow);
> > + pfree(cmd.data);
> >
> > Don't you need to free cmd as well here? I think it is better to avoid
> > freeing it due to reasons mentioned in the previous point.
>
> cmd need not be freed here as we use initStringInfo for initialization
> and initStringInfo allocates memory for data member only.
>
> The attached patch has the changes for the same.
>
> Regards,
> Vignesh

  275. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-08T02:53:00Z

    On Thu, Sep 8, 2022 at 7:57 AM vignesh C <vignesh21@gmail.com> wrote:
>
> There is a buildfarm failure on mylodon at [1] because of the new test
> added. I will analyse and share the findings for the same.
>
> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mylodon&dt=2022-09-08%2001%3A40%3A27
>

The log is as below:

error running SQL: 'psql:<stdin>:1: ERROR:  could not drop relation
mapping for subscription "tap_sub_a_b_2"
DETAIL:  Table synchronization for relation "tab_new" is in progress
and is in state "s".
HINT:  Use ALTER SUBSCRIPTION ... ENABLE to enable subscription if not
already enabled or use DROP SUBSCRIPTION ... to drop the
subscription.'
while running 'psql -XAtq -d port=50352 host=/tmp/WMoRd6ngw2
dbname='postgres' -f - -v ON_ERROR_STOP=1' with sql 'DROP TABLE
tab_new' at /mnt/resource/bf/build/mylodon/HEAD/pgsql.build/../pgsql/src/test/perl/PostgreSQL/Test/Cluster.pm
line 1860.
### Stopping node "node_A" using mode immediate

This clearly indicates the problem. We can't drop the relation till it
is marked ready in pg_subscription_rel and prior to dropping, the test
just does $node_A->wait_for_subscription_sync($node_B, $subname_AB2);.
This doesn't ensure that relation is in 'ready' state as it will
finish even when the relation is in 'syncdone' state.

So, I think if we want to drop the tables, we need to poll for 'ready'
state or otherwise, anyway, this is the end of all tests and nodes
won't be reused, so we can remove the clean-up in the end. Any other
ideas?

-- 
With Regards,
Amit Kapila.

  276. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-08T03:27:00Z

    On Thu, 8 Sept 2022 at 08:23, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Sep 8, 2022 at 7:57 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > There is a buildfarm failure on mylodon at [1] because of the new test
> > added. I will analyse and share the findings for the same.
> >
> > [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mylodon&dt=2022-09-08%2001%3A40%3A27
> >
>
> The log is as below:
>
> error running SQL: 'psql:<stdin>:1: ERROR:  could not drop relation
> mapping for subscription "tap_sub_a_b_2"
> DETAIL:  Table synchronization for relation "tab_new" is in progress
> and is in state "s".
> HINT:  Use ALTER SUBSCRIPTION ... ENABLE to enable subscription if not
> already enabled or use DROP SUBSCRIPTION ... to drop the
> subscription.'
> while running 'psql -XAtq -d port=50352 host=/tmp/WMoRd6ngw2
> dbname='postgres' -f - -v ON_ERROR_STOP=1' with sql 'DROP TABLE
> tab_new' at /mnt/resource/bf/build/mylodon/HEAD/pgsql.build/../pgsql/src/test/perl/PostgreSQL/Test/Cluster.pm
> line 1860.
> ### Stopping node "node_A" using mode immediate
>
> This clearly indicates the problem. We can't drop the relation till it
> is marked ready in pg_subscription_rel and prior to dropping, the test
> just does $node_A->wait_for_subscription_sync($node_B, $subname_AB2);.
> This doesn't ensure that relation is in 'ready' state as it will
> finish even when the relation is in 'syncdone' state.

I agree with the analysis, adding wait for ready state before dropping
the table approach looks good to me. I will prepare a patch for the
same and share it.

Regards,
Vignesh

  277. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-08T04:01:50Z

    On Thu, 8 Sept 2022 at 08:57, vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 8 Sept 2022 at 08:23, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Sep 8, 2022 at 7:57 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > There is a buildfarm failure on mylodon at [1] because of the new test
> > > added. I will analyse and share the findings for the same.
> > >
> > > [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mylodon&dt=2022-09-08%2001%3A40%3A27
> > >
> >
> > The log is as below:
> >
> > error running SQL: 'psql:<stdin>:1: ERROR:  could not drop relation
> > mapping for subscription "tap_sub_a_b_2"
> > DETAIL:  Table synchronization for relation "tab_new" is in progress
> > and is in state "s".
> > HINT:  Use ALTER SUBSCRIPTION ... ENABLE to enable subscription if not
> > already enabled or use DROP SUBSCRIPTION ... to drop the
> > subscription.'
> > while running 'psql -XAtq -d port=50352 host=/tmp/WMoRd6ngw2
> > dbname='postgres' -f - -v ON_ERROR_STOP=1' with sql 'DROP TABLE
> > tab_new' at /mnt/resource/bf/build/mylodon/HEAD/pgsql.build/../pgsql/src/test/perl/PostgreSQL/Test/Cluster.pm
> > line 1860.
> > ### Stopping node "node_A" using mode immediate
> >
> > This clearly indicates the problem. We can't drop the relation till it
> > is marked ready in pg_subscription_rel and prior to dropping, the test
> > just does $node_A->wait_for_subscription_sync($node_B, $subname_AB2);.
> > This doesn't ensure that relation is in 'ready' state as it will
> > finish even when the relation is in 'syncdone' state.
>
> I agree with the analysis, adding wait for ready state before dropping
> the table approach looks good to me. I will prepare a patch for the
> same and share it.

The attached patch has the changes to handle the same.

Regards,
Vignesh

  278. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2022-09-09T05:42:02Z

    On Thu, Sep 8, 2022 at 9:32 AM vignesh C <vignesh21@gmail.com> wrote:
>
>
> The attached patch has the changes to handle the same.
>

Pushed. I am not completely sure whether we want the remaining
documentation patch in this thread in its current form or by modifying
it. Johnathan has shown some interest in it. I feel you can start a
separate thread for it to see if there is any interest in the same and
close the CF entry for this work.

-- 
With Regards,
Amit Kapila.

  279. Re: Handle infinite recursion in logical replication setup

    vignesh C <vignesh21@gmail.com> — 2022-09-12T05:23:08Z

    On Fri, 9 Sept 2022 at 11:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Sep 8, 2022 at 9:32 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> >
> > The attached patch has the changes to handle the same.
> >
>
> Pushed. I am not completely sure whether we want the remaining
> documentation patch in this thread in its current form or by modifying
> it. Johnathan has shown some interest in it. I feel you can start a
> separate thread for it to see if there is any interest in the same and
> close the CF entry for this work.

Thanks for pushing the patch. I have closed this entry in commitfest.
I will wait for some more time and see the response regarding the
documentation patch and then start a new thread if required.

Regards,
Vignesh

  280. Re: Handle infinite recursion in logical replication setup

    Jonathan S. Katz <jkatz@postgresql.org> — 2023-01-10T02:43:15Z

    On 9/12/22 1:23 AM, vignesh C wrote:
> On Fri, 9 Sept 2022 at 11:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Thu, Sep 8, 2022 at 9:32 AM vignesh C <vignesh21@gmail.com> wrote:
>>>
>>>
>>> The attached patch has the changes to handle the same.
>>>
>>
>> Pushed. I am not completely sure whether we want the remaining
>> documentation patch in this thread in its current form or by modifying
>> it. Johnathan has shown some interest in it. I feel you can start a
>> separate thread for it to see if there is any interest in the same and
>> close the CF entry for this work.
> 
> Thanks for pushing the patch. I have closed this entry in commitfest.
> I will wait for some more time and see the response regarding the
> documentation patch and then start a new thread if required.

I've been testing this patch in advancing of working on the 
documentation and came across a behavior I wanted to note. Specifically, 
I am hitting a deadlock while trying to synchronous replicate between 
the two instances at any `synchronous_commit` level above `local`.

Here is my set up. I have two instances, "A" and "B".

On A and B, run:

   CREATE TABLE sync (id int PRIMARY KEY, info float);
   CREATE PUBLICATION sync FOR TABLE sync;

On A, run:

   CREATE SUBSCRIPTION sync
   CONNECTION 'connstr-to-B'
   PUBLICATION sync
   WITH (
     streaming=true, copy_data=false,
     origin=none, synchronous_commit='on');

On B, run:

   CREATE SUBSCRIPTION sync
   CONNECTION 'connstr-to-A'
   PUBLICATION sync
   WITH (
     streaming=true, copy_data=false,
     origin=none, synchronous_commit='on');

On A and B, run:

   ALTER SYSTEM SET synchronous_standby_names TO 'sync';
   SELECT pg_reload_conf();

Verify on A and B that pg_stat_replication.sync_state is set to "sync"

   SELECT application_name, sync_state = 'sync' AS is_sync
   FROM pg_stat_replication
   WHERE application_name = 'sync';

The next to commands should be run simultaneously on A and B:

-- run this on A
INSERT INTO sync
SELECT x, random() FROM generate_series(1,2000000, 2) x;

-- run this on B
INSERT INTO sync
SELECT x, random() FROM generate_series(2,2000000, 2) x;

This consistently created the deadlock in my testing.

Discussing with Masahiko off-list, this is due to a deadlock from 4 
processes: the walsenders on A and B, and the apply workers on A and B. 
The walsenders are waiting for feedback from the apply workers, and the 
apply workers are waiting for the walsenders to synchronize (I may be 
oversimplifying).

He suggested I try the above example instead with `synchronous_commit` 
set to `local`. In this case, I verified that there is no more deadlock, 
but he informed me that we would not be able to use cascading 
synchronous replication when "origin=none".

If we decide that this is a documentation issue, I'd suggest we improve 
the guidance around using `synchronous_commit`[1] on the CREATE 
SUBSCRIPTION page, as the GUC page[2] warns against using `local`:

"The setting local causes commits to wait for local flush to disk, but 
not for replication. This is usually not desirable when synchronous 
replication is in use, but is provided for completeness."

Thanks,

Jonathan

[1] https://www.postgresql.org/docs/devel/sql-createsubscription.html
[2] 
https://www.postgresql.org/docs/devel/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT

  281. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2023-01-10T15:17:13Z

    On Tue, Jan 10, 2023 at 8:13 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 9/12/22 1:23 AM, vignesh C wrote:
> > On Fri, 9 Sept 2022 at 11:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Thanks for pushing the patch. I have closed this entry in commitfest.
> > I will wait for some more time and see the response regarding the
> > documentation patch and then start a new thread if required.
>
> I've been testing this patch in advancing of working on the
> documentation and came across a behavior I wanted to note. Specifically,
> I am hitting a deadlock while trying to synchronous replicate between
> the two instances at any `synchronous_commit` level above `local`.
>
> Here is my set up. I have two instances, "A" and "B".
>
> On A and B, run:
>
>    CREATE TABLE sync (id int PRIMARY KEY, info float);
>    CREATE PUBLICATION sync FOR TABLE sync;
>
> On A, run:
>
>    CREATE SUBSCRIPTION sync
>    CONNECTION 'connstr-to-B'
>    PUBLICATION sync
>    WITH (
>      streaming=true, copy_data=false,
>      origin=none, synchronous_commit='on');
>
> On B, run:
>
>    CREATE SUBSCRIPTION sync
>    CONNECTION 'connstr-to-A'
>    PUBLICATION sync
>    WITH (
>      streaming=true, copy_data=false,
>      origin=none, synchronous_commit='on');
>
> On A and B, run:
>
>    ALTER SYSTEM SET synchronous_standby_names TO 'sync';
>    SELECT pg_reload_conf();
>
> Verify on A and B that pg_stat_replication.sync_state is set to "sync"
>
>    SELECT application_name, sync_state = 'sync' AS is_sync
>    FROM pg_stat_replication
>    WHERE application_name = 'sync';
>
> The next to commands should be run simultaneously on A and B:
>
> -- run this on A
> INSERT INTO sync
> SELECT x, random() FROM generate_series(1,2000000, 2) x;
>
> -- run this on B
> INSERT INTO sync
> SELECT x, random() FROM generate_series(2,2000000, 2) x;
>
> This consistently created the deadlock in my testing.
>
> Discussing with Masahiko off-list, this is due to a deadlock from 4
> processes: the walsenders on A and B, and the apply workers on A and B.
> The walsenders are waiting for feedback from the apply workers, and the
> apply workers are waiting for the walsenders to synchronize (I may be
> oversimplifying).
>
> He suggested I try the above example instead with `synchronous_commit`
> set to `local`. In this case, I verified that there is no more deadlock,
> but he informed me that we would not be able to use cascading
> synchronous replication when "origin=none".
>

This has nothing to do with the origin feature. I mean this should
happen with origin=any or even in PG15 without using origin at all.
Am, I missing something? One related point to note is that in physical
replication cascading replication is asynchronous. See docs [1]
(Cascading replication is currently asynchronous....)

> If we decide that this is a documentation issue, I'd suggest we improve
> the guidance around using `synchronous_commit`[1] on the CREATE
> SUBSCRIPTION page, as the GUC page[2] warns against using `local`:
>

Yeah, but on Create Subscription page, we have mentioned that it is
safe to use off for logical replication. One can use local or higher
for reducing the latency for COMMIT when synchronous replication is
used in the publisher. Won't using 'local' while creating subscription
would suffice the need to consistently replicate the data? I mean it
is equivalent to somebody using levels greater than local in case of
physical replication. I think in the case of physical replication, we
won't wait for standby to replicate to another node before sending a
response, so why to wait in the case of logical replication? If this
understanding is correct, then probably it is sufficient to support
'local' for a subscription.

[1] - https://www.postgresql.org/docs/devel/warm-standby.html

-- 
With Regards,
Amit Kapila.

  282. Re: Handle infinite recursion in logical replication setup

    Jonathan S. Katz <jkatz@postgresql.org> — 2023-01-10T17:54:28Z

    On 1/10/23 10:17 AM, Amit Kapila wrote:
> On Tue, Jan 10, 2023 at 8:13 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:

>> This consistently created the deadlock in my testing.
>>
>> Discussing with Masahiko off-list, this is due to a deadlock from 4
>> processes: the walsenders on A and B, and the apply workers on A and B.
>> The walsenders are waiting for feedback from the apply workers, and the
>> apply workers are waiting for the walsenders to synchronize (I may be
>> oversimplifying).
>>
>> He suggested I try the above example instead with `synchronous_commit`
>> set to `local`. In this case, I verified that there is no more deadlock,
>> but he informed me that we would not be able to use cascading
>> synchronous replication when "origin=none".
>>
> 
> This has nothing to do with the origin feature. I mean this should
> happen with origin=any or even in PG15 without using origin at all.
> Am, I missing something? One related point to note is that in physical
> replication cascading replication is asynchronous. See docs [1]
> (Cascading replication is currently asynchronous....)

This is not directly related to the origin feature, but the origin 
feature makes it apparent. There is a common use-case where people want 
to replicate data synchronously between two tables, and this is enabled 
by the "origin" feature.

To be clear, my bigger concern is that it's fairly easy for users to 
create a deadlock situation based on how they set "synchronous_commit" 
with the origin feature -- this is the main reason why I brought it up. 
I'm less concerned about the "cascading" case, though I want to try out 
sync rep between 3 instances to see what happens.

>> If we decide that this is a documentation issue, I'd suggest we improve
>> the guidance around using `synchronous_commit`[1] on the CREATE
>> SUBSCRIPTION page, as the GUC page[2] warns against using `local`:
>>
> 
> Yeah, but on Create Subscription page, we have mentioned that it is
> safe to use off for logical replication.

While I think you can infer that it's "safe" for synchronous 
replication, I don't think it's clear.

We say it's "safe to use `off` for logical replication", but provide a 
lengthy explanation around synchronous logical replication that 
concludes that it's "advantageous to set synchronous_commit to local or 
higher" but does not address safety. The first line in the explanation 
of the parameter links to the `synchronous_commit` GUC which 
specifically advises against using "local" for synchronous replication.

Additionally, because we say "local" or higher, we increase the risk of 
the aforementioned in HEAD with the origin feature.

I know I'm hammering on this point a bit, but it feels like this is 
relatively easy to misconfigure with the upcoming "origin" change (I did 
so myself from reading the devel docs) and we should ensure we guide our 
users appropriately.

> One can use local or higher
> for reducing the latency for COMMIT when synchronous replication is
> used in the publisher. Won't using 'local' while creating subscription
> would suffice the need to consistently replicate the data? I mean it
> is equivalent to somebody using levels greater than local in case of
> physical replication. I think in the case of physical replication, we
> won't wait for standby to replicate to another node before sending a
> response, so why to wait in the case of logical replication? If this
> understanding is correct, then probably it is sufficient to support
> 'local' for a subscription.

I think this is a good explanation. We should incorporate some version 
of this into the docs, and add some clarity around the use of 
`synchronous_commit` option in `CREATE SUBSCRIPTION` in particular with 
the origin feature. I can make an attempt at this.

Perhaps another question: based on this, should we disallow setting 
values of `synchronous_commit` greater than `local` when using 
"origin=none"? That might be too strict, but maybe we should warn around 
the risk of deadlock either in the logs or in the docs.

Thanks,

Jonathan

  283. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2023-01-11T02:16:42Z

    On Tue, Jan 10, 2023 at 11:24 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 1/10/23 10:17 AM, Amit Kapila wrote:
> > On Tue, Jan 10, 2023 at 8:13 AM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> > One can use local or higher
> > for reducing the latency for COMMIT when synchronous replication is
> > used in the publisher. Won't using 'local' while creating subscription
> > would suffice the need to consistently replicate the data? I mean it
> > is equivalent to somebody using levels greater than local in case of
> > physical replication. I think in the case of physical replication, we
> > won't wait for standby to replicate to another node before sending a
> > response, so why to wait in the case of logical replication? If this
> > understanding is correct, then probably it is sufficient to support
> > 'local' for a subscription.
>
> I think this is a good explanation. We should incorporate some version
> of this into the docs, and add some clarity around the use of
> `synchronous_commit` option in `CREATE SUBSCRIPTION` in particular with
> the origin feature. I can make an attempt at this.
>

Okay, thanks!

> Perhaps another question: based on this, should we disallow setting
> values of `synchronous_commit` greater than `local` when using
> "origin=none"?
>

I think it should be done irrespective of the value of origin because
it can create a deadlock in those cases as well. I think one idea as
you suggest is to block levels higher than local and the other is to
make it behave like 'local' even if the level is higher than 'local'
which would be somewhat similar to our physical replication.

> That might be too strict, but maybe we should warn around
> the risk of deadlock either in the logs or in the docs.
>

Yeah, we can mention it in docs as well. We can probably document as
part of the docs work for this thread but it would be better to start
a separate thread to fix this behavior by either of the above two
approaches.

-- 
With Regards,
Amit Kapila.

  284. Re: Handle infinite recursion in logical replication setup

    Peter Eisentraut <peter@eisentraut.org> — 2023-08-08T08:20:42Z

    On 12.09.22 07:23, vignesh C wrote:
> On Fri, 9 Sept 2022 at 11:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Thu, Sep 8, 2022 at 9:32 AM vignesh C <vignesh21@gmail.com> wrote:
>>>
>>>
>>> The attached patch has the changes to handle the same.
>>>
>>
>> Pushed. I am not completely sure whether we want the remaining
>> documentation patch in this thread in its current form or by modifying
>> it. Johnathan has shown some interest in it. I feel you can start a
>> separate thread for it to see if there is any interest in the same and
>> close the CF entry for this work.
> 
> Thanks for pushing the patch. I have closed this entry in commitfest.
> I will wait for some more time and see the response regarding the
> documentation patch and then start a new thread if required.

This patch added the following error message:

errdetail_plural("Subscribed publication %s is subscribing to other 
publications.",
"Subscribed publications %s are subscribing to other publications.",
list_length(publist), pubnames->data),

But in PostgreSQL, a publication cannot subscribe to a publication, so 
this is not giving accurate information.  Apparently, what it is trying 
to say is that

The subscription that you are creating subscribes to publications that 
contain tables that are written to by other subscriptions.

Can we get to a more accurate wording like this?

There is also a translatability issue there, in the way the publication 
list is pasted into the message.

Is the list of affected publications really that interesting?  I wonder 
whether the list of affected tables might be more relevant?

  285. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-08T08:52:43Z

    On Tue, Aug 8, 2023 at 1:50 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 12.09.22 07:23, vignesh C wrote:
> > On Fri, 9 Sept 2022 at 11:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> On Thu, Sep 8, 2022 at 9:32 AM vignesh C <vignesh21@gmail.com> wrote:
> >>>
> >>>
> >>> The attached patch has the changes to handle the same.
> >>>
> >>
> >> Pushed. I am not completely sure whether we want the remaining
> >> documentation patch in this thread in its current form or by modifying
> >> it. Johnathan has shown some interest in it. I feel you can start a
> >> separate thread for it to see if there is any interest in the same and
> >> close the CF entry for this work.
> >
> > Thanks for pushing the patch. I have closed this entry in commitfest.
> > I will wait for some more time and see the response regarding the
> > documentation patch and then start a new thread if required.
>
> This patch added the following error message:
>
> errdetail_plural("Subscribed publication %s is subscribing to other
> publications.",
> "Subscribed publications %s are subscribing to other publications.",
> list_length(publist), pubnames->data),
>
> But in PostgreSQL, a publication cannot subscribe to a publication, so
> this is not giving accurate information.  Apparently, what it is trying
> to say is that
>
> The subscription that you are creating subscribes to publications that
> contain tables that are written to by other subscriptions.
>
> Can we get to a more accurate wording like this?
>

+1 for changing the message as per your suggestion.

> There is also a translatability issue there, in the way the publication
> list is pasted into the message.
>
> Is the list of affected publications really that interesting?  I wonder
> whether the list of affected tables might be more relevant?
>

In that case, we need to specify both schema name and table name in
that case. I guess the list could be very long and not sure what to do
for schema publications ( Create Publication ... For Schema).

-- 
With Regards,
Amit Kapila.

  286. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2023-08-09T02:50:10Z

    On Tue, Aug 8, 2023 at 6:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Aug 8, 2023 at 1:50 PM Peter Eisentraut <peter@eisentraut.org> wrote:
> >
> > This patch added the following error message:
> >
> > errdetail_plural("Subscribed publication %s is subscribing to other
> > publications.",
> > "Subscribed publications %s are subscribing to other publications.",
> > list_length(publist), pubnames->data),
> >
> > But in PostgreSQL, a publication cannot subscribe to a publication, so
> > this is not giving accurate information.  Apparently, what it is trying
> > to say is that
> >
> > The subscription that you are creating subscribes to publications that
> > contain tables that are written to by other subscriptions.
> >
> > Can we get to a more accurate wording like this?
> >
>
> +1 for changing the message as per your suggestion.
>

PSA a patch to change this message text. The message now has wording
similar to the suggestion.

> > There is also a translatability issue there, in the way the publication
> > list is pasted into the message.
> >

The name/list substitution is now done within parentheses, which AFAIK
will be enough to eliminate any translation ambiguities.

> > Is the list of affected publications really that interesting?  I wonder
> > whether the list of affected tables might be more relevant?
> >
>
> In that case, we need to specify both schema name and table name in
> that case. I guess the list could be very long and not sure what to do
> for schema publications ( Create Publication ... For Schema).

Right, IIUC that was the reason for not choosing to show the tables in
the original patch -- i.e. the list might easily be very long with
100s or 1000s of tables it, and so inappropriate to substitute in the
message. OTOH, showing only problematic publications is a short list
but it is still more useful than showing nothing (e.g. there other
publications of the subscription might be OK so those ones are not
listed)

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  287. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-09T04:44:05Z

    On Wed, Aug 9, 2023 at 8:20 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Aug 8, 2023 at 6:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Aug 8, 2023 at 1:50 PM Peter Eisentraut <peter@eisentraut.org> wrote:
> > >
> > > This patch added the following error message:
> > >
> > > errdetail_plural("Subscribed publication %s is subscribing to other
> > > publications.",
> > > "Subscribed publications %s are subscribing to other publications.",
> > > list_length(publist), pubnames->data),
> > >
> > > But in PostgreSQL, a publication cannot subscribe to a publication, so
> > > this is not giving accurate information.  Apparently, what it is trying
> > > to say is that
> > >
> > > The subscription that you are creating subscribes to publications that
> > > contain tables that are written to by other subscriptions.
> > >
> > > Can we get to a more accurate wording like this?
> > >
> >
> > +1 for changing the message as per your suggestion.
> >
>
> PSA a patch to change this message text. The message now has wording
> similar to the suggestion.
>
> > > There is also a translatability issue there, in the way the publication
> > > list is pasted into the message.
> > >
>
> The name/list substitution is now done within parentheses, which AFAIK
> will be enough to eliminate any translation ambiguities.
>

A similar instance as below uses \"%s\" instead. So, isn't using the
same a better idea?
errdetail_plural("LDAP search for filter \"%s\" on server \"%s\"
returned %d entry.",
  "LDAP search for filter \"%s\" on server \"%s\" returned %d entries.",


-- 
With Regards,
Amit Kapila.

  288. Re: Handle infinite recursion in logical replication setup

    Peter Smith <smithpb2250@gmail.com> — 2023-08-09T05:29:31Z

    On Wed, Aug 9, 2023 at 2:44 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 9, 2023 at 8:20 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Tue, Aug 8, 2023 at 6:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Tue, Aug 8, 2023 at 1:50 PM Peter Eisentraut <peter@eisentraut.org> wrote:
> > > >
> > > > This patch added the following error message:
> > > >
> > > > errdetail_plural("Subscribed publication %s is subscribing to other
> > > > publications.",
> > > > "Subscribed publications %s are subscribing to other publications.",
> > > > list_length(publist), pubnames->data),
> > > >
> > > > But in PostgreSQL, a publication cannot subscribe to a publication, so
> > > > this is not giving accurate information.  Apparently, what it is trying
> > > > to say is that
> > > >
> > > > The subscription that you are creating subscribes to publications that
> > > > contain tables that are written to by other subscriptions.
> > > >
> > > > Can we get to a more accurate wording like this?
> > > >
> > >
> > > +1 for changing the message as per your suggestion.
> > >
> >
> > PSA a patch to change this message text. The message now has wording
> > similar to the suggestion.
> >
> > > > There is also a translatability issue there, in the way the publication
> > > > list is pasted into the message.
> > > >
> >
> > The name/list substitution is now done within parentheses, which AFAIK
> > will be enough to eliminate any translation ambiguities.
> >
>
> A similar instance as below uses \"%s\" instead. So, isn't using the
> same a better idea?
> errdetail_plural("LDAP search for filter \"%s\" on server \"%s\"
> returned %d entry.",
>   "LDAP search for filter \"%s\" on server \"%s\" returned %d entries.",
>
>

Hmm. I don't see any similarity -- there the plural is for the %d, not
for a list of string items. And in our case the publication name (or
list of names) are already quoted by the function returning that list,
so quoting them again doesn't really make sense.

Example output using the patch look like this:

SINGLULAR
test_sub=# create subscription sub_test connection 'dbname=test_pub'
publication pub_all_at_pub with(origin=NONE);
WARNING:  subscription "sub_test" requested copy_data with origin =
NONE but might copy data that had a different origin
DETAIL:  The subscription that you are creating has a publication
("pub_all_at_pub") containing tables written to by other
subscriptions.
HINT:  Verify that initial data copied from the publisher tables did
not come from other origins.
NOTICE:  created replication slot "sub_test" on publisher
CREATE SUBSCRIPTION

PLURAL
test_sub=# create subscription sub_test3 connection 'dbname=test_pub'
publication pub_all_at_pub,pub_s1_at_pub,pub_s2_at_pub
with(origin=NONE);
WARNING:  subscription "sub_test3" requested copy_data with origin =
NONE but might copy data that had a different origin
DETAIL:  The subscription that you are creating has publications
("pub_s1_at_pub", "pub_all_at_pub") containing tables written to by
other subscriptions.
HINT:  Verify that initial data copied from the publisher tables did
not come from other origins.
NOTICE:  created replication slot "sub_test3" on publisher
CREATE SUBSCRIPTION

~

I thought above looked fine but if PeterE also does not like the ()
then the message can be rearranged slightly like below to put the
offending publications at the end of the message, like this:

DETAIL:  The subscription that you are creating has the following
publications containing tables written to by other subscriptions:
"pub_s1_at_pub", "pub_all_at_pub"

------
Kind Regards,
Peter Smith.
Fujitsu Australia

  289. Re: Handle infinite recursion in logical replication setup

    Amit Kapila <amit.kapila16@gmail.com> — 2023-08-09T06:28:23Z

    On Wed, Aug 9, 2023 at 10:59 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> Example output using the patch look like this:
>
> SINGLULAR
> test_sub=# create subscription sub_test connection 'dbname=test_pub'
> publication pub_all_at_pub with(origin=NONE);
> WARNING:  subscription "sub_test" requested copy_data with origin =
> NONE but might copy data that had a different origin
> DETAIL:  The subscription that you are creating has a publication
> ("pub_all_at_pub") containing tables written to by other
> subscriptions.
> HINT:  Verify that initial data copied from the publisher tables did
> not come from other origins.
> NOTICE:  created replication slot "sub_test" on publisher
> CREATE SUBSCRIPTION
>
> PLURAL
> test_sub=# create subscription sub_test3 connection 'dbname=test_pub'
> publication pub_all_at_pub,pub_s1_at_pub,pub_s2_at_pub
> with(origin=NONE);
> WARNING:  subscription "sub_test3" requested copy_data with origin =
> NONE but might copy data that had a different origin
> DETAIL:  The subscription that you are creating has publications
> ("pub_s1_at_pub", "pub_all_at_pub") containing tables written to by
> other subscriptions.
> HINT:  Verify that initial data copied from the publisher tables did
> not come from other origins.
> NOTICE:  created replication slot "sub_test3" on publisher
> CREATE SUBSCRIPTION
>
> ~
>
> I thought above looked fine but if PeterE also does not like the ()
> then the message can be rearranged slightly like below to put the
> offending publications at the end of the message, like this:
>

Okay, I didn't know strings were already quoted but not sure if Round
Brackets () exactly will address the translatability concern.

> DETAIL:  The subscription that you are creating has the following
> publications containing tables written to by other subscriptions:
> "pub_s1_at_pub", "pub_all_at_pub"
>

Fair enough. Peter E., do let us know what you think makes sense here?

-- 
With Regards,
Amit Kapila.

  290. Re: Handle infinite recursion in logical replication setup

    Peter Eisentraut <peter@eisentraut.org> — 2023-08-16T13:50:09Z

    On 09.08.23 04:50, Peter Smith wrote:
> On Tue, Aug 8, 2023 at 6:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Tue, Aug 8, 2023 at 1:50 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>>>
>>> This patch added the following error message:
>>>
>>> errdetail_plural("Subscribed publication %s is subscribing to other
>>> publications.",
>>> "Subscribed publications %s are subscribing to other publications.",
>>> list_length(publist), pubnames->data),
>>>
>>> But in PostgreSQL, a publication cannot subscribe to a publication, so
>>> this is not giving accurate information.  Apparently, what it is trying
>>> to say is that
>>>
>>> The subscription that you are creating subscribes to publications that
>>> contain tables that are written to by other subscriptions.
>>>
>>> Can we get to a more accurate wording like this?
>>>
>>
>> +1 for changing the message as per your suggestion.
>>
> 
> PSA a patch to change this message text. The message now has wording
> similar to the suggestion.

committed, thanks