Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Doc: Generated column replication.

  2. Rename pubgencols_type to pubgencols in pg_publication.

  3. Doc: Fix column name in pg_publication catalog.

  4. Fix buildfarm failure introduced by commit e65dbc9927.

  5. Change publication's publish_generated_columns option type to enum.

  6. Fix \dRp+ output when describing publications with a lower server version.

  7. Replicate generated columns when 'publish_generated_columns' is set.

  8. Doc: Update the behavior of generated columns in Logical Replication.

  9. Replicate generated columns when specified in the column list.

  10. Doc: Generated columns are skipped for logical replication.

  1. Pgoutput not capturing the generated columns

    Rajendra Kumar Dangwal <dangwalrajendra888@gmail.com> — 2023-08-01T06:47:33Z

    Hi PG Users.

We are using Debezium to capture the CDC events into Kafka. 
With decoderbufs and wal2json plugins the connector is able to capture the generated columns in the table but not with pgoutput plugin.

We tested with the following example:

CREATE TABLE employees (
   id SERIAL PRIMARY KEY,
   first_name VARCHAR(50),
   last_name VARCHAR(50),
   full_name VARCHAR(100) GENERATED ALWAYS AS (first_name || ' ' || last_name) STORED
);

// Inserted few records when the connector was running

Insert into employees (first_name, last_name) VALUES ('ABC' , 'XYZ’);


With decoderbufs and wal2json the connector is able to capture the generated column `full_name` in above example. But with pgoutput the generated column was not captured. 
Is this a known limitation of pgoutput plugin? If yes, where can we request to add support for this feature?

Thanks.
Rajendra.

  2. Re: Pgoutput not capturing the generated columns

    Euler Taveira <euler@eulerto.com> — 2023-08-01T14:51:45Z

    On Tue, Aug 1, 2023, at 3:47 AM, Rajendra Kumar Dangwal wrote:
> With decoderbufs and wal2json the connector is able to capture the generated column `full_name` in above example. But with pgoutput the generated column was not captured. 

wal2json materializes the generated columns before delivering the output. I
decided to materialized the generated columns in the output plugin because the
target consumers expects a complete row.

> Is this a known limitation of pgoutput plugin? If yes, where can we request to add support for this feature?

I wouldn't say limitation but a design decision.

The logical replication design decides to compute the generated columns at
subscriber side. It was a wise decision aiming optimization (it doesn't
overload the publisher that is *already* in charge of logical decoding).

Should pgoutput provide a complete row? Probably. If it is an option that
defaults to false and doesn't impact performance.

The request for features should be done in this mailing list.


--
Euler Taveira
EDB   https://www.enterprisedb.com/

  3. Re: Pgoutput not capturing the generated columns

    Rajendra Kumar Dangwal <dangwalrajendra888@gmail.com> — 2023-08-21T07:45:55Z

    Thanks Euler,
Greatly appreciate your inputs.

> Should pgoutput provide a complete row? Probably. If it is an option that defaults to false and doesn't impact performance.

Yes, it would be great if this feature can be implemented.

> The logical replication design decides to compute the generated columns at subscriber side.

If I understand correctly, this approach involves establishing a function on the subscriber's side that emulates the operation executed to derive the generated column values.
If yes, I see one potential issue where disparities might surface between the values of generated columns on the subscriber's side and those computed within Postgres. This could happen if the generated column's value relies on the current_time function.

Please let me know how can we track the feature requests and the discussions around that.

Thanks,
Rajendra.

  4. Re: Pgoutput not capturing the generated columns

    Rajendra Kumar Dangwal <dangwalrajendra888@gmail.com> — 2023-09-11T09:23:00Z

    Hi PG Hackers.

We are interested in enhancing the functionality of the pgoutput plugin by adding support for generated columns. 
Could you please guide us on the necessary steps to achieve this? Additionally, do you have a platform for tracking such feature requests? Any insights or assistance you can provide on this matter would be greatly appreciated.

Many thanks.
Rajendra.

  5. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-05-08T07:13:45Z

    On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal
<dangwalrajendra888@gmail.com> wrote:
>
> Hi PG Hackers.
>
> We are interested in enhancing the functionality of the pgoutput plugin by adding support for generated columns.
> Could you please guide us on the necessary steps to achieve this? Additionally, do you have a platform for tracking such feature requests? Any insights or assistance you can provide on this matter would be greatly appreciated.

The attached patch has the changes to support capturing generated
column data using ‘pgoutput’ and’ test_decoding’ plugin. Now if the
‘include_generated_columns’ option is specified, the generated column
information and generated column data also will be sent.

Usage from pgoutput plugin:
CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS
(a * 2) STORED);
CREATE publication pub1 for all tables;
SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'pgoutput');
SELECT * FROM pg_logical_slot_peek_binary_changes('slot1', NULL, NULL,
'proto_version', '1', 'publication_names', 'pub1',
'include_generated_columns', 'true');

Usage from test_decoding plugin:
SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding');
CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS
(a * 2) STORED);
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0', 'skip-empty-xacts', '1',
'include_generated_columns', '1');

Currently it is not supported as a subscription option because table
sync for the generated column is not possible as copy command does not
support getting data for the generated column. If this feature is
required we can remove this limitation from the copy command and then
add it as a subscription option later.
Thoughts?

Thanks and Regards,
Shubham Khanna.

  6. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-05-08T12:07:09Z

    Dear Shubham,

Thanks for creating a patch! Here are high-level comments.

1.
Please document the feature. If it is hard to describe, we should change the API.

2.
Currently, the option is implemented as streaming option. Are there any reasons
to choose the way? Another approach is to implement as slot option, like failover
and temporary.

3.
You said that subscription option is not supported for now. Not sure, is it mean
that logical replication feature cannot be used for generated columns? If so,
the restriction won't be acceptable. If the combination between this and initial
sync is problematic, can't we exclude them in CreateSubscrition and AlterSubscription?
E.g., create_slot option cannot be set if slot_name is NONE.

4.
Regarding the test_decoding plugin, it has already been able to decode the
generated columns. So... as the first place, is the proposed option really needed
for the plugin? Why do you include it?
If you anyway want to add the option, the default value should be on - which keeps
current behavior.

5.
Assuming that the feature become usable used for logical replicaiton. Not sure,
should we change the protocol version at that time? Nodes prior than PG17 may
not want receive values for generated columns. Can we control only by the option?

6. logicalrep_write_tuple()

```
-        if (!column_in_column_list(att->attnum, columns))
+        if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
+            continue;
```

Hmm, does above mean that generated columns are decoded even if they are not in
the column list? If so, why? I think such columns should not be sent.

7.

Some functions refer data->publish_generated_column many times. Can we store
the value to a variable?

Below comments are for test_decoding part, but they may be not needed.

=====

a. pg_decode_startup()

```
+        else if (strcmp(elem->defname, "include_generated_columns") == 0)
```

Other options for test_decoding do not have underscore. It should be
"include-generated-columns".

b. pg_decode_change()

data->include_generated_columns is referred four times in the function.
Can you store the value to a varibable?


c. pg_decode_change()

```
-                                    true);
+                                    true, data->include_generated_columns );
```

Please remove the blank.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/

  7. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-05-16T06:04:56Z

    Here are some review comments for the patch v1-0001.

======
GENERAL

G.1. Use consistent names

It seems to add unnecessary complications by having different names
for all the new options, fields and API parameters.

e.g. sometimes 'include_generated_columns'
e.g. sometimes 'publish_generated_columns'

Won't it be better to just use identical names everywhere for
everything? I don't mind which one you choose; I just felt you only
need one name, not two. This comment overrides everything else in this
post so whatever name you choose, make adjustments for all my other
review comments as necessary.

======

G.2. Is it possible to just use the existing bms?

A very large part of this patch is adding more API parameters to
delegate the 'publish_generated_columns' flag value down to when it is
finally checked and used. e.g.

The functions:
- logicalrep_write_insert(), logicalrep_write_update(),
logicalrep_write_delete()
... are delegating the new parameter 'publish_generated_column' down to:
- logicalrep_write_tuple

The functions:
- logicalrep_write_rel()
... are delegating the new parameter 'publish_generated_column' down to:
- logicalrep_write_attrs

AFAICT in all these places the API is already passing a "Bitmapset
*columns". I was wondering if it might be possible to modify the
"Bitmapset *columns" BEFORE any of those functions get called so that
the "columns" BMS either does or doesn't include generated cols (as
appropriate according to the option).

Well, it might not be so simple because there are some NULL BMS
considerations also, but I think it would be worth investigating at
least, because if indeed you can find some common place (somewhere
like pgoutput_change()?) where the columns BMS can be filtered to
remove bits for generated cols then it could mean none of those other
patch API changes are needed at all -- then the patch would only be
1/2 the size.

======
Commit message

1.
Now if include_generated_columns option is specified, the generated
column information and generated column data also will be sent.

Usage from pgoutput plugin:
SELECT * FROM pg_logical_slot_peek_binary_changes('slot1', NULL, NULL,
'proto_version', '1', 'publication_names', 'pub1',
'include_generated_columns', 'true');

Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0', 'skip-empty-xacts', '1',
'include_generated_columns', '1');

~

I think there needs to be more background information given here. This
commit message doesn't seem to describe anything about what is the
problem and how this patch fixes it. It just jumps straight into
giving usages of a 'include_generated_columns' option.

It also doesn't say that this is an option that was newly *introduced*
by the patch -- it refers to it as though the reader should already
know about it.

Furthermore, your hacker's post says "Currently it is not supported as
a subscription option because table sync for the generated column is
not possible as copy command does not support getting data for the
generated column. If this feature is required we can remove this
limitation from the copy command and then add it as a subscription
option later." IMO that all seems like the kind of information that
ought to also be mentioned in this commit message.

======
contrib/test_decoding/sql/ddl.sql

2.
+-- check include_generated_columns option with generated column
+CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
AS (a * 2) STORED);
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include_generated_columns', '1');
+DROP TABLE gencoltable;
+

2a.
Perhaps you should include both option values to demonstrate the
difference in behaviour:

'include_generated_columns', '0'
'include_generated_columns', '1'

~

2b.
I think you maybe need to include more some test combinations where
there is and isn't a COLUMN LIST, because I am not 100% sure I
understand the current logic/expectations for all combinations.

e.g. When the generated column is in a column list but
'publish_generated_columns' is false then what should happen? etc.
Also if there are any special rules then those should be mentioned in
the commit message.

======
src/backend/replication/logical/proto.c

3.
For all the API changes the new parameter name should be plural.

/publish_generated_column/publish_generated_columns/

~~~

4. logical_rep_write_tuple:

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

- if (!column_in_column_list(att->attnum, columns))
+ if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
+ continue;
+
+ if (att->attgenerated && !publish_generated_column)
  continue;
That code seems confusing. Shouldn't the logic be exactly as also in
logicalrep_write_attrs()?

e.g. Shouldn't they both look like this:

if (att->attisdropped)
  continue;

if (att->attgenerated && !publish_generated_column)
  continue;

if (!column_in_column_list(att->attnum, columns))
  continue;
======
src/backend/replication/pgoutput/pgoutput.c

5.
 static void send_relation_and_attrs(Relation relation, TransactionId xid,
  LogicalDecodingContext *ctx,
- Bitmapset *columns);
+ Bitmapset *columns,
+ bool publish_generated_column);

Use plural. /publish_generated_column/publish_generated_columns/

~~~

6. parse_output_parameters

  bool origin_option_given = false;
+ bool generate_column_option_given = false;

  data->binary = false;
  data->streaming = LOGICALREP_STREAM_OFF;
  data->messages = false;
  data->two_phase = false;
+ data->publish_generated_column = false;

I think the 1st var should be 'include_generated_columns_option_given'
for consistency with the name of the actual option that was given.

======
src/include/replication/logicalproto.h

7.
(Same as a previous review comment)

For all the API changes the new parameter name should be plural.

/publish_generated_column/publish_generated_columns/

======
src/include/replication/pgoutput.h

8.
  bool publish_no_origin;
+ bool publish_generated_column;
 } PGOutputData;

/publish_generated_column/publish_generated_columns/

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  8. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-05-20T06:46:17Z

    Hi Kuroda-san,

Thanks for reviewing the patch. I have fixed some of the comments
> 2.
> Currently, the option is implemented as streaming option. Are there any reasons
> to choose the way? Another approach is to implement as slot option, like failover
> and temporary.
I think the current approach is appropriate. The options such as
failover and temporary seem like properties of a slot and I think
decoding of generated column should not be slot specific. Also adding
a new option for slot may create an overhead.

> 3.
> You said that subscription option is not supported for now. Not sure, is it mean
> that logical replication feature cannot be used for generated columns? If so,
> the restriction won't be acceptable. If the combination between this and initial
> sync is problematic, can't we exclude them in CreateSubscrition and AlterSubscription?
> E.g., create_slot option cannot be set if slot_name is NONE.
Added an option 'generated_column' for create subscription. Currently
it allow to set 'generated_column' option as true only if 'copy_data'
is set to false.
Also we don't allow user to alter the 'generated_column' option.

> 6. logicalrep_write_tuple()
>
> ```
> -        if (!column_in_column_list(att->attnum, columns))
> +        if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
> +            continue;
> ```
>
> Hmm, does above mean that generated columns are decoded even if they are not in
> the column list? If so, why? I think such columns should not be sent.
Fixed

Thanks and Regards,
Shlok Kyal

  9. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-05-20T08:18:41Z

    Hi,

On Wed, May 8, 2024 at 4:14 PM Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal
> <dangwalrajendra888@gmail.com> wrote:
> >
> > Hi PG Hackers.
> >
> > We are interested in enhancing the functionality of the pgoutput plugin by adding support for generated columns.
> > Could you please guide us on the necessary steps to achieve this? Additionally, do you have a platform for tracking such feature requests? Any insights or assistance you can provide on this matter would be greatly appreciated.
>
> The attached patch has the changes to support capturing generated
> column data using ‘pgoutput’ and’ test_decoding’ plugin. Now if the
> ‘include_generated_columns’ option is specified, the generated column
> information and generated column data also will be sent.

As Euler mentioned earlier, I think it's a decision not to replicate
generated columns because we don't know the target table on the
subscriber has the same expression and there could be locale issues
even if it looks the same. I can see that a benefit of this proposal
would be to save cost to compute generated column values if the user
wants the target table on the subscriber to have exactly the same data
as the publisher's one. Are there other benefits or use cases?

>
> Usage from pgoutput plugin:
> CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS
> (a * 2) STORED);
> CREATE publication pub1 for all tables;
> SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'pgoutput');
> SELECT * FROM pg_logical_slot_peek_binary_changes('slot1', NULL, NULL,
> 'proto_version', '1', 'publication_names', 'pub1',
> 'include_generated_columns', 'true');
>
> Usage from test_decoding plugin:
> SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding');
> CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS
> (a * 2) STORED);
> INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
> 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include_generated_columns', '1');
>
> Currently it is not supported as a subscription option because table
> sync for the generated column is not possible as copy command does not
> support getting data for the generated column. If this feature is
> required we can remove this limitation from the copy command and then
> add it as a subscription option later.
> Thoughts?

I think that if we want to support an option to replicate generated
columns, the initial tablesync should support it too. Otherwise, we
end up filling the target columns data with NULL during the initial
tablesync but with replicated data during the streaming changes.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  10. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-05-20T08:58:30Z

    On Mon, 20 May 2024 at 13:49, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Hi,
>
> On Wed, May 8, 2024 at 4:14 PM Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal
> > <dangwalrajendra888@gmail.com> wrote:
> > >
> > > Hi PG Hackers.
> > >
> > > We are interested in enhancing the functionality of the pgoutput plugin by adding support for generated columns.
> > > Could you please guide us on the necessary steps to achieve this? Additionally, do you have a platform for tracking such feature requests? Any insights or assistance you can provide on this matter would be greatly appreciated.
> >
> > The attached patch has the changes to support capturing generated
> > column data using ‘pgoutput’ and’ test_decoding’ plugin. Now if the
> > ‘include_generated_columns’ option is specified, the generated column
> > information and generated column data also will be sent.
>
> As Euler mentioned earlier, I think it's a decision not to replicate
> generated columns because we don't know the target table on the
> subscriber has the same expression and there could be locale issues
> even if it looks the same. I can see that a benefit of this proposal
> would be to save cost to compute generated column values if the user
> wants the target table on the subscriber to have exactly the same data
> as the publisher's one. Are there other benefits or use cases?

I think this will be useful mainly for the use cases where the
publisher has generated columns and the subscriber does not have
generated  columns.
In the case where both the publisher and subscriber have generated
columns, the current patch will overwrite the generated column values
based on the expression for the generated column in the subscriber.

> >
> > Usage from pgoutput plugin:
> > CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS
> > (a * 2) STORED);
> > CREATE publication pub1 for all tables;
> > SELECT 'init' FROM pg_create_logical_replication_slot('slot1', 'pgoutput');
> > SELECT * FROM pg_logical_slot_peek_binary_changes('slot1', NULL, NULL,
> > 'proto_version', '1', 'publication_names', 'pub1',
> > 'include_generated_columns', 'true');
> >
> > Usage from test_decoding plugin:
> > SELECT 'init' FROM pg_create_logical_replication_slot('slot2', 'test_decoding');
> > CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS
> > (a * 2) STORED);
> > INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> > SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
> > 'include-xids', '0', 'skip-empty-xacts', '1',
> > 'include_generated_columns', '1');
> >
> > Currently it is not supported as a subscription option because table
> > sync for the generated column is not possible as copy command does not
> > support getting data for the generated column. If this feature is
> > required we can remove this limitation from the copy command and then
> > add it as a subscription option later.
> > Thoughts?
>
> I think that if we want to support an option to replicate generated
> columns, the initial tablesync should support it too. Otherwise, we
> end up filling the target columns data with NULL during the initial
> tablesync but with replicated data during the streaming changes.

+1 for supporting initial sync.
Currently copy_data = true and generate_column = true are not
supported, this limitation will be removed in one of the upcoming
patches.

Regards,
Vignesh

  11. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-05-21T06:52:33Z

    Hi,

AFAICT this v2-0001 patch differences from v1 is mostly about adding
the new CREATE SUBSCRIPTION option. Specifically, I don't think it is
addressing any of my previous review comments for patch v1. [1]. So
these comments below are limited only to the new option code; All my
previous review comments probably still apply.

======
Commit message

1. (General)
The commit message is seriously lacking background explanation to describe:
- What is the current behaviour w.r.t. generated columns
- What is the problem with the current behaviour?
- What exactly is this patch doing to address that problem?

~

2.
New option generated_option is added in create subscription. Now if this
option is specified as 'true' during create subscription, generated
columns in the tables, present in publisher (to which this subscription is
subscribed) can also be replicated.

-

2A.
"generated_option" is not the name of the new option.

~

2B.
"create subscription" stmt should be UPPERCASE; will also be more
readable if the option name is quoted.

~

2C.
Needs more information like under what condition is this option ignored etc.

======
doc/src/sgml/ref/create_subscription.sgml

3.
+       <varlistentry id="sql-createsubscription-params-with-generated-column">
+        <term><literal>generated-column</literal> (<type>boolean</type>)</term>
+        <listitem>
+         <para>
+          Specifies whether the generated columns present in the tables
+          associated with the subscription should be replicated. The default is
+          <literal>false</literal>.
+         </para>
+
+         <para>
+          This parameter can only be set true if copy_data is set to false.
+          This option works fine when a generated column (in
publisher) is replicated to a
+          non-generated column (in subscriber). Else if it is
replicated to a generated
+          column, it will ignore the replicated data and fill the
column with computed or
+          default data.
+         </para>
+        </listitem>
+       </varlistentry>

3A.
There is a typo in the name "generated-column" because we should use
underscores (not hyphens) for the option names.

~

3B.
This it is not a good option name because there is no verb so it
doesn't mean anything to set it true/false -- actually there IS a verb
"generate" but we are not saying generate = true/false, so this name
is also quite confusing.

I think "include_generated_columns" would be much better, but if
others think that name is too long then maybe "include_generated_cols"
or "include_gen_cols" or similar. Of course, whatever if the final
decision should be propagated same thru all the code comments, params,
fields, etc.

~

3C.
copy_data and false should be marked up as <literal> fonts in the sgml

~

3D.

Suggest re-word this part. Don't need to explain when it "works fine".

BEFORE
This option works fine when a generated column (in publisher) is
replicated to a non-generated column (in subscriber). Else if it is
replicated to a generated column, it will ignore the replicated data
and fill the column with computed or default data.

SUGGESTION
If the subscriber-side column is also a generated column then this
option has no effect; the replicated data will be ignored and the
subscriber column will be filled as normal with the subscriber-side
computed or default data.

======
src/backend/commands/subscriptioncmds.c

4. AlterSubscription
    SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
    SUBOPT_PASSWORD_REQUIRED |
    SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
-   SUBOPT_ORIGIN);
+   SUBOPT_ORIGIN | SUBOPT_GENERATED_COLUMN);

Hmm. Is this correct? If ALTER is not allowed (later in this patch
there is a message "toggling generated_column option is not allowed."
then why are we even saying that SUBOPT_GENERATED_COLUMN is a
support_opt for ALTER?

~~~

5.
+ if (IsSet(opts.specified_opts, SUBOPT_GENERATED_COLUMN))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("toggling generated_column option is not allowed.")));
+ }

5A.
I suspect this is not even needed if the 'supported_opt' is fixed per
the previous comment.

~

5B.
But if this message is still needed then I think it should say "ALTER
is not allowed" (not "toggling is not allowed") and also the option
name should be quoted as per the new guidelines for error messages.

======
src/backend/replication/logical/proto.c


6. logicalrep_write_tuple

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

  if (!column_in_column_list(att->attnum, columns))
  continue;

+ if (att->attgenerated && !publish_generated_column)
+

Calling column_in_column_list() might be a more expensive operation
than checking just generated columns flag so maybe reverse the order
and check the generated columns first for a tiny performance gain.

~~

7.
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

  if (!column_in_column_list(att->attnum, columns))
  continue;

+ if (att->attgenerated && !publish_generated_column)
+ continue;

ditto #6

~~

8. logicalrep_write_attrs

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

  if (!column_in_column_list(att->attnum, columns))
  continue;

+ if (att->attgenerated && !publish_generated_column)
+ continue;
+

ditto #6

~~

9.
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

  if (!column_in_column_list(att->attnum, columns))
  continue;

+ if (att->attgenerated && !publish_generated_column)
+ continue;

ditto #6

======
src/include/catalog/pg_subscription.h


10. CATALOG

+ bool subgeneratedcolumn; /* True if generated colums must be published */

/colums/columns/

======
src/test/regress/sql/publication.sql

11.
--- error: generated column "d" can't be in list
+-- ok


Maybe change "ok" to say like "ok: generated cols can be in the list too"

======

12.
GENERAL - Missing CREATE SUBSCRIPTION test?
GENERAL - Missing ALTER SUBSCRIPTION test?

How come this patch adds a new CREATE SUBSCRIPTION option but does not
seem to include any test case for that option in either the CREATE
SUBSCRIPTION or ALTER SUBSCRIPTION regression tests?

======
[1] My v1 review -
https://www.postgresql.org/message-id/CAHut+PsuJfcaeg6zst=6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng@mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  12. Re: Pgoutput not capturing the generated columns

    Peter Eisentraut <peter@eisentraut.org> — 2024-05-22T11:40:27Z

    On 08.05.24 09:13, Shubham Khanna wrote:
> The attached patch has the changes to support capturing generated
> column data using ‘pgoutput’ and’ test_decoding’ plugin. Now if the
> ‘include_generated_columns’ option is specified, the generated column
> information and generated column data also will be sent.

It might be worth keeping half an eye on the development of virtual 
generated columns [0].  I think it won't be possible to include those 
into the replication output stream.

I think having an option for including stored generated columns is in 
general ok.


[0]: 
https://www.postgresql.org/message-id/flat/a368248e-69e4-40be-9c07-6c3b5880b0a6@eisentraut.org

  13. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-05-23T03:49:03Z

    > Dear Shubham,
>
> Thanks for creating a patch! Here are high-level comments.

> 1.
> Please document the feature. If it is hard to describe, we should change the API.

I have added the feature in the document.

> 4.
> Regarding the test_decoding plugin, it has already been able to decode the
> generated columns. So... as the first place, is the proposed option really needed
> for the plugin? Why do you include it?
> If you anyway want to add the option, the default value should be on - which keeps
> current behavior.

I have made the generated column options as true for test_decoding
plugin so by default we will send generated column data.

> 5.
> Assuming that the feature become usable used for logical replicaiton. Not sure,
> should we change the protocol version at that time? Nodes prior than PG17 may
> not want receive values for generated columns. Can we control only by the option?

I verified the backward compatibility test by using the generated
column option and it worked fine. I think there is no need to make any
further changes.

> 7.
>
> Some functions refer data->publish_generated_column many times. Can we store
> the value to a variable?
>
> Below comments are for test_decoding part, but they may be not needed.
>
> =====
>
> a. pg_decode_startup()
>
> ```
> +        else if (strcmp(elem->defname, "include_generated_columns") == 0)
> ```
>
> Other options for test_decoding do not have underscore. It should be
> "include-generated-columns".
>
> b. pg_decode_change()
>
> data->include_generated_columns is referred four times in the function.
> Can you store the value to a varibable?
>
>
> c. pg_decode_change()
>
> ```
> -                                    true);
> +                                    true, data->include_generated_columns );
> ```
>
> Please remove the blank.

Fixed.
The attached v3 Patch has the changes for the same.

Thanks and Regards,
Shubham Khanna.

  14. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-05-23T05:26:38Z

    Dear Shubham,

Thanks for updating the patch! I checked your patches briefly. Here are my comments.

01. API

Since the option for test_decoding is enabled by default, I think it should be renamed.
E.g., "skip-generated-columns" or something.

02. ddl.sql

```
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
+                            data                             
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
```

We should test non-default case, which the generated columns are not generated.

03. ddl.sql

Not sure new tests are in the correct place. Do we have to add new file and move tests to it?
Thought?

04. protocol.sgml

Please keep the format of the sgml file.

05. protocol.sgml

The option is implemented as the streaming option of pgoutput plugin, so they should be
located under "Logical Streaming Replication Parameters" section.

05. AlterSubscription

```
+				if (IsSet(opts.specified_opts, SUBOPT_GENERATED_COLUMN))
+				{
+					ereport(ERROR,
+							(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+							 errmsg("toggling generated_column option is not allowed.")));
+				}
```

If you don't want to support the option, you can remove SUBOPT_GENERATED_COLUMN
macro from the function. But can you clarify the reason why you do not want?

07. logicalrep_write_tuple

```
-		if (!column_in_column_list(att->attnum, columns))
+		if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
+			continue;
+
+		if (att->attgenerated && !publish_generated_column)
 			continue;
```

I think changes in v2 was reverted or wrongly merged.

08. test code

Can you add tests that generated columns are replicated by the logical replication?

Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/

  15. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-05-23T12:26:11Z

    On Thu, 23 May 2024 at 09:19, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> > Dear Shubham,
> >
> > Thanks for creating a patch! Here are high-level comments.
>
> > 1.
> > Please document the feature. If it is hard to describe, we should change the API.
>
> I have added the feature in the document.
>
> > 4.
> > Regarding the test_decoding plugin, it has already been able to decode the
> > generated columns. So... as the first place, is the proposed option really needed
> > for the plugin? Why do you include it?
> > If you anyway want to add the option, the default value should be on - which keeps
> > current behavior.
>
> I have made the generated column options as true for test_decoding
> plugin so by default we will send generated column data.
>
> > 5.
> > Assuming that the feature become usable used for logical replicaiton. Not sure,
> > should we change the protocol version at that time? Nodes prior than PG17 may
> > not want receive values for generated columns. Can we control only by the option?
>
> I verified the backward compatibility test by using the generated
> column option and it worked fine. I think there is no need to make any
> further changes.
>
> > 7.
> >
> > Some functions refer data->publish_generated_column many times. Can we store
> > the value to a variable?
> >
> > Below comments are for test_decoding part, but they may be not needed.
> >
> > =====
> >
> > a. pg_decode_startup()
> >
> > ```
> > +        else if (strcmp(elem->defname, "include_generated_columns") == 0)
> > ```
> >
> > Other options for test_decoding do not have underscore. It should be
> > "include-generated-columns".
> >
> > b. pg_decode_change()
> >
> > data->include_generated_columns is referred four times in the function.
> > Can you store the value to a varibable?
> >
> >
> > c. pg_decode_change()
> >
> > ```
> > -                                    true);
> > +                                    true, data->include_generated_columns );
> > ```
> >
> > Please remove the blank.
>
> Fixed.
> The attached v3 Patch has the changes for the same.

Few comments:
1) Since this is removed, tupdesc variable is not required anymore:
+++ b/src/backend/catalog/pg_publication.c
@@ -534,12 +534,6 @@ publication_translate_columns(Relation targetrel,
List *columns,
                                        errmsg("cannot use system
column \"%s\" in publication column list",
                                                   colname));

-               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
-                       ereport(ERROR,
-
errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
-                                       errmsg("cannot use generated
column \"%s\" in publication column list",
-                                                  colname));

2) In test_decoding include_generated_columns option is used:
+               else if (strcmp(elem->defname,
"include_generated_columns") == 0)
+               {
+                       if (elem->arg == NULL)
+                               continue;
+                       else if (!parse_bool(strVal(elem->arg),
&data->include_generated_columns))
+                               ereport(ERROR,
+
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+                                                errmsg("could not
parse value \"%s\" for parameter \"%s\"",
+
strVal(elem->arg), elem->defname)));
+               }

In subscription we have used generated_column, we can try to use the
same option in both places:
+               else if (IsSet(supported_opts, SUBOPT_GENERATED_COLUMN) &&
+                                strcmp(defel->defname,
"generated_column") == 0)
+               {
+                       if (IsSet(opts->specified_opts,
SUBOPT_GENERATED_COLUMN))
+                               errorConflictingDefElem(defel, pstate);
+
+                       opts->specified_opts |= SUBOPT_GENERATED_COLUMN;
+                       opts->generated_column = defGetBoolean(defel);
+               }

3) Tab completion can be added for create subscription to include
generated_column option

4) There are few whitespace issues while applying the patch, check for
git diff --check

5) Add few tests for the new option added

Regards,
Vignesh

  16. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-05-24T02:55:34Z

    Hi,

Here are some review comments for the patch v3-0001.

I don't think v3 addressed any of my previous review comments for
patches v1 and v2. [1][2]

So the comments below are limited only to the new code (i.e. the v3
versus v2 differences). Meanwhile, all my previous review comments may
still apply.

======
GENERAL

The patch applied gives whitespace warnings:

[postgres@CentOS7-x64 oss_postgres_misc]$ git apply
../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch
../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch:150:
trailing whitespace.

../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch:202:
trailing whitespace.

../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch:730:
trailing whitespace.
warning: 3 lines add whitespace errors.

======
contrib/test_decoding/test_decoding.c

1. pg_decode_change

  MemoryContext old;
+ bool include_generated_columns;
+

I'm not really convinced this variable saves any code.

======
doc/src/sgml/protocol.sgml

2.
+        <varlistentry>
+         <term><replaceable
class="parameter">include-generated-columns</replaceable></term>
+         <listitem>
+        <para>
+        The include-generated-columns option controls whether
generated columns should be included in the string representation of
tuples during logical decoding in PostgreSQL. This allows users to
customize the output format based on whether they want to include
these columns or not.
+         </para>
+         </listitem>
+         </varlistentry>

2a.
Something is not correct when this name has hyphens and all the nearby
parameter names do not. Shouldn't it be all uppercase like the other
boolean parameter?

~

2b.
Text in the SGML file should be wrapped properly.

~

2c.
IMO the comment can be more terse and it also needs to specify that it
is a boolean type, and what is the default value if not passed.

SUGGESTION

INCLUDE_GENERATED_COLUMNS [ boolean ]

If true, then generated columns should be included in the string
representation of tuples during logical decoding in PostgreSQL. The
default is false.

======
src/backend/replication/logical/proto.c

3. logicalrep_write_tuple

- if (!column_in_column_list(att->attnum, columns))
+ if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
+ continue;
+
+ if (att->attgenerated && !publish_generated_column)
  continue;

3a.
This code seems overcomplicated checking the same flag multiple times.

SUGGESTION
if (att->attgenerated)
{
  if (!publish_generated_column)
    continue;
}
else
{
  if (!column_in_column_list(att->attnum, columns))
    continue;
}

~

3b.
The same logic occurs several times in logicalrep_write_tuple

~~~

4. logicalrep_write_attrs

  if (!column_in_column_list(att->attnum, columns))
  continue;

+ if (att->attgenerated && !publish_generated_column)
+ continue;
+

Shouldn't these code fragments (2x in this function) look the same as
in logicalrep_write_tuple? See the above review comments.

======
src/backend/replication/pgoutput/pgoutput.c

5. maybe_send_schema

  TransactionId topxid = InvalidTransactionId;
+ bool publish_generated_column = data->publish_generated_column;

I'm not convinced this saves any code, and anyway, it is not
consistent with other fields in this function that are not extracted
to another variable (e.g. data->streaming).

~~~

6. pgoutput_change
-
+ bool publish_generated_column = data->publish_generated_column;
+

I'm not convinced this saves any code, and anyway, it is not
consistent with other fields in this function that are not extracted
to another variable (e.g. data->binary).

======
[1] My v1 review -
https://www.postgresql.org/message-id/CAHut+PsuJfcaeg6zst=6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng@mail.gmail.com
[2] My v2 review -
https://www.postgresql.org/message-id/CAHut%2BPv4RpOsUgkEaXDX%3DW2rhHAsJLiMWdUrUGZOcoRHuWj5%2BQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  17. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-03T07:33:00Z

    On Thu, May 16, 2024 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for the patch v1-0001.
>
> ======
> GENERAL
>
> G.1. Use consistent names
>
> It seems to add unnecessary complications by having different names
> for all the new options, fields and API parameters.
>
> e.g. sometimes 'include_generated_columns'
> e.g. sometimes 'publish_generated_columns'
>
> Won't it be better to just use identical names everywhere for
> everything? I don't mind which one you choose; I just felt you only
> need one name, not two. This comment overrides everything else in this
> post so whatever name you choose, make adjustments for all my other
> review comments as necessary.

I have updated the name to 'include_generated_columns' everywhere in the Patch.

> ======
>
> G.2. Is it possible to just use the existing bms?
>
> A very large part of this patch is adding more API parameters to
> delegate the 'publish_generated_columns' flag value down to when it is
> finally checked and used. e.g.
>
> The functions:
> - logicalrep_write_insert(), logicalrep_write_update(),
> logicalrep_write_delete()
> ... are delegating the new parameter 'publish_generated_column' down to:
> - logicalrep_write_tuple
>
> The functions:
> - logicalrep_write_rel()
> ... are delegating the new parameter 'publish_generated_column' down to:
> - logicalrep_write_attrs
>
> AFAICT in all these places the API is already passing a "Bitmapset
> *columns". I was wondering if it might be possible to modify the
> "Bitmapset *columns" BEFORE any of those functions get called so that
> the "columns" BMS either does or doesn't include generated cols (as
> appropriate according to the option).
>
> Well, it might not be so simple because there are some NULL BMS
> considerations also, but I think it would be worth investigating at
> least, because if indeed you can find some common place (somewhere
> like pgoutput_change()?) where the columns BMS can be filtered to
> remove bits for generated cols then it could mean none of those other
> patch API changes are needed at all -- then the patch would only be
> 1/2 the size.

I will analyse and reply to this in the next version.

> ======
> Commit message
>
> 1.
> Now if include_generated_columns option is specified, the generated
> column information and generated column data also will be sent.
>
> Usage from pgoutput plugin:
> SELECT * FROM pg_logical_slot_peek_binary_changes('slot1', NULL, NULL,
> 'proto_version', '1', 'publication_names', 'pub1',
> 'include_generated_columns', 'true');
>
> Usage from test_decoding plugin:
> SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
> 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include_generated_columns', '1');
>
> ~
>
> I think there needs to be more background information given here. This
> commit message doesn't seem to describe anything about what is the
> problem and how this patch fixes it. It just jumps straight into
> giving usages of a 'include_generated_columns' option.
>
> It also doesn't say that this is an option that was newly *introduced*
> by the patch -- it refers to it as though the reader should already
> know about it.
>
> Furthermore, your hacker's post says "Currently it is not supported as
> a subscription option because table sync for the generated column is
> not possible as copy command does not support getting data for the
> generated column. If this feature is required we can remove this
> limitation from the copy command and then add it as a subscription
> option later." IMO that all seems like the kind of information that
> ought to also be mentioned in this commit message.

I have updated the Commit message mentioning the suggested changes.

> ======
> contrib/test_decoding/sql/ddl.sql
>
> 2.
> +-- check include_generated_columns option with generated column
> +CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
> AS (a * 2) STORED);
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include_generated_columns', '1');
> +DROP TABLE gencoltable;
> +
>
> 2a.
> Perhaps you should include both option values to demonstrate the
> difference in behaviour:
>
> 'include_generated_columns', '0'
> 'include_generated_columns', '1'

Added the other option values to demonstrate the difference in behaviour:

> 2b.
> I think you maybe need to include more some test combinations where
> there is and isn't a COLUMN LIST, because I am not 100% sure I
> understand the current logic/expectations for all combinations.
>
> e.g. When the generated column is in a column list but
> 'publish_generated_columns' is false then what should happen? etc.
> Also if there are any special rules then those should be mentioned in
> the commit message.

Test case is added and the same is mentioned in the documentation.

> ======
> src/backend/replication/logical/proto.c
>
> 3.
> For all the API changes the new parameter name should be plural.
>
> /publish_generated_column/publish_generated_columns/

Updated the name to 'include_generated_columns'

> 4. logical_rep_write_tuple:
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
> - if (!column_in_column_list(att->attnum, columns))
> + if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
> + continue;
> +
> + if (att->attgenerated && !publish_generated_column)
>   continue;
> That code seems confusing. Shouldn't the logic be exactly as also in
> logicalrep_write_attrs()?
>
> e.g. Shouldn't they both look like this:
>
> if (att->attisdropped)
>   continue;
>
> if (att->attgenerated && !publish_generated_column)
>   continue;
>
> if (!column_in_column_list(att->attnum, columns))
>   continue;

Fixed.

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 5.
>  static void send_relation_and_attrs(Relation relation, TransactionId xid,
>   LogicalDecodingContext *ctx,
> - Bitmapset *columns);
> + Bitmapset *columns,
> + bool publish_generated_column);
>
> Use plural. /publish_generated_column/publish_generated_columns/

Updated the name to 'include_generated_columns'

> 6. parse_output_parameters
>
>   bool origin_option_given = false;
> + bool generate_column_option_given = false;
>
>   data->binary = false;
>   data->streaming = LOGICALREP_STREAM_OFF;
>   data->messages = false;
>   data->two_phase = false;
> + data->publish_generated_column = false;
>
> I think the 1st var should be 'include_generated_columns_option_given'
> for consistency with the name of the actual option that was given.

Updated the name to 'include_generated_columns_option_given'

> ======
> src/include/replication/logicalproto.h
>
> 7.
> (Same as a previous review comment)
>
> For all the API changes the new parameter name should be plural.
>
> /publish_generated_column/publish_generated_columns/

Updated the name to 'include_generated_columns'

> ======
> src/include/replication/pgoutput.h
>
> 8.
>   bool publish_no_origin;
> + bool publish_generated_column;
>  } PGOutputData;
>
> /publish_generated_column/publish_generated_columns/

Updated the name to 'include_generated_columns'

The attached Patch contains the suggested changes.

Thanks and Regards,
Shubham Khanna.

  18. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-03T09:30:38Z

    On Tue, May 21, 2024 at 12:23 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> AFAICT this v2-0001 patch differences from v1 is mostly about adding
> the new CREATE SUBSCRIPTION option. Specifically, I don't think it is
> addressing any of my previous review comments for patch v1. [1]. So
> these comments below are limited only to the new option code; All my
> previous review comments probably still apply.
>
> ======
> Commit message
>
> 1. (General)
> The commit message is seriously lacking background explanation to describe:
> - What is the current behaviour w.r.t. generated columns
> - What is the problem with the current behaviour?
> - What exactly is this patch doing to address that problem?

Added the information related to this inside the Patch.

> 2.
> New option generated_option is added in create subscription. Now if this
> option is specified as 'true' during create subscription, generated
> columns in the tables, present in publisher (to which this subscription is
> subscribed) can also be replicated.
>
> -
>
> 2A.
> "generated_option" is not the name of the new option.
>
> ~
>
> 2B.
> "create subscription" stmt should be UPPERCASE; will also be more
> readable if the option name is quoted.
>
> ~
>
> 2C.
> Needs more information like under what condition is this option ignored etc.

Fixed.

> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 3.
> +       <varlistentry id="sql-createsubscription-params-with-generated-column">
> +        <term><literal>generated-column</literal> (<type>boolean</type>)</term>
> +        <listitem>
> +         <para>
> +          Specifies whether the generated columns present in the tables
> +          associated with the subscription should be replicated. The default is
> +          <literal>false</literal>.
> +         </para>
> +
> +         <para>
> +          This parameter can only be set true if copy_data is set to false.
> +          This option works fine when a generated column (in
> publisher) is replicated to a
> +          non-generated column (in subscriber). Else if it is
> replicated to a generated
> +          column, it will ignore the replicated data and fill the
> column with computed or
> +          default data.
> +         </para>
> +        </listitem>
> +       </varlistentry>
>
> 3A.
> There is a typo in the name "generated-column" because we should use
> underscores (not hyphens) for the option names.
>
> ~
>
> 3B.
> This it is not a good option name because there is no verb so it
> doesn't mean anything to set it true/false -- actually there IS a verb
> "generate" but we are not saying generate = true/false, so this name
> is also quite confusing.
>
> I think "include_generated_columns" would be much better, but if
> others think that name is too long then maybe "include_generated_cols"
> or "include_gen_cols" or similar. Of course, whatever if the final
> decision should be propagated same thru all the code comments, params,
> fields, etc.
>
> ~
>
> 3C.
> copy_data and false should be marked up as <literal> fonts in the sgml
>
> ~
>
> 3D.
>
> Suggest re-word this part. Don't need to explain when it "works fine".
>
> BEFORE
> This option works fine when a generated column (in publisher) is
> replicated to a non-generated column (in subscriber). Else if it is
> replicated to a generated column, it will ignore the replicated data
> and fill the column with computed or default data.
>
> SUGGESTION
> If the subscriber-side column is also a generated column then this
> option has no effect; the replicated data will be ignored and the
> subscriber column will be filled as normal with the subscriber-side
> computed or default data.

Fixed.

> ======
> src/backend/commands/subscriptioncmds.c
>
> 4. AlterSubscription
>     SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
>     SUBOPT_PASSWORD_REQUIRED |
>     SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
> -   SUBOPT_ORIGIN);
> +   SUBOPT_ORIGIN | SUBOPT_GENERATED_COLUMN);
>
> Hmm. Is this correct? If ALTER is not allowed (later in this patch
> there is a message "toggling generated_column option is not allowed."
> then why are we even saying that SUBOPT_GENERATED_COLUMN is a
> support_opt for ALTER?

Fixed.

> 5.
> + if (IsSet(opts.specified_opts, SUBOPT_GENERATED_COLUMN))
> + {
> + ereport(ERROR,
> + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("toggling generated_column option is not allowed.")));
> + }
>
> 5A.
> I suspect this is not even needed if the 'supported_opt' is fixed per
> the previous comment.
>
> ~
>
> 5B.
> But if this message is still needed then I think it should say "ALTER
> is not allowed" (not "toggling is not allowed") and also the option
> name should be quoted as per the new guidelines for error messages.
>
> ======
> src/backend/replication/logical/proto.c

Fixed.

> 6. logicalrep_write_tuple
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
>   if (!column_in_column_list(att->attnum, columns))
>   continue;
>
> + if (att->attgenerated && !publish_generated_column)
> +
>
> Calling column_in_column_list() might be a more expensive operation
> than checking just generated columns flag so maybe reverse the order
> and check the generated columns first for a tiny performance gain.

Fixed.

> 7.
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
>   if (!column_in_column_list(att->attnum, columns))
>   continue;
>
> + if (att->attgenerated && !publish_generated_column)
> + continue;
>
> ditto #6

Fixed.

> 8. logicalrep_write_attrs
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
>   if (!column_in_column_list(att->attnum, columns))
>   continue;
>
> + if (att->attgenerated && !publish_generated_column)
> + continue;
> +
>
> ditto #6

Fixed.

> 9.
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
>   if (!column_in_column_list(att->attnum, columns))
>   continue;
>
> + if (att->attgenerated && !publish_generated_column)
> + continue;
>
> ditto #6
>
> ======
> src/include/catalog/pg_subscription.h

Fixed.

> 10. CATALOG
>
> + bool subgeneratedcolumn; /* True if generated colums must be published */
>
> /colums/columns/
>
> ======
> src/test/regress/sql/publication.sql

Fixed.

> 11.
> --- error: generated column "d" can't be in list
> +-- ok
>
>
> Maybe change "ok" to say like "ok: generated cols can be in the list too"

Fixed.

> 12.
> GENERAL - Missing CREATE SUBSCRIPTION test?
> GENERAL - Missing ALTER SUBSCRIPTION test?
>
> How come this patch adds a new CREATE SUBSCRIPTION option but does not
> seem to include any test case for that option in either the CREATE
> SUBSCRIPTION or ALTER SUBSCRIPTION regression tests?

Added the test cases for the same.

Patch v4-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJcOsk%3Dy%2BvJ3y%2BvXhzR9ZUzUEURvS_90hQW3MNfJ5di7A%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  19. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-03T09:55:38Z

    On Thu, May 23, 2024 at 10:56 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Shubham,
>
> Thanks for updating the patch! I checked your patches briefly. Here are my comments.
>
> 01. API
>
> Since the option for test_decoding is enabled by default, I think it should be renamed.
> E.g., "skip-generated-columns" or something.

Let's keep the same name 'include_generated_columns' for both the cases.

> 02. ddl.sql
>
> ```
> +-- check include-generated-columns option with generated column
> +CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-generated-columns', '1');
> +                            data
> +-------------------------------------------------------------
> + BEGIN
> + table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
> + table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
> + table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
> + COMMIT
> +(5 rows)
> ```
>
> We should test non-default case, which the generated columns are not generated.

Added the non-default case, which the generated columns are not generated.

> 03. ddl.sql
>
> Not sure new tests are in the correct place. Do we have to add new file and move tests to it?
> Thought?

Added the new tests in the 'decoding_into_rel.out' file.

> 04. protocol.sgml
>
> Please keep the format of the sgml file.

Fixed.

> 05. protocol.sgml
>
> The option is implemented as the streaming option of pgoutput plugin, so they should be
> located under "Logical Streaming Replication Parameters" section.

Fixed.

> 05. AlterSubscription
>
> ```
> +                               if (IsSet(opts.specified_opts, SUBOPT_GENERATED_COLUMN))
> +                               {
> +                                       ereport(ERROR,
> +                                                       (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> +                                                        errmsg("toggling generated_column option is not allowed.")));
> +                               }
> ```
>
> If you don't want to support the option, you can remove SUBOPT_GENERATED_COLUMN
> macro from the function. But can you clarify the reason why you do not want?

Fixed.

> 07. logicalrep_write_tuple
>
> ```
> -               if (!column_in_column_list(att->attnum, columns))
> +               if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
> +                       continue;
> +
> +               if (att->attgenerated && !publish_generated_column)
>                         continue;
> ```
>
> I think changes in v2 was reverted or wrongly merged.

Fixed.

> 08. test code
>
> Can you add tests that generated columns are replicated by the logical replication?

Added the test cases.

Patch v4-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJcOsk%3Dy%2BvJ3y%2BvXhzR9ZUzUEURvS_90hQW3MNfJ5di7A%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  20. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-03T10:01:46Z

    On Thu, May 23, 2024 at 5:56 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 23 May 2024 at 09:19, Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > > Dear Shubham,
> > >
> > > Thanks for creating a patch! Here are high-level comments.
> >
> > > 1.
> > > Please document the feature. If it is hard to describe, we should change the API.
> >
> > I have added the feature in the document.
> >
> > > 4.
> > > Regarding the test_decoding plugin, it has already been able to decode the
> > > generated columns. So... as the first place, is the proposed option really needed
> > > for the plugin? Why do you include it?
> > > If you anyway want to add the option, the default value should be on - which keeps
> > > current behavior.
> >
> > I have made the generated column options as true for test_decoding
> > plugin so by default we will send generated column data.
> >
> > > 5.
> > > Assuming that the feature become usable used for logical replicaiton. Not sure,
> > > should we change the protocol version at that time? Nodes prior than PG17 may
> > > not want receive values for generated columns. Can we control only by the option?
> >
> > I verified the backward compatibility test by using the generated
> > column option and it worked fine. I think there is no need to make any
> > further changes.
> >
> > > 7.
> > >
> > > Some functions refer data->publish_generated_column many times. Can we store
> > > the value to a variable?
> > >
> > > Below comments are for test_decoding part, but they may be not needed.
> > >
> > > =====
> > >
> > > a. pg_decode_startup()
> > >
> > > ```
> > > +        else if (strcmp(elem->defname, "include_generated_columns") == 0)
> > > ```
> > >
> > > Other options for test_decoding do not have underscore. It should be
> > > "include-generated-columns".
> > >
> > > b. pg_decode_change()
> > >
> > > data->include_generated_columns is referred four times in the function.
> > > Can you store the value to a varibable?
> > >
> > >
> > > c. pg_decode_change()
> > >
> > > ```
> > > -                                    true);
> > > +                                    true, data->include_generated_columns );
> > > ```
> > >
> > > Please remove the blank.
> >
> > Fixed.
> > The attached v3 Patch has the changes for the same.
>
> Few comments:
> 1) Since this is removed, tupdesc variable is not required anymore:
> +++ b/src/backend/catalog/pg_publication.c
> @@ -534,12 +534,6 @@ publication_translate_columns(Relation targetrel,
> List *columns,
>                                         errmsg("cannot use system
> column \"%s\" in publication column list",
>                                                    colname));
>
> -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> -                       ereport(ERROR,
> -
> errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> -                                       errmsg("cannot use generated
> column \"%s\" in publication column list",
> -                                                  colname));

Fixed.

> 2) In test_decoding include_generated_columns option is used:
> +               else if (strcmp(elem->defname,
> "include_generated_columns") == 0)
> +               {
> +                       if (elem->arg == NULL)
> +                               continue;
> +                       else if (!parse_bool(strVal(elem->arg),
> &data->include_generated_columns))
> +                               ereport(ERROR,
> +
> (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> +                                                errmsg("could not
> parse value \"%s\" for parameter \"%s\"",
> +
> strVal(elem->arg), elem->defname)));
> +               }
>
> In subscription we have used generated_column, we can try to use the
> same option in both places:
> +               else if (IsSet(supported_opts, SUBOPT_GENERATED_COLUMN) &&
> +                                strcmp(defel->defname,
> "generated_column") == 0)
> +               {
> +                       if (IsSet(opts->specified_opts,
> SUBOPT_GENERATED_COLUMN))
> +                               errorConflictingDefElem(defel, pstate);
> +
> +                       opts->specified_opts |= SUBOPT_GENERATED_COLUMN;
> +                       opts->generated_column = defGetBoolean(defel);
> +               }

Will update the name to 'include_generated_columns' in the next
version of the Patch.

> 3) Tab completion can be added for create subscription to include
> generated_column option

Fixed.

> 4) There are few whitespace issues while applying the patch, check for
> git diff --check

Fixed.

> 5) Add few tests for the new option added

Added new test cases.

Patch v4-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJcOsk%3Dy%2BvJ3y%2BvXhzR9ZUzUEURvS_90hQW3MNfJ5di7A%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  21. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-03T10:24:21Z

    On Fri, May 24, 2024 at 8:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> Here are some review comments for the patch v3-0001.
>
> I don't think v3 addressed any of my previous review comments for
> patches v1 and v2. [1][2]
>
> So the comments below are limited only to the new code (i.e. the v3
> versus v2 differences). Meanwhile, all my previous review comments may
> still apply.

Patch v4-0001 addresses the previous review comments for patches v1
and v2. [1][2]

> ======
> GENERAL
>
> The patch applied gives whitespace warnings:
>
> [postgres@CentOS7-x64 oss_postgres_misc]$ git apply
> ../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch
> ../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch:150:
> trailing whitespace.
>
> ../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch:202:
> trailing whitespace.
>
> ../patches_misc/v3-0001-Support-generated-column-capturing-generated-colu.patch:730:
> trailing whitespace.
> warning: 3 lines add whitespace errors.

Fixed.

> ======
> contrib/test_decoding/test_decoding.c
>
> 1. pg_decode_change
>
>   MemoryContext old;
> + bool include_generated_columns;
> +
>
> I'm not really convinced this variable saves any code.

Fixed.

> ======
> doc/src/sgml/protocol.sgml
>
> 2.
> +        <varlistentry>
> +         <term><replaceable
> class="parameter">include-generated-columns</replaceable></term>
> +         <listitem>
> +        <para>
> +        The include-generated-columns option controls whether
> generated columns should be included in the string representation of
> tuples during logical decoding in PostgreSQL. This allows users to
> customize the output format based on whether they want to include
> these columns or not.
> +         </para>
> +         </listitem>
> +         </varlistentry>
>
> 2a.
> Something is not correct when this name has hyphens and all the nearby
> parameter names do not. Shouldn't it be all uppercase like the other
> boolean parameter?
>
> ~
>
> 2b.
> Text in the SGML file should be wrapped properly.
>
> ~
>
> 2c.
> IMO the comment can be more terse and it also needs to specify that it
> is a boolean type, and what is the default value if not passed.
>
> SUGGESTION
>
> INCLUDE_GENERATED_COLUMNS [ boolean ]
>
> If true, then generated columns should be included in the string
> representation of tuples during logical decoding in PostgreSQL. The
> default is false.

Fixed.

> ======
> src/backend/replication/logical/proto.c
>
> 3. logicalrep_write_tuple
>
> - if (!column_in_column_list(att->attnum, columns))
> + if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
> + continue;
> +
> + if (att->attgenerated && !publish_generated_column)
>   continue;
>
> 3a.
> This code seems overcomplicated checking the same flag multiple times.
>
> SUGGESTION
> if (att->attgenerated)
> {
>   if (!publish_generated_column)
>     continue;
> }
> else
> {
>   if (!column_in_column_list(att->attnum, columns))
>     continue;
> }
>
> ~
>
> 3b.
> The same logic occurs several times in logicalrep_write_tuple

Fixed.

> 4. logicalrep_write_attrs
>
>   if (!column_in_column_list(att->attnum, columns))
>   continue;
>
> + if (att->attgenerated && !publish_generated_column)
> + continue;
> +
>
> Shouldn't these code fragments (2x in this function) look the same as
> in logicalrep_write_tuple? See the above review comments.

Fixed.

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 5. maybe_send_schema
>
>   TransactionId topxid = InvalidTransactionId;
> + bool publish_generated_column = data->publish_generated_column;
>
> I'm not convinced this saves any code, and anyway, it is not
> consistent with other fields in this function that are not extracted
> to another variable (e.g. data->streaming).

Fixed.

> 6. pgoutput_change
> -
> + bool publish_generated_column = data->publish_generated_column;
> +
>
> I'm not convinced this saves any code, and anyway, it is not
> consistent with other fields in this function that are not extracted
> to another variable (e.g. data->binary).

Fixed.

> ======
> [1] My v1 review -
> https://www.postgresql.org/message-id/CAHut+PsuJfcaeg6zst=6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng@mail.gmail.com
> [2] My v2 review -
> https://www.postgresql.org/message-id/CAHut%2BPv4RpOsUgkEaXDX%3DW2rhHAsJLiMWdUrUGZOcoRHuWj5%2BQ%40mail.gmail.com

Patch v4-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJcOsk%3Dy%2BvJ3y%2BvXhzR9ZUzUEURvS_90hQW3MNfJ5di7A%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  22. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-06-03T10:43:29Z

    On Mon, 3 Jun 2024 at 13:03, Shubham Khanna <khannashubham1197@gmail.com> wrote:
>
> On Thu, May 16, 2024 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are some review comments for the patch v1-0001.
> >
> > ======
> > GENERAL
> >
> > G.1. Use consistent names
> >
> > It seems to add unnecessary complications by having different names
> > for all the new options, fields and API parameters.
> >
> > e.g. sometimes 'include_generated_columns'
> > e.g. sometimes 'publish_generated_columns'
> >
> > Won't it be better to just use identical names everywhere for
> > everything? I don't mind which one you choose; I just felt you only
> > need one name, not two. This comment overrides everything else in this
> > post so whatever name you choose, make adjustments for all my other
> > review comments as necessary.
>
> I have updated the name to 'include_generated_columns' everywhere in the Patch.
>
> > ======
> >
> > G.2. Is it possible to just use the existing bms?
> >
> > A very large part of this patch is adding more API parameters to
> > delegate the 'publish_generated_columns' flag value down to when it is
> > finally checked and used. e.g.
> >
> > The functions:
> > - logicalrep_write_insert(), logicalrep_write_update(),
> > logicalrep_write_delete()
> > ... are delegating the new parameter 'publish_generated_column' down to:
> > - logicalrep_write_tuple
> >
> > The functions:
> > - logicalrep_write_rel()
> > ... are delegating the new parameter 'publish_generated_column' down to:
> > - logicalrep_write_attrs
> >
> > AFAICT in all these places the API is already passing a "Bitmapset
> > *columns". I was wondering if it might be possible to modify the
> > "Bitmapset *columns" BEFORE any of those functions get called so that
> > the "columns" BMS either does or doesn't include generated cols (as
> > appropriate according to the option).
> >
> > Well, it might not be so simple because there are some NULL BMS
> > considerations also, but I think it would be worth investigating at
> > least, because if indeed you can find some common place (somewhere
> > like pgoutput_change()?) where the columns BMS can be filtered to
> > remove bits for generated cols then it could mean none of those other
> > patch API changes are needed at all -- then the patch would only be
> > 1/2 the size.
>
> I will analyse and reply to this in the next version.
>
> > ======
> > Commit message
> >
> > 1.
> > Now if include_generated_columns option is specified, the generated
> > column information and generated column data also will be sent.
> >
> > Usage from pgoutput plugin:
> > SELECT * FROM pg_logical_slot_peek_binary_changes('slot1', NULL, NULL,
> > 'proto_version', '1', 'publication_names', 'pub1',
> > 'include_generated_columns', 'true');
> >
> > Usage from test_decoding plugin:
> > SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
> > 'include-xids', '0', 'skip-empty-xacts', '1',
> > 'include_generated_columns', '1');
> >
> > ~
> >
> > I think there needs to be more background information given here. This
> > commit message doesn't seem to describe anything about what is the
> > problem and how this patch fixes it. It just jumps straight into
> > giving usages of a 'include_generated_columns' option.
> >
> > It also doesn't say that this is an option that was newly *introduced*
> > by the patch -- it refers to it as though the reader should already
> > know about it.
> >
> > Furthermore, your hacker's post says "Currently it is not supported as
> > a subscription option because table sync for the generated column is
> > not possible as copy command does not support getting data for the
> > generated column. If this feature is required we can remove this
> > limitation from the copy command and then add it as a subscription
> > option later." IMO that all seems like the kind of information that
> > ought to also be mentioned in this commit message.
>
> I have updated the Commit message mentioning the suggested changes.
>
> > ======
> > contrib/test_decoding/sql/ddl.sql
> >
> > 2.
> > +-- check include_generated_columns option with generated column
> > +CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
> > AS (a * 2) STORED);
> > +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> > +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> > NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> > 'include_generated_columns', '1');
> > +DROP TABLE gencoltable;
> > +
> >
> > 2a.
> > Perhaps you should include both option values to demonstrate the
> > difference in behaviour:
> >
> > 'include_generated_columns', '0'
> > 'include_generated_columns', '1'
>
> Added the other option values to demonstrate the difference in behaviour:
>
> > 2b.
> > I think you maybe need to include more some test combinations where
> > there is and isn't a COLUMN LIST, because I am not 100% sure I
> > understand the current logic/expectations for all combinations.
> >
> > e.g. When the generated column is in a column list but
> > 'publish_generated_columns' is false then what should happen? etc.
> > Also if there are any special rules then those should be mentioned in
> > the commit message.
>
> Test case is added and the same is mentioned in the documentation.
>
> > ======
> > src/backend/replication/logical/proto.c
> >
> > 3.
> > For all the API changes the new parameter name should be plural.
> >
> > /publish_generated_column/publish_generated_columns/
>
> Updated the name to 'include_generated_columns'
>
> > 4. logical_rep_write_tuple:
> >
> > - if (att->attisdropped || att->attgenerated)
> > + if (att->attisdropped)
> >   continue;
> >
> > - if (!column_in_column_list(att->attnum, columns))
> > + if (!column_in_column_list(att->attnum, columns) && !att->attgenerated)
> > + continue;
> > +
> > + if (att->attgenerated && !publish_generated_column)
> >   continue;
> > That code seems confusing. Shouldn't the logic be exactly as also in
> > logicalrep_write_attrs()?
> >
> > e.g. Shouldn't they both look like this:
> >
> > if (att->attisdropped)
> >   continue;
> >
> > if (att->attgenerated && !publish_generated_column)
> >   continue;
> >
> > if (!column_in_column_list(att->attnum, columns))
> >   continue;
>
> Fixed.
>
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 5.
> >  static void send_relation_and_attrs(Relation relation, TransactionId xid,
> >   LogicalDecodingContext *ctx,
> > - Bitmapset *columns);
> > + Bitmapset *columns,
> > + bool publish_generated_column);
> >
> > Use plural. /publish_generated_column/publish_generated_columns/
>
> Updated the name to 'include_generated_columns'
>
> > 6. parse_output_parameters
> >
> >   bool origin_option_given = false;
> > + bool generate_column_option_given = false;
> >
> >   data->binary = false;
> >   data->streaming = LOGICALREP_STREAM_OFF;
> >   data->messages = false;
> >   data->two_phase = false;
> > + data->publish_generated_column = false;
> >
> > I think the 1st var should be 'include_generated_columns_option_given'
> > for consistency with the name of the actual option that was given.
>
> Updated the name to 'include_generated_columns_option_given'
>
> > ======
> > src/include/replication/logicalproto.h
> >
> > 7.
> > (Same as a previous review comment)
> >
> > For all the API changes the new parameter name should be plural.
> >
> > /publish_generated_column/publish_generated_columns/
>
> Updated the name to 'include_generated_columns'
>
> > ======
> > src/include/replication/pgoutput.h
> >
> > 8.
> >   bool publish_no_origin;
> > + bool publish_generated_column;
> >  } PGOutputData;
> >
> > /publish_generated_column/publish_generated_columns/
>
> Updated the name to 'include_generated_columns'
>
> The attached Patch contains the suggested changes.

Thanks for the updated patch, few comments:
1) The option name seems wrong here:
In one place include_generated_column is specified and other place
include_generated_columns is specified:

+               else if (IsSet(supported_opts,
SUBOPT_INCLUDE_GENERATED_COLUMN) &&
+                                strcmp(defel->defname,
"include_generated_column") == 0)
+               {
+                       if (IsSet(opts->specified_opts,
SUBOPT_INCLUDE_GENERATED_COLUMN))
+                               errorConflictingDefElem(defel, pstate);
+
+                       opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMN;
+                       opts->include_generated_column = defGetBoolean(defel);
+               }

diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..e8ff752fd9 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3365,7 +3365,7 @@ psql_completion(const char *text, int start, int end)
                COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
                                          "disable_on_error",
"enabled", "failover", "origin",
                                          "password_required",
"run_as_owner", "slot_name",
-                                         "streaming",
"synchronous_commit", "two_phase");
+                                         "streaming",
"synchronous_commit", "two_phase","include_generated_columns");

2) This small data table need not have a primary key column as it will
create an index and insertion will happen in the index too.
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
AS (a * 2) STORED);
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '1');

3) Please add a test case for this:
+          set to <literal>false</literal>. If the subscriber-side
column is also a
+          generated column then this option has no effect; the
replicated data will
+          be ignored and the subscriber column will be filled as
normal with the
+          subscriber-side computed or default data.

4) You can use a new style of ereport to remove the brackets around errcode
4.a)
+                       else if (!parse_bool(strVal(elem->arg),
&data->include_generated_columns))
+                               ereport(ERROR,
+
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+                                                errmsg("could not
parse value \"%s\" for parameter \"%s\"",
+
strVal(elem->arg), elem->defname)));

4.b) similarly here too:
+               ereport(ERROR,
+                               (errcode(ERRCODE_SYNTAX_ERROR),
+               /*- translator: both %s are strings of the form
"option = value" */
+                                       errmsg("%s and %s are mutually
exclusive options",
+                                               "copy_data = true",
"include_generated_column = true")));

4.c) similarly here too:
+                       if (include_generated_columns_option_given)
+                               ereport(ERROR,
+                                               (errcode(ERRCODE_SYNTAX_ERROR),
+                                                errmsg("conflicting
or redundant options")));

5) These variable names can be changed to keep it smaller, something
like gencol or generatedcol or gencolumn, etc
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW
  * slots) in the upstream database are enabled
  * to be synchronized to the standbys. */

+ bool subincludegeneratedcolumn; /* True if generated columns must be
published */
+
 #ifdef CATALOG_VARLEN /* variable-length fields start here */
  /* Connection string to the publisher */
  text subconninfo BKI_FORCE_NOT_NULL;
@@ -157,6 +159,7 @@ typedef struct Subscription
  List    *publications; /* List of publication names to subscribe to */
  char    *origin; /* Only publish data originating from the
  * specified origin */
+ bool includegeneratedcolumn; /* publish generated column data */
 } Subscription;

Regards,
Vignesh

  23. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-03T11:52:35Z

    >
> The attached Patch contains the suggested changes.
>

Hi,

Currently, COPY command does not work for generated columns and
therefore, COPY of generated column is not supported during tablesync
process. So, in patch v4-0001 we added a check to allow replication of
the generated column only if 'copy_data = false'.

I am attaching patches to resolve the above issues.

v5-0001: not changed
v5-0002: Support COPY of generated column
v5-0003: Support COPY of generated column during tablesync process

Thought?


Thanks and Regards,
Shlok Kyal

  24. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-04T02:41:34Z

    Hi,

Here are some review comments for patch v5-0001.

======
GENERAL G.1

The patch changes HEAD behaviour for PUBLICATION col-lists right? e.g.
maybe before they were always ignored, but now they are not?

OTOH, when 'include_generated_columns' is false then the PUBLICATION
col-list will ignore any generated cols even when they are present in
a PUBLICATION col-list, right?

These kinds of points should be noted in the commit message and in the
(col-list?) documentation.

======
Commit message

General 1a.
IMO the commit message needs some background to say something like:
"Currently generated column values are not replicated because it is
assumed that the corresponding subscriber-side table will generate its
own values for those columns."

~

General 1b.
Somewhere in this commit message, you need to give all the other
special rules --- e.g. the docs says "If the subscriber-side column is
also a generated column then this option has no effect"

~~~

2.
This commit enables support for the 'include_generated_columns' option
in logical replication, allowing the transmission of generated column
information and data alongside regular table changes. This option is
particularly useful for scenarios where applications require access to
generated column values for downstream processing or synchronization.

~

I don't think the sentence "This option is particularly useful..." is
helpful. It seems like just saying "This commit supports option XXX.
This is particularly useful if you want XXX".

~~~

3.
CREATE SUBSCRIPTION test1 connection 'dbname=postgres host=localhost port=9999
'publication pub1;

~

What is this CREATE SUBSCRIPTION for? Shouldn't it have an example of
the new parameter being used in it?

~~~

4.
Currently copy_data option with include_generated_columns option is
not supported. A future patch will remove this limitation.

~

Suggest to single-quote those parameter names for better readability.

~~~

5.
This commit aims to enhance the flexibility and utility of logical
replication by allowing users to include generated column information
in replication streams, paving the way for more robust data
synchronization and processing workflows.

~

IMO this paragraph can be omitted.

======
.../test_decoding/sql/decoding_into_rel.sql

6.
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
AS (a * 2) STORED);
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '1');
+INSERT INTO gencoltable (a) VALUES (4), (5), (6);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');
+DROP TABLE gencoltable;
+

6a.
I felt some additional explicit comments might help the readabilty of
the output file.

e.g.1
-- When 'include-generated=columns' = '1' the generated column 'b'
values will be replicated
SELECT data FROM pg_logical_slot_get_changes...

e.g.2
-- When 'include-generated=columns' = '0' the generated column 'b'
values will not be replicated
SELECT data FROM pg_logical_slot_get_changes...

~~

6b.
Suggest adding one more test case (where 'include-generated=columns'
is not set) to confirm/demonstrate the default behaviour for
replicated generated cols.

======
doc/src/sgml/protocol.sgml

7.
+    <varlistentry>
+     <term><replaceable
class="parameter">include-generated-columns</replaceable></term>
+      <listitem>
+       <para>
+        Boolean option to enable generated columns.
+        The include-generated-columns option controls whether generated
+        columns should be included in the string representation of tuples
+        during logical decoding in PostgreSQL. This allows users to
+        customize the output format based on whether they want to include
+        these columns or not. The default is false.
+       </para>
+      </listitem>
+    </varlistentry>

7a.
It doesn't render properly. e.g. Should not be bold italic (probably
the class is wrong?), because none of the nearby parameters look this
way.

~

7b.
The name here should NOT have hyphens. It needs underscores same as
all other nearby protocol parameters.

~

7c.
The description seems overly verbose.

SUGGESTION
Boolean option to enable generated columns. This option controls
whether generated columns should be included in the string
representation of tuples during logical decoding in PostgreSQL. The
default is false.

======
doc/src/sgml/ref/create_subscription.sgml

8.
+
+       <varlistentry
id="sql-createsubscription-params-with-include-generated-column">
+        <term><literal>include_generated_column</literal>
(<type>boolean</type>)</term>
+        <listitem>
+         <para>
+          Specifies whether the generated columns present in the tables
+          associated with the subscription should be replicated. The default is
+          <literal>false</literal>.
+         </para>

The parameter name should be plural (include_generated_columns).

======
src/backend/commands/subscriptioncmds.c

9.
 #define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_INCLUDE_GENERATED_COLUMN 0x00010000

Should be plural COLUMNS

~~~

10.
+ else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMN) &&
+ strcmp(defel->defname, "include_generated_column") == 0)

The new subscription parameter should be plural ("include_generated_columns").

~~~

11.
+
+ /*
+ * Do additional checking for disallowed combination when copy_data and
+ * include_generated_column are true. COPY of generated columns is
not supported
+ * yet.
+ */
+ if (opts->copy_data && opts->include_generated_column)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "copy_data = true", "include_generated_column = true")));
+ }

/combination/combinations/

The parameter name should be plural in the comment and also in the
error message.

======
src/bin/psql/tab-complete.c

12.
  COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
    "disable_on_error", "enabled", "failover", "origin",
    "password_required", "run_as_owner", "slot_name",
-   "streaming", "synchronous_commit", "two_phase");
+   "streaming", "synchronous_commit", "two_phase","include_generated_columns");

The new param should be added in alphabetical order same as all the others.

======
src/include/catalog/pg_subscription.h

13.
+ bool subincludegeneratedcolumn; /* True if generated columns must be
published */
+

The field name should be plural.

~~~

14.
+ bool includegeneratedcolumn; /* publish generated column data */
 } Subscription;

The field name should be plural.

======
src/include/replication/walreceiver.h

15.
  * prepare time */
  char    *origin; /* Only publish data originating from the
  * specified origin */
+ bool include_generated_column; /* publish generated columns */
  } logical;
  } proto;
 } WalRcvStreamOptions;

~

This new field name should be plural.

======
src/test/subscription/t/011_generated.pl

16.
+my ($cmdret, $stdout, $stderr) = $node_subscriber->psql('postgres', qq(
+ CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION
pub2 WITH (include_generated_column = true)
+));
+ok( $stderr =~
+   qr/copy_data = true and include_generated_column = true are
mutually exclusive options/,
+ 'cannot use both include_generated_column and copy_data as true');

Isn't this mutual exclusiveness of options something that could have
been tested in the regress test suite instead of TAP tests? e.g. AFAIK
you won't require a connection to test this case.

~~~

17. Missing test?

IIUC there is a missing test scenario. You can add another subscriber
table TAB3 which *already* has generated cols (e.g. generating
different values to the publisher) so then you can verify they are NOT
overwritten, even when the 'include_generated_cols' is true.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  25. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-04T04:51:30Z

    Hi,

Here are some review comments for patch v5-0002.

======
GENERAL

G1.
IIUC now you are unconditionally allowing all generated columns to be copied.

I think this is assuming that the table sync code (in the next patch
0003?) is going to explicitly name all the columns it wants to copy
(so if it wants to get generated cols then it will name the generated
cols, and if is doesn't want generated cols then it won't name them).

Maybe that is OK for the logical replication tablesync case, but I am
not sure if it will be desirable to *always* copy generated columns in
other user scenarios.

e.g. I was wondering if there should be a new COPY command option
introduced here -- INCLUDE_GENERATED_COLUMNS (with default false) so
then the current HEAD behaviour is unaffected unless that option is
enabled.

~~~

G2.
The current COPY command documentation [1] says "If no column list is
specified, all columns of the table except generated columns will be
copied."

But this 0002 patch has changed that documented behaviour, and so the
documentation needs to be changed as well, right?

======
Commit Message

1.
Currently COPY command do not copy generated column. With this commit
added support for COPY for generated column.

~

The grammar/cardinality is not good here. Try some tool (Grammarly or
chatGPT, etc) to help correct it.

======
src/backend/commands/copy.c

======
src/test/regress/expected/generated.out

======
src/test/regress/sql/generated.sql

2.
I think these COPY test cases require some explicit comments to
describe what they are doing, and what are the expected results.

Currently, I have doubts about some of this test input/output

e.g.1. Why is the 'b' column sometimes specified as 1? It needs some
explanation. Are you expecting this generated col value to be
ignored/overwritten or what?

COPY gtest1 (a, b) FROM stdin DELIMITER ' ';
5 1
6 1
\.

e.g.2. what is the reason for this new 'missing data for column "b"'
error? Or is it some introduced quirk because "b" now cannot be
generated since there is no value for "a"? I don't know if the
expected *.out here is OK or not, so some test comments may help to
clarify it.

======
[1] https://www.postgresql.org/docs/devel/sql-copy.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  26. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-04T09:30:49Z

    Hi,

Here are some review comments for patch v5-0003.

======
0. Whitespace warnings when the patch was applied.

[postgres@CentOS7-x64 oss_postgres_misc]$ git apply
../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch
../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch:29:
trailing whitespace.
          has no effect; the replicated data will be ignored and the subscriber
../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch:30:
trailing whitespace.
          column will be filled as normal with the subscriber-side computed or
../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch:189:
trailing whitespace.
(walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
warning: 3 lines add whitespace errors.

======
src/backend/commands/subscriptioncmds.c

1.
- res = walrcv_exec(wrconn, cmd.data, check_columnlist ? 3 : 2, tableRow);
+ column_count = (!include_generated_column && check_gen_col) ? 4 :
(check_columnlist ? 3 : 2);
+ res = walrcv_exec(wrconn, cmd.data, column_count, tableRow);

The 'column_count' seems out of control. Won't it be far simpler to
assign/increment the value dynamically only as required instead of the
tricky calculation at the end which is unnecessarily difficult to
understand?

~~~

2.
+ /*
+ * If include_generated_column option is false and all the column of
the table in the
+ * publication are generated then we should throw an error.
+ */
+ if (!isnull && !include_generated_column && check_gen_col)
+ {
+ attlist = DatumGetArrayTypeP(attlistdatum);
+ gen_col_count = DatumGetInt32(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
+
+ attcount = ArrayGetNItems(ARR_NDIM(attlist), ARR_DIMS(attlist));
+
+ if (attcount != 0 && attcount == gen_col_count)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use only generated column for table \"%s.%s\" in
publication when generated_column option is false",
+    nspname, relname));
+ }
+

Why do you think this new logic/error is necessary?

IIUC the 'include_generated_columns' should be false to match the
existing HEAD behavior. So this scenario where your publisher-side
table *only* has generated columns is something that could already
happen, right? IOW, this introduced error could be a candidate for
another discussion/thread/patch, but is it really required for this
current patch?

======
src/backend/replication/logical/tablesync.c

3.
  lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
-   "AND a.attgenerated = ''" : ""),
+ (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
+ (walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000 ||
+ !MySubscription->includegeneratedcolumn) ? "AND a.attgenerated = ''" : ""),

This ternary within one big appendStringInfo seems quite complicated.
Won't it be better to split the appendStringInfo into multiple parts
so the generated-cols calculation can be done more simply?

======
src/test/subscription/t/011_generated.pl

4.
I think there should be a variety of different tablesync scenarios
(when 'include_generated_columns' is true) tested here instead of just
one, and all varieties with lots of comments to say what they are
doing, expectations etc.

a. publisher-side gen-col "a" replicating to subscriber-side NOT
gen-col "a" (ok, value gets replicated)
b. publisher-side gen-col "a" replicating to subscriber-side gen-col
(ok, but ignored)
c. publisher-side NOT gen-col "b" replicating to subscriber-side
gen-col "b" (error?)

~~

5.
+$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
+is( $result, qq(1|2
+2|4
+3|6), 'generated columns initial sync with include_generated_column = true');

Should this say "ORDER BY..." so it will not fail if the row order
happens to be something unanticipated?

======

99.
Also, see the attached file with numerous other nitpicks:
- plural param- and var-names
- typos in comments
- missing spaces
- SQL keyword should be UPPERCASE
- etc.

Please apply any/all of these if you agree with them.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  27. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-05T00:18:53Z

    On Mon, Jun 3, 2024 at 9:52 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> >
> > The attached Patch contains the suggested changes.
> >
>
> Hi,
>
> Currently, COPY command does not work for generated columns and
> therefore, COPY of generated column is not supported during tablesync
> process. So, in patch v4-0001 we added a check to allow replication of
> the generated column only if 'copy_data = false'.
>
> I am attaching patches to resolve the above issues.
>
> v5-0001: not changed
> v5-0002: Support COPY of generated column
> v5-0003: Support COPY of generated column during tablesync process
>

Hi Shlok, I have a question about patch v5-0003.

According to the patch 0001 docs "If the subscriber-side column is
also a generated column then this option has no effect; the replicated
data will be ignored and the subscriber column will be filled as
normal with the subscriber-side computed or default data".

Doesn't this mean it will be a waste of effort/resources to COPY any
column value where the subscriber-side column is generated since we
know that any copied value will be ignored anyway?

But I don't recall seeing any comment or logic for this kind of copy
optimisation in the patch 0003. Is this already accounted for
somewhere and I missed it, or is my understanding wrong?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  28. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-06-06T02:59:19Z

    Dear Shlok and Shubham,

Thanks for updating the patch!

I briefly checked the v5-0002. IIUC, your patch allows to copy generated
columns unconditionally. I think the behavior affects many people so that it is
hard to get agreement.

Can we add a new option like `GENERATED_COLUMNS [boolean]`? If the default is set
to off, we can keep the current specification.

Thought?

Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/

  29. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-14T10:22:03Z

    > Thanks for the updated patch, few comments:
> 1) The option name seems wrong here:
> In one place include_generated_column is specified and other place
> include_generated_columns is specified:
>
> +               else if (IsSet(supported_opts,
> SUBOPT_INCLUDE_GENERATED_COLUMN) &&
> +                                strcmp(defel->defname,
> "include_generated_column") == 0)
> +               {
> +                       if (IsSet(opts->specified_opts,
> SUBOPT_INCLUDE_GENERATED_COLUMN))
> +                               errorConflictingDefElem(defel, pstate);
> +
> +                       opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMN;
> +                       opts->include_generated_column = defGetBoolean(defel);
> +               }

Fixed.

> diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
> index d453e224d9..e8ff752fd9 100644
> --- a/src/bin/psql/tab-complete.c
> +++ b/src/bin/psql/tab-complete.c
> @@ -3365,7 +3365,7 @@ psql_completion(const char *text, int start, int end)
>                 COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
>                                           "disable_on_error",
> "enabled", "failover", "origin",
>                                           "password_required",
> "run_as_owner", "slot_name",
> -                                         "streaming",
> "synchronous_commit", "two_phase");
> +                                         "streaming",
> "synchronous_commit", "two_phase","include_generated_columns");
>
> 2) This small data table need not have a primary key column as it will
> create an index and insertion will happen in the index too.
> +-- check include-generated-columns option with generated column
> +CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
> AS (a * 2) STORED);
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '1');

Fixed.

> 3) Please add a test case for this:
> +          set to <literal>false</literal>. If the subscriber-side
> column is also a
> +          generated column then this option has no effect; the
> replicated data will
> +          be ignored and the subscriber column will be filled as
> normal with the
> +          subscriber-side computed or default data.

Added the required test case.

> 4) You can use a new style of ereport to remove the brackets around errcode
> 4.a)
> +                       else if (!parse_bool(strVal(elem->arg),
> &data->include_generated_columns))
> +                               ereport(ERROR,
> +
> (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> +                                                errmsg("could not
> parse value \"%s\" for parameter \"%s\"",
> +
> strVal(elem->arg), elem->defname)));
>
> 4.b) similarly here too:
> +               ereport(ERROR,
> +                               (errcode(ERRCODE_SYNTAX_ERROR),
> +               /*- translator: both %s are strings of the form
> "option = value" */
> +                                       errmsg("%s and %s are mutually
> exclusive options",
> +                                               "copy_data = true",
> "include_generated_column = true")));
>
> 4.c) similarly here too:
> +                       if (include_generated_columns_option_given)
> +                               ereport(ERROR,
> +                                               (errcode(ERRCODE_SYNTAX_ERROR),
> +                                                errmsg("conflicting
> or redundant options")));

Fixed.

> 5) These variable names can be changed to keep it smaller, something
> like gencol or generatedcol or gencolumn, etc
> +++ b/src/include/catalog/pg_subscription.h
> @@ -98,6 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>   * slots) in the upstream database are enabled
>   * to be synchronized to the standbys. */
>
> + bool subincludegeneratedcolumn; /* True if generated columns must be
> published */
> +
>  #ifdef CATALOG_VARLEN /* variable-length fields start here */
>   /* Connection string to the publisher */
>   text subconninfo BKI_FORCE_NOT_NULL;
> @@ -157,6 +159,7 @@ typedef struct Subscription
>   List    *publications; /* List of publication names to subscribe to */
>   char    *origin; /* Only publish data originating from the
>   * specified origin */
> + bool includegeneratedcolumn; /* publish generated column data */
>  } Subscription;

Fixed.

The attached Patch contains the suggested changes.

Thanks and Regards,
Shubham Khanna.

  30. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-14T10:38:49Z

    On Tue, Jun 4, 2024 at 8:12 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> Here are some review comments for patch v5-0001.
>
> ======
> GENERAL G.1
>
> The patch changes HEAD behaviour for PUBLICATION col-lists right? e.g.
> maybe before they were always ignored, but now they are not?
>
> OTOH, when 'include_generated_columns' is false then the PUBLICATION
> col-list will ignore any generated cols even when they are present in
> a PUBLICATION col-list, right?
>
> These kinds of points should be noted in the commit message and in the
> (col-list?) documentation.

Fixed.

> ======
> Commit message
>
> General 1a.
> IMO the commit message needs some background to say something like:
> "Currently generated column values are not replicated because it is
> assumed that the corresponding subscriber-side table will generate its
> own values for those columns."
>
> ~
>
> General 1b.
> Somewhere in this commit message, you need to give all the other
> special rules --- e.g. the docs says "If the subscriber-side column is
> also a generated column then this option has no effect"
>
> ~~~

Fixed.

> 2.
> This commit enables support for the 'include_generated_columns' option
> in logical replication, allowing the transmission of generated column
> information and data alongside regular table changes. This option is
> particularly useful for scenarios where applications require access to
> generated column values for downstream processing or synchronization.
>
> ~
>
> I don't think the sentence "This option is particularly useful..." is
> helpful. It seems like just saying "This commit supports option XXX.
> This is particularly useful if you want XXX".
>

Fixed.

>
> 3.
> CREATE SUBSCRIPTION test1 connection 'dbname=postgres host=localhost port=9999
> 'publication pub1;
>
> ~
>
> What is this CREATE SUBSCRIPTION for? Shouldn't it have an example of
> the new parameter being used in it?
>

Added the description for this in the Patch.

>
> 4.
> Currently copy_data option with include_generated_columns option is
> not supported. A future patch will remove this limitation.
>
> ~
>
> Suggest to single-quote those parameter names for better readability.
>

Fixed.

>
> 5.
> This commit aims to enhance the flexibility and utility of logical
> replication by allowing users to include generated column information
> in replication streams, paving the way for more robust data
> synchronization and processing workflows.
>
> ~
>
> IMO this paragraph can be omitted.

Fixed.

> ======
> .../test_decoding/sql/decoding_into_rel.sql
>
> 6.
> +-- check include-generated-columns option with generated column
> +CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
> AS (a * 2) STORED);
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '1');
> +INSERT INTO gencoltable (a) VALUES (4), (5), (6);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
> +DROP TABLE gencoltable;
> +
>
> 6a.
> I felt some additional explicit comments might help the readabilty of
> the output file.
>
> e.g.1
> -- When 'include-generated=columns' = '1' the generated column 'b'
> values will be replicated
> SELECT data FROM pg_logical_slot_get_changes...
>
> e.g.2
> -- When 'include-generated=columns' = '0' the generated column 'b'
> values will not be replicated
> SELECT data FROM pg_logical_slot_get_changes...

Added the required description for this.

> 6b.
> Suggest adding one more test case (where 'include-generated=columns'
> is not set) to confirm/demonstrate the default behaviour for
> replicated generated cols.

Added the required Test case.

> ======
> doc/src/sgml/protocol.sgml
>
> 7.
> +    <varlistentry>
> +     <term><replaceable
> class="parameter">include-generated-columns</replaceable></term>
> +      <listitem>
> +       <para>
> +        Boolean option to enable generated columns.
> +        The include-generated-columns option controls whether generated
> +        columns should be included in the string representation of tuples
> +        during logical decoding in PostgreSQL. This allows users to
> +        customize the output format based on whether they want to include
> +        these columns or not. The default is false.
> +       </para>
> +      </listitem>
> +    </varlistentry>
>
> 7a.
> It doesn't render properly. e.g. Should not be bold italic (probably
> the class is wrong?), because none of the nearby parameters look this
> way.
>
> ~
>
> 7b.
> The name here should NOT have hyphens. It needs underscores same as
> all other nearby protocol parameters.
>
> ~
>
> 7c.
> The description seems overly verbose.
>
> SUGGESTION
> Boolean option to enable generated columns. This option controls
> whether generated columns should be included in the string
> representation of tuples during logical decoding in PostgreSQL. The
> default is false.

Fixed.

> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 8.
> +
> +       <varlistentry
> id="sql-createsubscription-params-with-include-generated-column">
> +        <term><literal>include_generated_column</literal>
> (<type>boolean</type>)</term>
> +        <listitem>
> +         <para>
> +          Specifies whether the generated columns present in the tables
> +          associated with the subscription should be replicated. The default is
> +          <literal>false</literal>.
> +         </para>
>
> The parameter name should be plural (include_generated_columns).

Fixed.

> ======
> src/backend/commands/subscriptioncmds.c
>
> 9.
>  #define SUBOPT_ORIGIN 0x00008000
> +#define SUBOPT_INCLUDE_GENERATED_COLUMN 0x00010000
>
> Should be plural COLUMNS
>
Fixed.

> 10.
> + else if (IsSet(supported_opts, SUBOPT_INCLUDE_GENERATED_COLUMN) &&
> + strcmp(defel->defname, "include_generated_column") == 0)
>
> The new subscription parameter should be plural ("include_generated_columns").

Fixed.

> 11.
> +
> + /*
> + * Do additional checking for disallowed combination when copy_data and
> + * include_generated_column are true. COPY of generated columns is
> not supported
> + * yet.
> + */
> + if (opts->copy_data && opts->include_generated_column)
> + {
> + ereport(ERROR,
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + /*- translator: both %s are strings of the form "option = value" */
> + errmsg("%s and %s are mutually exclusive options",
> + "copy_data = true", "include_generated_column = true")));
> + }
>
> /combination/combinations/
>
> The parameter name should be plural in the comment and also in the
> error message.

Fixed.

> ======
> src/bin/psql/tab-complete.c
>
> 12.
>   COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
>     "disable_on_error", "enabled", "failover", "origin",
>     "password_required", "run_as_owner", "slot_name",
> -   "streaming", "synchronous_commit", "two_phase");
> +   "streaming", "synchronous_commit", "two_phase","include_generated_columns");
>
> The new param should be added in alphabetical order same as all the others.

Fixed.

> ======
> src/include/catalog/pg_subscription.h
>
> 13.
> + bool subincludegeneratedcolumn; /* True if generated columns must be
> published */
> +
>
> The field name should be plural.

Fixed.

>
> 14.
> + bool includegeneratedcolumn; /* publish generated column data */
>  } Subscription;
>
> The field name should be plural.

Fixed.

> ======
> src/include/replication/walreceiver.h
>
> 15.
>   * prepare time */
>   char    *origin; /* Only publish data originating from the
>   * specified origin */
> + bool include_generated_column; /* publish generated columns */
>   } logical;
>   } proto;
>  } WalRcvStreamOptions;
>
> ~
>
> This new field name should be plural.

Fixed.

> ======
> src/test/subscription/t/011_generated.pl
>
> 16.
> +my ($cmdret, $stdout, $stderr) = $node_subscriber->psql('postgres', qq(
> + CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION
> pub2 WITH (include_generated_column = true)
> +));
> +ok( $stderr =~
> +   qr/copy_data = true and include_generated_column = true are
> mutually exclusive options/,
> + 'cannot use both include_generated_column and copy_data as true');
>
> Isn't this mutual exclusiveness of options something that could have
> been tested in the regress test suite instead of TAP tests? e.g. AFAIK
> you won't require a connection to test this case.


> 17. Missing test?
>
> IIUC there is a missing test scenario. You can add another subscriber
> table TAB3 which *already* has generated cols (e.g. generating
> different values to the publisher) so then you can verify they are NOT
> overwritten, even when the 'include_generated_cols' is true.
>
> ======

Moved this test case to the Regression test.

Patch v6-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJn6EiyAitJbbvkvVV2d45fV3Wjr2VmWFugm3RsbaU%2BRg%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  31. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-15T10:41:23Z

    On Tue, 4 Jun 2024 at 10:21, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> Here are some review comments for patch v5-0002.
>
> ======
> GENERAL
>
> G1.
> IIUC now you are unconditionally allowing all generated columns to be copied.
>
> I think this is assuming that the table sync code (in the next patch
> 0003?) is going to explicitly name all the columns it wants to copy
> (so if it wants to get generated cols then it will name the generated
> cols, and if is doesn't want generated cols then it won't name them).
>
> Maybe that is OK for the logical replication tablesync case, but I am
> not sure if it will be desirable to *always* copy generated columns in
> other user scenarios.
>
> e.g. I was wondering if there should be a new COPY command option
> introduced here -- INCLUDE_GENERATED_COLUMNS (with default false) so
> then the current HEAD behaviour is unaffected unless that option is
> enabled.
>
> ~~~
>
> G2.
> The current COPY command documentation [1] says "If no column list is
> specified, all columns of the table except generated columns will be
> copied."
>
> But this 0002 patch has changed that documented behaviour, and so the
> documentation needs to be changed as well, right?
>
> ======
> Commit Message
>
> 1.
> Currently COPY command do not copy generated column. With this commit
> added support for COPY for generated column.
>
> ~
>
> The grammar/cardinality is not good here. Try some tool (Grammarly or
> chatGPT, etc) to help correct it.
>
> ======
> src/backend/commands/copy.c
>
> ======
> src/test/regress/expected/generated.out
>
> ======
> src/test/regress/sql/generated.sql
>
> 2.
> I think these COPY test cases require some explicit comments to
> describe what they are doing, and what are the expected results.
>
> Currently, I have doubts about some of this test input/output
>
> e.g.1. Why is the 'b' column sometimes specified as 1? It needs some
> explanation. Are you expecting this generated col value to be
> ignored/overwritten or what?
>
> COPY gtest1 (a, b) FROM stdin DELIMITER ' ';
> 5 1
> 6 1
> \.
>
> e.g.2. what is the reason for this new 'missing data for column "b"'
> error? Or is it some introduced quirk because "b" now cannot be
> generated since there is no value for "a"? I don't know if the
> expected *.out here is OK or not, so some test comments may help to
> clarify it.
>
> ======
> [1] https://www.postgresql.org/docs/devel/sql-copy.html
>
Hi Peter,

I have removed the changes in the COPY command. I came up with an
approach which requires changes only in tablesync code. We can COPY
generated columns during tablesync using syntax 'COPY (SELECT
column_name from table) TO STDOUT.'

I have attached the patch for the same.
v7-0001 : Not Modified
v7-0002: Support replication of generated columns during initial sync.

Thanks and Regards,
Shlok Kyal

  32. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-15T10:44:44Z

    On Tue, 4 Jun 2024 at 15:01, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> Here are some review comments for patch v5-0003.
>
> ======
> 0. Whitespace warnings when the patch was applied.
>
> [postgres@CentOS7-x64 oss_postgres_misc]$ git apply
> ../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch
> ../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch:29:
> trailing whitespace.
>           has no effect; the replicated data will be ignored and the subscriber
> ../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch:30:
> trailing whitespace.
>           column will be filled as normal with the subscriber-side computed or
> ../patches_misc/v5-0003-Support-copy-of-generated-columns-during-tablesyn.patch:189:
> trailing whitespace.
> (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
> warning: 3 lines add whitespace errors.
>
Fixed

> ======
> src/backend/commands/subscriptioncmds.c
>
> 1.
> - res = walrcv_exec(wrconn, cmd.data, check_columnlist ? 3 : 2, tableRow);
> + column_count = (!include_generated_column && check_gen_col) ? 4 :
> (check_columnlist ? 3 : 2);
> + res = walrcv_exec(wrconn, cmd.data, column_count, tableRow);
>
> The 'column_count' seems out of control. Won't it be far simpler to
> assign/increment the value dynamically only as required instead of the
> tricky calculation at the end which is unnecessarily difficult to
> understand?
>
I have removed this piece of code.

> ~~~
>
> 2.
> + /*
> + * If include_generated_column option is false and all the column of
> the table in the
> + * publication are generated then we should throw an error.
> + */
> + if (!isnull && !include_generated_column && check_gen_col)
> + {
> + attlist = DatumGetArrayTypeP(attlistdatum);
> + gen_col_count = DatumGetInt32(slot_getattr(slot, 4, &isnull));
> + Assert(!isnull);
> +
> + attcount = ArrayGetNItems(ARR_NDIM(attlist), ARR_DIMS(attlist));
> +
> + if (attcount != 0 && attcount == gen_col_count)
> + ereport(ERROR,
> + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("cannot use only generated column for table \"%s.%s\" in
> publication when generated_column option is false",
> +    nspname, relname));
> + }
> +
>
> Why do you think this new logic/error is necessary?
>
> IIUC the 'include_generated_columns' should be false to match the
> existing HEAD behavior. So this scenario where your publisher-side
> table *only* has generated columns is something that could already
> happen, right? IOW, this introduced error could be a candidate for
> another discussion/thread/patch, but is it really required for this
> current patch?
>
Yes, this scenario can also happen in HEAD. For this patch I have
removed this check.

> ======
> src/backend/replication/logical/tablesync.c
>
> 3.
>   lrel->remoteid,
> - (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
> -   "AND a.attgenerated = ''" : ""),
> + (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
> + (walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000 ||
> + !MySubscription->includegeneratedcolumn) ? "AND a.attgenerated = ''" : ""),
>
> This ternary within one big appendStringInfo seems quite complicated.
> Won't it be better to split the appendStringInfo into multiple parts
> so the generated-cols calculation can be done more simply?
>
Fixed

> ======
> src/test/subscription/t/011_generated.pl
>
> 4.
> I think there should be a variety of different tablesync scenarios
> (when 'include_generated_columns' is true) tested here instead of just
> one, and all varieties with lots of comments to say what they are
> doing, expectations etc.
>
> a. publisher-side gen-col "a" replicating to subscriber-side NOT
> gen-col "a" (ok, value gets replicated)
> b. publisher-side gen-col "a" replicating to subscriber-side gen-col
> (ok, but ignored)
> c. publisher-side NOT gen-col "b" replicating to subscriber-side
> gen-col "b" (error?)
>
Added the tests

> ~~
>
> 5.
> +$result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab3");
> +is( $result, qq(1|2
> +2|4
> +3|6), 'generated columns initial sync with include_generated_column = true');
>
> Should this say "ORDER BY..." so it will not fail if the row order
> happens to be something unanticipated?
>
Fixed

> ======
>
> 99.
> Also, see the attached file with numerous other nitpicks:
> - plural param- and var-names
> - typos in comments
> - missing spaces
> - SQL keyword should be UPPERCASE
> - etc.
>
> Please apply any/all of these if you agree with them.
Fixed

Patch 7-0002 contains all the changes. Please refer [1]
[1]: https://www.postgresql.org/message-id/CANhcyEUz0FcyR3T76b%2BNhtmvWO7o96O_oEwsLZNZksEoPmVzXw%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  33. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-15T10:46:57Z

    On Wed, 5 Jun 2024 at 05:49, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Jun 3, 2024 at 9:52 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > >
> > > The attached Patch contains the suggested changes.
> > >
> >
> > Hi,
> >
> > Currently, COPY command does not work for generated columns and
> > therefore, COPY of generated column is not supported during tablesync
> > process. So, in patch v4-0001 we added a check to allow replication of
> > the generated column only if 'copy_data = false'.
> >
> > I am attaching patches to resolve the above issues.
> >
> > v5-0001: not changed
> > v5-0002: Support COPY of generated column
> > v5-0003: Support COPY of generated column during tablesync process
> >
>
> Hi Shlok, I have a question about patch v5-0003.
>
> According to the patch 0001 docs "If the subscriber-side column is
> also a generated column then this option has no effect; the replicated
> data will be ignored and the subscriber column will be filled as
> normal with the subscriber-side computed or default data".
>
> Doesn't this mean it will be a waste of effort/resources to COPY any
> column value where the subscriber-side column is generated since we
> know that any copied value will be ignored anyway?
>
> But I don't recall seeing any comment or logic for this kind of copy
> optimisation in the patch 0003. Is this already accounted for
> somewhere and I missed it, or is my understanding wrong?
Your understanding is correct.
With v7-0002, if a subscriber-side column is generated, then we do not
include that column in the column list during COPY. This will address
the above issue.

Patch 7-0002 contains all the changes. Please refer [1]
[1]: https://www.postgresql.org/message-id/CANhcyEUz0FcyR3T76b%2BNhtmvWO7o96O_oEwsLZNZksEoPmVzXw%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  34. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-15T10:48:31Z

    On Thu, 6 Jun 2024 at 08:29, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Shlok and Shubham,
>
> Thanks for updating the patch!
>
> I briefly checked the v5-0002. IIUC, your patch allows to copy generated
> columns unconditionally. I think the behavior affects many people so that it is
> hard to get agreement.
>
> Can we add a new option like `GENERATED_COLUMNS [boolean]`? If the default is set
> to off, we can keep the current specification.
>
> Thought?
Hi Kuroda-san,

I agree that we should not allow to copy generated columns unconditionally.
With patch v7-0002, I have used a different approach which does not
require any code changes in COPY.

Please refer [1] for patch v7-0002.
[1]: https://www.postgresql.org/message-id/CANhcyEUz0FcyR3T76b%2BNhtmvWO7o96O_oEwsLZNZksEoPmVzXw%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  35. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-17T06:20:33Z

    On Fri, 14 Jun 2024 at 15:52, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> The attached Patch contains the suggested changes.
>

Hi Shubham, thanks for providing a patch.
I have some comments for v6-0001.

1. create_subscription.sgml
There is repetition of the same line.

+         <para>
+          Specifies whether the generated columns present in the tables
+          associated with the subscription should be replicated. If the
+          subscriber-side column is also a generated column then this option
+          has no effect; the replicated data will be ignored and the subscriber
+          column will be filled as normal with the subscriber-side computed or
+          default data.
+          <literal>false</literal>.
+         </para>
+
+         <para>
+          This parameter can only be set true if
<literal>copy_data</literal> is
+          set to <literal>false</literal>. If the subscriber-side
column is also a
+          generated column then this option has no effect; the
replicated data will
+          be ignored and the subscriber column will be filled as
normal with the
+          subscriber-side computed or default data.
+         </para>

==============================
2. subscriptioncmds.c

2a. The macro name should be in uppercase. We can use a short name
like 'SUBOPT_INCLUDE_GEN_COL'. Thought?
+#define SUBOPT_include_generated_columns 0x00010000

2b.Update macro name accordingly
+ if (IsSet(supported_opts, SUBOPT_include_generated_columns))
+ opts->include_generated_columns = false;

2c. Update macro name accordingly
+ else if (IsSet(supported_opts, SUBOPT_include_generated_columns) &&
+ strcmp(defel->defname, "include_generated_columns") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_include_generated_columns))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_include_generated_columns;
+ opts->include_generated_columns = defGetBoolean(defel);
+ }

2d. Update macro name accordingly
+   SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
+   SUBOPT_include_generated_columns);


==============================

3. decoding_into_rel.out

3a. In comment, I think it should be "When 'include-generated-columns'
= '1' the generated column 'b' values will be replicated"
+-- When 'include-generated-columns' = '1' the generated column 'b'
values will not be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '1');
+                            data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT

3b. In comment, I think it should be "When 'include-generated-columns'
= '1' the generated column 'b' values will not be replicated"
+-- When 'include-generated-columns' = '0' the generated column 'b'
values will be replicated
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');
+                      data
+------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:4
+ table public.gencoltable: INSERT: a[integer]:5
+ table public.gencoltable: INSERT: a[integer]:6
+ COMMIT
+(5 rows)

=========================

4. Here names for both the tests are the same. I think we should use
different names.

+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
+is( $result, qq(4|8
+5|10), 'generated columns replicated to non-generated column on subscriber');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
+is( $result, qq(4|24
+5|25), 'generated columns replicated to non-generated column on subscriber');

Thanks and Regards,
Shlok Kyal

  36. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-06-17T06:27:05Z

    On Fri, 14 Jun 2024 at 15:52, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> > Thanks for the updated patch, few comments:
> > 1) The option name seems wrong here:
> > In one place include_generated_column is specified and other place
> > include_generated_columns is specified:
> >
> > +               else if (IsSet(supported_opts,
> > SUBOPT_INCLUDE_GENERATED_COLUMN) &&
> > +                                strcmp(defel->defname,
> > "include_generated_column") == 0)
> > +               {
> > +                       if (IsSet(opts->specified_opts,
> > SUBOPT_INCLUDE_GENERATED_COLUMN))
> > +                               errorConflictingDefElem(defel, pstate);
> > +
> > +                       opts->specified_opts |= SUBOPT_INCLUDE_GENERATED_COLUMN;
> > +                       opts->include_generated_column = defGetBoolean(defel);
> > +               }
>
> Fixed.
>
> > diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
> > index d453e224d9..e8ff752fd9 100644
> > --- a/src/bin/psql/tab-complete.c
> > +++ b/src/bin/psql/tab-complete.c
> > @@ -3365,7 +3365,7 @@ psql_completion(const char *text, int start, int end)
> >                 COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
> >                                           "disable_on_error",
> > "enabled", "failover", "origin",
> >                                           "password_required",
> > "run_as_owner", "slot_name",
> > -                                         "streaming",
> > "synchronous_commit", "two_phase");
> > +                                         "streaming",
> > "synchronous_commit", "two_phase","include_generated_columns");
> >
> > 2) This small data table need not have a primary key column as it will
> > create an index and insertion will happen in the index too.
> > +-- check include-generated-columns option with generated column
> > +CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS
> > AS (a * 2) STORED);
> > +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> > +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> > NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> > 'include-generated-columns', '1');
>
> Fixed.
>
> > 3) Please add a test case for this:
> > +          set to <literal>false</literal>. If the subscriber-side
> > column is also a
> > +          generated column then this option has no effect; the
> > replicated data will
> > +          be ignored and the subscriber column will be filled as
> > normal with the
> > +          subscriber-side computed or default data.
>
> Added the required test case.
>
> > 4) You can use a new style of ereport to remove the brackets around errcode
> > 4.a)
> > +                       else if (!parse_bool(strVal(elem->arg),
> > &data->include_generated_columns))
> > +                               ereport(ERROR,
> > +
> > (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> > +                                                errmsg("could not
> > parse value \"%s\" for parameter \"%s\"",
> > +
> > strVal(elem->arg), elem->defname)));
> >
> > 4.b) similarly here too:
> > +               ereport(ERROR,
> > +                               (errcode(ERRCODE_SYNTAX_ERROR),
> > +               /*- translator: both %s are strings of the form
> > "option = value" */
> > +                                       errmsg("%s and %s are mutually
> > exclusive options",
> > +                                               "copy_data = true",
> > "include_generated_column = true")));
> >
> > 4.c) similarly here too:
> > +                       if (include_generated_columns_option_given)
> > +                               ereport(ERROR,
> > +                                               (errcode(ERRCODE_SYNTAX_ERROR),
> > +                                                errmsg("conflicting
> > or redundant options")));
>
> Fixed.
>
> > 5) These variable names can be changed to keep it smaller, something
> > like gencol or generatedcol or gencolumn, etc
> > +++ b/src/include/catalog/pg_subscription.h
> > @@ -98,6 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> > BKI_SHARED_RELATION BKI_ROW
> >   * slots) in the upstream database are enabled
> >   * to be synchronized to the standbys. */
> >
> > + bool subincludegeneratedcolumn; /* True if generated columns must be
> > published */
> > +
> >  #ifdef CATALOG_VARLEN /* variable-length fields start here */
> >   /* Connection string to the publisher */
> >   text subconninfo BKI_FORCE_NOT_NULL;
> > @@ -157,6 +159,7 @@ typedef struct Subscription
> >   List    *publications; /* List of publication names to subscribe to */
> >   char    *origin; /* Only publish data originating from the
> >   * specified origin */
> > + bool includegeneratedcolumn; /* publish generated column data */
> >  } Subscription;
>
> Fixed.
>
> The attached Patch contains the suggested changes.

Few comments:
1) Here tab1 and tab2 are exactly the same tables, just check if the
table tab1 itself can be used for your tests.
@@ -24,20 +24,50 @@ $node_publisher->safe_psql('postgres',
        "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS
AS (a * 2) STORED)"
 );
+$node_publisher->safe_psql('postgres',
+       "CREATE TABLE tab2 (a int PRIMARY KEY, b int GENERATED ALWAYS
AS (a * 2) STORED)"
+);

2) We can document  that the include_generate_columns option cannot be altered.

3) You can mention that include-generated-columns is true by default
and generated column data will be selected
+-- When 'include-generated-columns' is not set
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+                            data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)

4)  The comment seems to be wrong here, the comment says b will not be
replicated but b is being selected:
-- When 'include-generated-columns' = '1' the generated column 'b'
values will not be replicated
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '1');
                            data
-------------------------------------------------------------
 BEGIN
 table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
 table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
 table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
 COMMIT
(5 rows)

5)  Similarly here too the comment seems to be wrong, the comment says
b will not replicated but b is not being selected:
INSERT INTO gencoltable (a) VALUES (4), (5), (6);
-- When 'include-generated-columns' = '0' the generated column 'b'
values will be replicated
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');
                      data
------------------------------------------------
 BEGIN
 table public.gencoltable: INSERT: a[integer]:4
 table public.gencoltable: INSERT: a[integer]:5
 table public.gencoltable: INSERT: a[integer]:6
 COMMIT
(5 rows)

6) SUBOPT_include_generated_columns change it to SUBOPT_GENERATED to
keep the name consistent:
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -72,6 +72,7 @@
 #define SUBOPT_FAILOVER                                0x00002000
 #define SUBOPT_LSN                                     0x00004000
 #define SUBOPT_ORIGIN                          0x00008000
+#define SUBOPT_include_generated_columns               0x00010000

7) The comment style seems to be inconsistent, both of them can start
in lower case
+-- check include-generated-columns option with generated column
+CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);
+-- When 'include-generated-columns' is not set
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+                            data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)
+
+-- When 'include-generated-columns' = '1' the generated column 'b'
values will not be replicated

8) This could be changed to remove the insert statements by using
pg_logical_slot_peek_changes:
-- When 'include-generated-columns' is not set
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-- When 'include-generated-columns' = '1' the generated column 'b'
values will not be replicated
INSERT INTO gencoltable (a) VALUES (1), (2), (3);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '1');
INSERT INTO gencoltable (a) VALUES (4), (5), (6);
-- When 'include-generated-columns' = '0' the generated column 'b'
values will be replicated
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');
to:
-- When 'include-generated-columns' is not set
SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-- When 'include-generated-columns' = '1' the generated column 'b'
values will not be replicated
SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '1');
-- When 'include-generated-columns' = '0' the generated column 'b'
values will be replicated
SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');

9) In commit message  the  option used is wrong
include_generated_columns should actually be
include-generated-columns:
Usage from test_decoding plugin:
SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
'include-xids', '0', 'skip-empty-xacts', '1',
                                      'include_generated_columns','1');

Regards,
Vignesh

  37. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-17T08:27:30Z

    Hi, here are my review comments for patch v7-0001.

======
1. GENERAL - \dRs+

Shouldn't the new SUBSCRIPTION parameter be exposed via "describe"
(e.g. \dRs+ mysub) the same as the other boolean parameters?

======
Commit message

2.
When 'include_generated_columns' is false then the PUBLICATION
col-list will ignore any generated cols even when they are present in
a PUBLICATION col-list

~

Maybe you don't need to mention "PUBLICATION col-list" twice.

SUGGESTION
When 'include_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.

~~~

2.
CREATE SUBSCRIPTION test1 connection 'dbname=postgres host=localhost port=9999
'publication pub1;

~

2a.
(I've questioned this one in previous reviews)

What exactly is the purpose of this statement in the commit message?
Was this supposed to demonstrate the usage of the
'include_generated_columns' parameter?

~

2b.
/publication/ PUBLICATION/


~~~

3.
If the subscriber-side column is also a generated column then
thisoption has no effect; the replicated data will be ignored and the
subscriber column will be filled as normal with the subscriber-side
computed or default data.

~

Missing space: /thisoption/this option/

======
.../expected/decoding_into_rel.out

4.
+-- When 'include-generated-columns' is not set
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+                            data
+-------------------------------------------------------------
+ BEGIN
+ table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
+ table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
+ table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
+ COMMIT
+(5 rows)

Why is the default value here equivalent to
'include-generated-columns' = '1' here instead of '0'? The default for
the CREATE SUBSCRIPTION parameter 'include_generated_columns' is
false, and IMO it seems confusing for these 2 defaults to be
different. Here I think it should default to '0' *regardless* of what
the previous functionality might have done -- e.g. this is a "test
decoder" so the parameter should behave sensibly.

======
.../test_decoding/sql/decoding_into_rel.sql

NITPICK - wrong comments.

======
doc/src/sgml/protocol.sgml

5.
+    <varlistentry>
+     <term>include_generated_columns</term>
+      <listitem>
+       <para>
+        Boolean option to enable generated columns. This option controls
+        whether generated columns should be included in the string
+        representation of tuples during logical decoding in PostgreSQL.
+        The default is false.
+       </para>
+      </listitem>
+    </varlistentry>
+

Does the protocol version need to be bumped to support this new option
and should that be mentioned on this page similar to how all other
version values are mentioned?

======
doc/src/sgml/ref/create_subscription.sgml

NITPICK - some missing words/sentence.
NITPICK - some missing <literal> tags.
NITPICK - remove duplicated sentence.
NITPICK - add another <para>.

======
src/backend/commands/subscriptioncmds.c

6.
 #define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_include_generated_columns 0x00010000

Please use UPPERCASE for consistency with other macros.

======
.../libpqwalreceiver/libpqwalreceiver.c

7.
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'on'");
+

IMO it makes more sense to say 'true' here instead of 'on'. It seems
like this was just cut/paste from the above code (where 'on' was
sensible).

======
src/include/catalog/pg_subscription.h

8.
@@ -98,6 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
BKI_SHARED_RELATION BKI_ROW
  * slots) in the upstream database are enabled
  * to be synchronized to the standbys. */

+ bool subincludegencol; /* True if generated columns must be published */
+

Not fixed as claimed. This field name ought to be plural.

/subincludegencol/subincludegencols/

~~~

9.
  char    *origin; /* Only publish data originating from the
  * specified origin */
+ bool includegencol; /* publish generated column data */
 } Subscription;

Not fixed as claimed. This field name ought to be plural.

/includegencol/includegencols/

======
src/test/subscription/t/031_column_list.pl

10.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
* 2) STORED)"
+);
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
+ 10) STORED)"
+);
+
 $node_subscriber->safe_psql('postgres',
  "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
* 22) STORED, c int)"
 );

+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b int)"
+);
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
+ 20) STORED)"
+);

IMO the test needs lots more comments to describe what it is doing:

For example, the setup deliberately has made:
* publisher-side tab2 has generated col 'b' but subscriber-side tab2
has NON-gnerated col 'b'.
* publisher-side tab3 has generated col 'b' but subscriber-side tab2
has DIFFERENT COMPUTATION generated col 'b'.

So it will be better to have comments to explain all this instead of
having to figure it out.

~~~

11.
 # data for initial sync

 $node_publisher->safe_psql('postgres',
  "INSERT INTO tab1 (a) VALUES (1), (2), (3)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 (a) VALUES (1), (2), (3)");

 $node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE tab2");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE tab3");
+

# Wait for initial sync of all subscriptions
$node_subscriber->wait_for_subscription_sync;

my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
is( $result, qq(1|22
2|44
3|66), 'generated columns initial sync');

~

IMO (and for completeness) it would be better to INSERT data for all
the tables and alsot to validate that tables tab2 and tab3 has zero
rows replicated. Yes, I know there is 'copy_data=false', but it is
just easier to see all the tables instead of guessing why some are
omitted, and anyway this test case will be needed after the next patch
implements the COPY support for gen-cols.

~~~

12.
+$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub2');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
+is( $result, qq(4|8
+5|10), 'generated columns replicated to non-generated column on subscriber');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
+is( $result, qq(4|24
+5|25), 'generated columns replicated to non-generated column on subscriber');
+

Here also I think there should be explicit comments about what these
cases are testing, what results you are expecting, and why. The
comments will look something like the message parameter of those
safe_psql(...)

e.g.
# confirm generated columns ARE replicated when the subscriber-side
column is not generated

e.g.
# confirm generated columns are NOT replicated when the
subscriber-side column is also generated

======

99.
Please also see my nitpicks attachment patch for various other
cosmetic and docs problems, and apply theseif you agree:
- documentation wording/rendering
- wrong comments
- spacing
- etc.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  38. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-18T05:26:45Z

    Hi, here are my review comments for patch v7-0002

======
Commit Message

NITPICKS
- rearrange paragraphs
- typo "donot"
- don't start a sentence with "And"
- etc.

Please see the attachment for my suggested commit message text updates
and take from it whatever you agree with.

======
doc/src/sgml/ref/create_subscription.sgml

1.
+          If the subscriber-side column is also a generated column
then this option
+          has no effect; the replicated data will be ignored and the subscriber
+          column will be filled as normal with the subscriber-side computed or
+          default data. And during table synchronization, the data
corresponding to
+          the generated column on subscriber-side will not be sent from the
+          publisher to the subscriber.

This text already mentions subscriber-side generated cols. IMO you
don't need to say anything at all about table synchronization --
that's just an internal code optimization, which is not something the
user needs to know about. IOW, the entire last sentence ("And
during...") should be removed.

======
src/backend/replication/logical/relation.c

2. logicalrep_rel_open

- if (attr->attisdropped)
+ if (attr->attisdropped ||
+ (!MySubscription->includegencol && attr->attgenerated))
  {
  entry->attrmap->attnums[i] = -1;
  continue;

~

Maybe I'm mistaken, but isn't this code for skipping checking for
"missing" subscriber-side (aka local) columns? Can't it just
unconditionally skip every attr->attgenerated -- i.e. why does it
matter if the MySubscription->includegencol was set or not?

======
src/backend/replication/logical/tablesync.c

3. make_copy_attnamelist

- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+
+ for (i = 0; i < desc->natts; i++)
  {
- attnamelist = lappend(attnamelist,
-   makeString(rel->remoterel.attnames[i]));
+ int attnum;
+ Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+ if (!attr->attgenerated)
+ continue;
+
+ attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+ NameStr(attr->attname));
+
+ /*
+ * Check if subscription table have a generated column with same
+ * column name as a non-generated column in the corresponding
+ * publication table.
+ */
+ if (attnum >=0 && !attgenlist[attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" is missing
replicated column: \"%s\"",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+
+ if (attnum >= 0)
+ gencollist = lappend_int(gencollist, attnum);
  }

~

NITPICK - Use C99-style for loop variables
NITPICK - Typo in comment
NITPICK - spaces

~

3a.
I think above code should be refactored so there is only one check for
"if (attnum >= 0)" -- e.g. other condition should be nested.

~

3b.
That ERROR message says "missing replicated column", but that doesn't
seem much like what the code-comment was saying this code is about.

~~~

4.
+ for (i = 0; i < rel->remoterel.natts; i++)
+ {
+
+ if (gencollist != NIL && j < gencollist->length &&
+ list_nth_int(gencollist, j) == i)
+ j++;
+ else
+ attnamelist = lappend(attnamelist,
+   makeString(rel->remoterel.attnames[i]));
+ }

NITPICK - Use C99-style for loop variables
NITPICK - Unnecessary blank lines

~

IIUC the subscriber-side table and the publisher-side table do NOT
have to have all the columns in identical order for the logical
replication to work correcly. AFAIK it works fine so long as the
column names match for the replicated columns. Therefore, I am
suspicious that this new patch code seems to be imposing some new
ordering assumptions/restrictions (e.g. list_nth_int stuff) which are
not current requirements.

~~~

copy_table:

NITPICK - comment typo
NITPICK - comment wording

~

5.
+ int i = 0;
+ ListCell *l;
+
  appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach(l, attnamelist)
  {
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, quote_identifier(strVal(lfirst(l))));
+ if (i < attnamelist->length - 1)
  appendStringInfoString(&cmd, ", ");
+ i++;
  }
IIUC for new code like this, it is preferred to use the foreach*
macros instead of ListCell.

======
src/test/regress/sql/subscription.sql

6.
--- fail - copy_data and include_generated_columns are mutually
exclusive options
-CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist'
PUBLICATION testpub WITH (include_generated_columns = true);
-ERROR:  copy_data = true and include_generated_columns = true are
mutually exclusive options

It is OK to delete this test now but IMO still needs to be some
"include_generated_columns must be boolean" test cases (e.g. same as
there was two_phase). Actually, this should probably be done by the
0001 patch.

======
src/test/subscription/t/011_generated.pl

7.
All the PRIMARY KEY stuff may be overkill. Are primary keys really
needed for these tests?

~~~

8.
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
* 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
+);
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab5 (a int PRIMARY KEY, b int)"
+);
+

Maybe add comments on what is special about all these tables, so don't
have to read the tests later to deduce their purpose.

tab4: publisher-side generated col 'b' and 'c'  ==> subscriber-side
non-generated col 'b', and generated-col 'c'
tab5: publisher-side non-generated col 'b' --> subscriber-side
non-generated col 'b'

~~~

9.
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr'
PUBLICATION pub4 WITH (include_generated_columns = true)"
+ );
+

All the publications are created together, and all the subscriptions
are created together except for 'sub5'. Consider including a comment
to say why you deliberately created the 'sub5' subscription separate
from all others.

======

99.
Please also see my code nitpicks attachment patch for various other
cosmetic problems, and apply them if you agree.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  39. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-19T11:22:20Z

    > Hi Shubham, thanks for providing a patch.
> I have some comments for v6-0001.
>
> 1. create_subscription.sgml
> There is repetition of the same line.
>
> +         <para>
> +          Specifies whether the generated columns present in the tables
> +          associated with the subscription should be replicated. If the
> +          subscriber-side column is also a generated column then this option
> +          has no effect; the replicated data will be ignored and the subscriber
> +          column will be filled as normal with the subscriber-side computed or
> +          default data.
> +          <literal>false</literal>.
> +         </para>
> +
> +         <para>
> +          This parameter can only be set true if
> <literal>copy_data</literal> is
> +          set to <literal>false</literal>. If the subscriber-side
> column is also a
> +          generated column then this option has no effect; the
> replicated data will
> +          be ignored and the subscriber column will be filled as
> normal with the
> +          subscriber-side computed or default data.
> +         </para>
>
> ==============================
> 2. subscriptioncmds.c
>
> 2a. The macro name should be in uppercase. We can use a short name
> like 'SUBOPT_INCLUDE_GEN_COL'. Thought?
> +#define SUBOPT_include_generated_columns 0x00010000
>
> 2b.Update macro name accordingly
> + if (IsSet(supported_opts, SUBOPT_include_generated_columns))
> + opts->include_generated_columns = false;
>
> 2c. Update macro name accordingly
> + else if (IsSet(supported_opts, SUBOPT_include_generated_columns) &&
> + strcmp(defel->defname, "include_generated_columns") == 0)
> + {
> + if (IsSet(opts->specified_opts, SUBOPT_include_generated_columns))
> + errorConflictingDefElem(defel, pstate);
> +
> + opts->specified_opts |= SUBOPT_include_generated_columns;
> + opts->include_generated_columns = defGetBoolean(defel);
> + }
>
> 2d. Update macro name accordingly
> +   SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN |
> +   SUBOPT_include_generated_columns);
>
>
> ==============================
>
> 3. decoding_into_rel.out
>
> 3a. In comment, I think it should be "When 'include-generated-columns'
> = '1' the generated column 'b' values will be replicated"
> +-- When 'include-generated-columns' = '1' the generated column 'b'
> values will not be replicated
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '1');
> +                            data
> +-------------------------------------------------------------
> + BEGIN
> + table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
> + table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
> + table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
> + COMMIT
>
> 3b. In comment, I think it should be "When 'include-generated-columns'
> = '1' the generated column 'b' values will not be replicated"
> +-- When 'include-generated-columns' = '0' the generated column 'b'
> values will be replicated
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
> +                      data
> +------------------------------------------------
> + BEGIN
> + table public.gencoltable: INSERT: a[integer]:4
> + table public.gencoltable: INSERT: a[integer]:5
> + table public.gencoltable: INSERT: a[integer]:6
> + COMMIT
> +(5 rows)
>
> =========================
>
> 4. Here names for both the tests are the same. I think we should use
> different names.
>
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
> +is( $result, qq(4|8
> +5|10), 'generated columns replicated to non-generated column on subscriber');
> +
> +$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
> +
> +$node_publisher->wait_for_catchup('sub3');
> +
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
> +is( $result, qq(4|24
> +5|25), 'generated columns replicated to non-generated column on subscriber');

All the comments are handled.

The attached Patch contains all the suggested changes.

Thanks and Regards,
Shubham Khanna.

  40. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-19T11:36:31Z

    > Few comments:
> 1) Here tab1 and tab2 are exactly the same tables, just check if the
> table tab1 itself can be used for your tests.
> @@ -24,20 +24,50 @@ $node_publisher->safe_psql('postgres',
>         "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS
> AS (a * 2) STORED)"
>  );
> +$node_publisher->safe_psql('postgres',
> +       "CREATE TABLE tab2 (a int PRIMARY KEY, b int GENERATED ALWAYS
> AS (a * 2) STORED)"
> +);

On the subscription side the tables have different descriptions, so we
need to have different tables on the publisher side.

> 2) We can document  that the include_generate_columns option cannot be altered.
>
> 3) You can mention that include-generated-columns is true by default
> and generated column data will be selected
> +-- When 'include-generated-columns' is not set
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
> +                            data
> +-------------------------------------------------------------
> + BEGIN
> + table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
> + table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
> + table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
> + COMMIT
> +(5 rows)
>
> 4)  The comment seems to be wrong here, the comment says b will not be
> replicated but b is being selected:
> -- When 'include-generated-columns' = '1' the generated column 'b'
> values will not be replicated
> INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '1');
>                             data
> -------------------------------------------------------------
>  BEGIN
>  table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
>  table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
>  table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
>  COMMIT
> (5 rows)
>
> 5)  Similarly here too the comment seems to be wrong, the comment says
> b will not replicated but b is not being selected:
> INSERT INTO gencoltable (a) VALUES (4), (5), (6);
> -- When 'include-generated-columns' = '0' the generated column 'b'
> values will be replicated
> SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
>                       data
> ------------------------------------------------
>  BEGIN
>  table public.gencoltable: INSERT: a[integer]:4
>  table public.gencoltable: INSERT: a[integer]:5
>  table public.gencoltable: INSERT: a[integer]:6
>  COMMIT
> (5 rows)
>
> 6) SUBOPT_include_generated_columns change it to SUBOPT_GENERATED to
> keep the name consistent:
> --- a/src/backend/commands/subscriptioncmds.c
> +++ b/src/backend/commands/subscriptioncmds.c
> @@ -72,6 +72,7 @@
>  #define SUBOPT_FAILOVER                                0x00002000
>  #define SUBOPT_LSN                                     0x00004000
>  #define SUBOPT_ORIGIN                          0x00008000
> +#define SUBOPT_include_generated_columns               0x00010000
>
> 7) The comment style seems to be inconsistent, both of them can start
> in lower case
> +-- check include-generated-columns option with generated column
> +CREATE TABLE gencoltable (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> +-- When 'include-generated-columns' is not set
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
> +                            data
> +-------------------------------------------------------------
> + BEGIN
> + table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
> + table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
> + table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
> + COMMIT
> +(5 rows)
> +
> +-- When 'include-generated-columns' = '1' the generated column 'b'
> values will not be replicated
>
> 8) This could be changed to remove the insert statements by using
> pg_logical_slot_peek_changes:
> -- When 'include-generated-columns' is not set
> SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
> -- When 'include-generated-columns' = '1' the generated column 'b'
> values will not be replicated
> INSERT INTO gencoltable (a) VALUES (1), (2), (3);
> SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '1');
> INSERT INTO gencoltable (a) VALUES (4), (5), (6);
> -- When 'include-generated-columns' = '0' the generated column 'b'
> values will be replicated
> SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
> to:
> -- When 'include-generated-columns' is not set
> SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
> -- When 'include-generated-columns' = '1' the generated column 'b'
> values will not be replicated
> SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '1');
> -- When 'include-generated-columns' = '0' the generated column 'b'
> values will be replicated
> SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
>
> 9) In commit message  the  option used is wrong
> include_generated_columns should actually be
> include-generated-columns:
> Usage from test_decoding plugin:
> SELECT data FROM pg_logical_slot_get_changes('slot2', NULL, NULL,
> 'include-xids', '0', 'skip-empty-xacts', '1',
>                                       'include_generated_columns','1');

All the comments are handled.

Patch v8-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2BAi0CgtXiAga82bWpWB8fVcOWycNyJ_jqXm788v3R8rQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  41. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-19T12:00:06Z

    On Mon, Jun 17, 2024 at 1:57 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for patch v7-0001.
>
> ======
> 1. GENERAL - \dRs+
>
> Shouldn't the new SUBSCRIPTION parameter be exposed via "describe"
> (e.g. \dRs+ mysub) the same as the other boolean parameters?
>
> ======
> Commit message
>
> 2.
> When 'include_generated_columns' is false then the PUBLICATION
> col-list will ignore any generated cols even when they are present in
> a PUBLICATION col-list
>
> ~
>
> Maybe you don't need to mention "PUBLICATION col-list" twice.
>
> SUGGESTION
> When 'include_generated_columns' is false, generated columns are not
> replicated, even when present in a PUBLICATION col-list.
>
> ~~~
>
> 2.
> CREATE SUBSCRIPTION test1 connection 'dbname=postgres host=localhost port=9999
> 'publication pub1;
>
> ~
>
> 2a.
> (I've questioned this one in previous reviews)
>
> What exactly is the purpose of this statement in the commit message?
> Was this supposed to demonstrate the usage of the
> 'include_generated_columns' parameter?
>
> ~
>
> 2b.
> /publication/ PUBLICATION/
>
>
> ~~~
>
> 3.
> If the subscriber-side column is also a generated column then
> thisoption has no effect; the replicated data will be ignored and the
> subscriber column will be filled as normal with the subscriber-side
> computed or default data.
>
> ~
>
> Missing space: /thisoption/this option/
>
> ======
> .../expected/decoding_into_rel.out
>
> 4.
> +-- When 'include-generated-columns' is not set
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
> +                            data
> +-------------------------------------------------------------
> + BEGIN
> + table public.gencoltable: INSERT: a[integer]:1 b[integer]:2
> + table public.gencoltable: INSERT: a[integer]:2 b[integer]:4
> + table public.gencoltable: INSERT: a[integer]:3 b[integer]:6
> + COMMIT
> +(5 rows)
>
> Why is the default value here equivalent to
> 'include-generated-columns' = '1' here instead of '0'? The default for
> the CREATE SUBSCRIPTION parameter 'include_generated_columns' is
> false, and IMO it seems confusing for these 2 defaults to be
> different. Here I think it should default to '0' *regardless* of what
> the previous functionality might have done -- e.g. this is a "test
> decoder" so the parameter should behave sensibly.
>
> ======
> .../test_decoding/sql/decoding_into_rel.sql
>
> NITPICK - wrong comments.
>
> ======
> doc/src/sgml/protocol.sgml
>
> 5.
> +    <varlistentry>
> +     <term>include_generated_columns</term>
> +      <listitem>
> +       <para>
> +        Boolean option to enable generated columns. This option controls
> +        whether generated columns should be included in the string
> +        representation of tuples during logical decoding in PostgreSQL.
> +        The default is false.
> +       </para>
> +      </listitem>
> +    </varlistentry>
> +
>
> Does the protocol version need to be bumped to support this new option
> and should that be mentioned on this page similar to how all other
> version values are mentioned?

I already did the Backward Compatibility test earlier and decided that
protocol bump is not needed.

> doc/src/sgml/ref/create_subscription.sgml
>
> NITPICK - some missing words/sentence.
> NITPICK - some missing <literal> tags.
> NITPICK - remove duplicated sentence.
> NITPICK - add another <para>.
>
> ======
> src/backend/commands/subscriptioncmds.c
>
> 6.
>  #define SUBOPT_ORIGIN 0x00008000
> +#define SUBOPT_include_generated_columns 0x00010000
>
> Please use UPPERCASE for consistency with other macros.
>
> ======
> .../libpqwalreceiver/libpqwalreceiver.c
>
> 7.
> + if (options->proto.logical.include_generated_columns &&
> + PQserverVersion(conn->streamConn) >= 170000)
> + appendStringInfoString(&cmd, ", include_generated_columns 'on'");
> +
>
> IMO it makes more sense to say 'true' here instead of 'on'. It seems
> like this was just cut/paste from the above code (where 'on' was
> sensible).
>
> ======
> src/include/catalog/pg_subscription.h
>
> 8.
> @@ -98,6 +98,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId)
> BKI_SHARED_RELATION BKI_ROW
>   * slots) in the upstream database are enabled
>   * to be synchronized to the standbys. */
>
> + bool subincludegencol; /* True if generated columns must be published */
> +
>
> Not fixed as claimed. This field name ought to be plural.
>
> /subincludegencol/subincludegencols/
>
> ~~~
>
> 9.
>   char    *origin; /* Only publish data originating from the
>   * specified origin */
> + bool includegencol; /* publish generated column data */
>  } Subscription;
>
> Not fixed as claimed. This field name ought to be plural.
>
> /includegencol/includegencols/
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 10.
> +$node_publisher->safe_psql('postgres',
> + "CREATE TABLE tab2 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> * 2) STORED)"
> +);
> +
> +$node_publisher->safe_psql('postgres',
> + "CREATE TABLE tab3 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> + 10) STORED)"
> +);
> +
>  $node_subscriber->safe_psql('postgres',
>   "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> * 22) STORED, c int)"
>  );
>
> +$node_subscriber->safe_psql('postgres',
> + "CREATE TABLE tab2 (a int PRIMARY KEY, b int)"
> +);
> +
> +$node_subscriber->safe_psql('postgres',
> + "CREATE TABLE tab3 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> + 20) STORED)"
> +);
>
> IMO the test needs lots more comments to describe what it is doing:
>
> For example, the setup deliberately has made:
> * publisher-side tab2 has generated col 'b' but subscriber-side tab2
> has NON-gnerated col 'b'.
> * publisher-side tab3 has generated col 'b' but subscriber-side tab2
> has DIFFERENT COMPUTATION generated col 'b'.
>
> So it will be better to have comments to explain all this instead of
> having to figure it out.
>
> ~~~
>
> 11.
>  # data for initial sync
>
>  $node_publisher->safe_psql('postgres',
>   "INSERT INTO tab1 (a) VALUES (1), (2), (3)");
> +$node_publisher->safe_psql('postgres',
> + "INSERT INTO tab2 (a) VALUES (1), (2), (3)");
>
>  $node_publisher->safe_psql('postgres',
> - "CREATE PUBLICATION pub1 FOR ALL TABLES");
> + "CREATE PUBLICATION pub1 FOR TABLE tab1");
> +$node_publisher->safe_psql('postgres',
> + "CREATE PUBLICATION pub2 FOR TABLE tab2");
> +$node_publisher->safe_psql('postgres',
> + "CREATE PUBLICATION pub3 FOR TABLE tab3");
> +
>
> # Wait for initial sync of all subscriptions
> $node_subscriber->wait_for_subscription_sync;
>
> my $result = $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab1");
> is( $result, qq(1|22
> 2|44
> 3|66), 'generated columns initial sync');
>
> ~
>
> IMO (and for completeness) it would be better to INSERT data for all
> the tables and alsot to validate that tables tab2 and tab3 has zero
> rows replicated. Yes, I know there is 'copy_data=false', but it is
> just easier to see all the tables instead of guessing why some are
> omitted, and anyway this test case will be needed after the next patch
> implements the COPY support for gen-cols.
>
> ~~~
>
> 12.
> +$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (4), (5)");
> +
> +$node_publisher->wait_for_catchup('sub2');
> +
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
> +is( $result, qq(4|8
> +5|10), 'generated columns replicated to non-generated column on subscriber');
> +
> +$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
> +
> +$node_publisher->wait_for_catchup('sub3');
> +
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
> +is( $result, qq(4|24
> +5|25), 'generated columns replicated to non-generated column on subscriber');
> +
>
> Here also I think there should be explicit comments about what these
> cases are testing, what results you are expecting, and why. The
> comments will look something like the message parameter of those
> safe_psql(...)
>
> e.g.
> # confirm generated columns ARE replicated when the subscriber-side
> column is not generated
>
> e.g.
> # confirm generated columns are NOT replicated when the
> subscriber-side column is also generated
>
> ======
>
> 99.
> Please also see my nitpicks attachment patch for various other
> cosmetic and docs problems, and apply theseif you agree:
> - documentation wording/rendering
> - wrong comments
> - spacing
> - etc.

All the comments are handled.

Patch v8-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2BAi0CgtXiAga82bWpWB8fVcOWycNyJ_jqXm788v3R8rQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  42. Re: Pgoutput not capturing the generated columns

    Peter Eisentraut <peter@eisentraut.org> — 2024-06-19T16:13:14Z

    On 19.06.24 13:22, Shubham Khanna wrote:
> All the comments are handled.
> 
> The attached Patch contains all the suggested changes.

Please also take a look at the proposed patch for virtual generated 
columns [0] and consider how that would affect your patch.  I think your 
feature can only replicate *stored* generated columns.  So perhaps the 
documentation and terminology in your patch should reflect that.


[0]: 
https://www.postgresql.org/message-id/flat/a368248e-69e4-40be-9c07-6c3b5880b0a6@eisentraut.org

  43. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-20T03:33:14Z

    Hi, here are my review comments for v8-0001.

======
Commit message.

1.
It seems like the patch name was accidentally omitted, so it became a
mess when it defaulted to the 1st paragraph of the commit message.

======
contrib/test_decoding/test_decoding.c

2.
+ data->include_generated_columns = true;

I previously posted a comment [1, #4] that this should default to
false; IMO it is unintuitive for the test_decoding to have an
*opposite* default behaviour compared to CREATE SUBSCRIPTION.

======
doc/src/sgml/ref/create_subscription.sgml

NITPICK - remove the inconsistent blank line in SGML

======
src/backend/commands/subscriptioncmds.c

3.
+#define SUBOPT_include_generated_columns 0x00010000

I previously posted a comment [1, #6] that this should be UPPERCASE,
but it is not yet fixed.

======
src/bin/psql/describe.c

NITPICK - move and reword the bogus comment

~

4.
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subincludegencols AS \"%s\"\n",
+ gettext_noop("include_generated_columns"));

4a.
For consistency with every other parameter, that column title should
be written in words "Include generated columns" (not
"include_generated_columns").

~

4b.
IMO this new column belongs with the other subscription parameter
columns (e.g. put it ahead of the "Conninfo" column).

======
src/test/subscription/t/011_generated.pl

NITPICK - fixed a comment

5.
IMO, it would be better for readability if all the matching CREATE
TABLE for publisher and subscriber are kept together, instead of the
current code which is creating all publisher tables and then creating
all subscriber tables.

~~~

6.
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
+is( $result, qq(4|8
+5|10), 'confirm generated columns ARE replicated when the
subscriber-side column is not generated');
+
...
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
+is( $result, qq(4|24
+5|25), 'confirm generated columns are NOT replicated when the
subscriber-side column is also generated');
+

6a.
These SELECT all need ORDER BY to protect against the SELECT *
returning rows in some unexpected order.

~

6b.
IMO there should be more comments here to explain how you can tell the
column was NOT replicated. E.g. it is because the result value of 'b'
is the subscriber-side computed value (which you made deliberately
different to the publisher-side computed value).

======

99.
Please also refer to the attached nitpicks top-up patch for minor
cosmetic stuff.

======
[1] https://www.postgresql.org/message-id/CAHv8RjLeZtTeXpFdoY6xCPO41HtuOPMSSZgshVdb%2BV%3Dp2YHL8Q%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  44. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-06-20T07:21:57Z

    On Wed, 19 Jun 2024 at 21:43, Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 19.06.24 13:22, Shubham Khanna wrote:
> > All the comments are handled.
> >
> > The attached Patch contains all the suggested changes.
>
> Please also take a look at the proposed patch for virtual generated
> columns [0] and consider how that would affect your patch.  I think your
> feature can only replicate *stored* generated columns.  So perhaps the
> documentation and terminology in your patch should reflect that.

This patch is unable to manage virtual generated columns because it
stores NULL values for them. Along with documentation the initial sync
command being generated also should be changed to sync data
exclusively for stored generated columns, omitting virtual ones. I
suggest treating these changes as a separate patch(0003) for future
merging or a separate commit, depending on the order of patch
acceptance.

Regards,
Vignesh

  45. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-20T13:00:19Z

    On Tue, 18 Jun 2024 at 10:57, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for patch v7-0002
>
> ======
> Commit Message
>
> NITPICKS
> - rearrange paragraphs
> - typo "donot"
> - don't start a sentence with "And"
> - etc.
>
> Please see the attachment for my suggested commit message text updates
> and take from it whatever you agree with.
>
Fixed

> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 1.
> +          If the subscriber-side column is also a generated column
> then this option
> +          has no effect; the replicated data will be ignored and the subscriber
> +          column will be filled as normal with the subscriber-side computed or
> +          default data. And during table synchronization, the data
> corresponding to
> +          the generated column on subscriber-side will not be sent from the
> +          publisher to the subscriber.
>
> This text already mentions subscriber-side generated cols. IMO you
> don't need to say anything at all about table synchronization --
> that's just an internal code optimization, which is not something the
> user needs to know about. IOW, the entire last sentence ("And
> during...") should be removed.
>
Fixed

> ======
> src/backend/replication/logical/relation.c
>
> 2. logicalrep_rel_open
>
> - if (attr->attisdropped)
> + if (attr->attisdropped ||
> + (!MySubscription->includegencol && attr->attgenerated))
>   {
>   entry->attrmap->attnums[i] = -1;
>   continue;
>
> ~
>
> Maybe I'm mistaken, but isn't this code for skipping checking for
> "missing" subscriber-side (aka local) columns? Can't it just
> unconditionally skip every attr->attgenerated -- i.e. why does it
> matter if the MySubscription->includegencol was set or not?
>
In case 'include_generated_columns' is 'true'. column list in
remoterel will have an entry for generated columns.
So, in this case if we skip every attr->attgenerated, we will get a
missing column error.

> ======
> src/backend/replication/logical/tablesync.c
>
> 3. make_copy_attnamelist
>
> - for (i = 0; i < rel->remoterel.natts; i++)
> + desc = RelationGetDescr(rel->localrel);
> +
> + for (i = 0; i < desc->natts; i++)
>   {
> - attnamelist = lappend(attnamelist,
> -   makeString(rel->remoterel.attnames[i]));
> + int attnum;
> + Form_pg_attribute attr = TupleDescAttr(desc, i);
> +
> + if (!attr->attgenerated)
> + continue;
> +
> + attnum = logicalrep_rel_att_by_name(&rel->remoterel,
> + NameStr(attr->attname));
> +
> + /*
> + * Check if subscription table have a generated column with same
> + * column name as a non-generated column in the corresponding
> + * publication table.
> + */
> + if (attnum >=0 && !attgenlist[attnum])
> + ereport(ERROR,
> + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("logical replication target relation \"%s.%s\" is missing
> replicated column: \"%s\"",
> + rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
> +
> + if (attnum >= 0)
> + gencollist = lappend_int(gencollist, attnum);
>   }
>
> ~
>
> NITPICK - Use C99-style for loop variables
> NITPICK - Typo in comment
> NITPICK - spaces
>
> ~
>
> 3a.
> I think above code should be refactored so there is only one check for
> "if (attnum >= 0)" -- e.g. other condition should be nested.
>
> ~
>
> 3b.
> That ERROR message says "missing replicated column", but that doesn't
> seem much like what the code-comment was saying this code is about.
>
Fixed

> ~~~
>
> 4.
> + for (i = 0; i < rel->remoterel.natts; i++)
> + {
> +
> + if (gencollist != NIL && j < gencollist->length &&
> + list_nth_int(gencollist, j) == i)
> + j++;
> + else
> + attnamelist = lappend(attnamelist,
> +   makeString(rel->remoterel.attnames[i]));
> + }
>
> NITPICK - Use C99-style for loop variables
> NITPICK - Unnecessary blank lines
>
> ~
>
> IIUC the subscriber-side table and the publisher-side table do NOT
> have to have all the columns in identical order for the logical
> replication to work correcly. AFAIK it works fine so long as the
> column names match for the replicated columns. Therefore, I am
> suspicious that this new patch code seems to be imposing some new
> ordering assumptions/restrictions (e.g. list_nth_int stuff) which are
> not current requirements.
>
> ~~~
>
> copy_table:
>
> NITPICK - comment typo
> NITPICK - comment wording
>
Fixed

> ~
>
> 5.
> + int i = 0;
> + ListCell *l;
> +
>   appendStringInfoString(&cmd, "COPY (SELECT ");
> - for (int i = 0; i < lrel.natts; i++)
> + foreach(l, attnamelist)
>   {
> - appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
> - if (i < lrel.natts - 1)
> + appendStringInfoString(&cmd, quote_identifier(strVal(lfirst(l))));
> + if (i < attnamelist->length - 1)
>   appendStringInfoString(&cmd, ", ");
> + i++;
>   }
> IIUC for new code like this, it is preferred to use the foreach*
> macros instead of ListCell.
>
Fixed

> ======
> src/test/regress/sql/subscription.sql
>
> 6.
> --- fail - copy_data and include_generated_columns are mutually
> exclusive options
> -CREATE SUBSCRIPTION sub2 CONNECTION 'dbname=regress_doesnotexist'
> PUBLICATION testpub WITH (include_generated_columns = true);
> -ERROR:  copy_data = true and include_generated_columns = true are
> mutually exclusive options
>
> It is OK to delete this test now but IMO still needs to be some
> "include_generated_columns must be boolean" test cases (e.g. same as
> there was two_phase). Actually, this should probably be done by the
> 0001 patch.
>
Fixed

> ======
> src/test/subscription/t/011_generated.pl
>
> 7.
> All the PRIMARY KEY stuff may be overkill. Are primary keys really
> needed for these tests?
>
Fixed

> ~~~
>
> 8.
> +$node_publisher->safe_psql('postgres',
> + "CREATE TABLE tab4 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> * 2) STORED, c int GENERATED ALWAYS AS (a * 2) STORED)"
> +);
> +
> +$node_publisher->safe_psql('postgres',
> + "CREATE TABLE tab5 (a int PRIMARY KEY, b int)"
> +);
> +
>
> Maybe add comments on what is special about all these tables, so don't
> have to read the tests later to deduce their purpose.
>
> tab4: publisher-side generated col 'b' and 'c'  ==> subscriber-side
> non-generated col 'b', and generated-col 'c'
> tab5: publisher-side non-generated col 'b' --> subscriber-side
> non-generated col 'b'
>
Fixed

> ~~~
>
> 9.
> +$node_subscriber->safe_psql('postgres',
> + "CREATE SUBSCRIPTION sub4 CONNECTION '$publisher_connstr'
> PUBLICATION pub4 WITH (include_generated_columns = true)"
> + );
> +
>
> All the publications are created together, and all the subscriptions
> are created together except for 'sub5'. Consider including a comment
> to say why you deliberately created the 'sub5' subscription separate
> from all others.
>
Fixed

> ======
>
> 99.
> Please also see my code nitpicks attachment patch for various other
> cosmetic problems, and apply them if you agree.
>
Applied the changes

I have fixed the comments and attached the patches. I have also
attached the v9-0003 patch. It will resolve the issue suggested by
Vignesh in [1]. I have also updated the documentation for the same.
v9-0001 - Not Modified
v9-0002 - Support replication of generated columns during initial sync.
v9-0003 - Fix behaviour of tablesync for Virtual Generated Columns.

[1]: https://www.postgresql.org/message-id/CALDaNm3Ufg872XqgPvBVzXHvUVenu-8%2BGz2dyEuKq3CN0UxfKw%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  46. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-20T13:03:15Z

    On Thu, 20 Jun 2024 at 12:52, vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 19 Jun 2024 at 21:43, Peter Eisentraut <peter@eisentraut.org> wrote:
> >
> > On 19.06.24 13:22, Shubham Khanna wrote:
> > > All the comments are handled.
> > >
> > > The attached Patch contains all the suggested changes.
> >
> > Please also take a look at the proposed patch for virtual generated
> > columns [0] and consider how that would affect your patch.  I think your
> > feature can only replicate *stored* generated columns.  So perhaps the
> > documentation and terminology in your patch should reflect that.
>
> This patch is unable to manage virtual generated columns because it
> stores NULL values for them. Along with documentation the initial sync
> command being generated also should be changed to sync data
> exclusively for stored generated columns, omitting virtual ones. I
> suggest treating these changes as a separate patch(0003) for future
> merging or a separate commit, depending on the order of patch
> acceptance.
>

I have addressed the issue and updated the documentation accordingly.
And created a new 0003 patch.
Please refer to v9-0003 patch for the same in [1].

[1]: https://www.postgresql.org/message-id/CANhcyEXmjLEPNgOSAtjS4YGb9JvS8w-SO9S%2BjRzzzXo2RavNWw%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  47. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-21T02:52:57Z

    Hi Shubham, here are some more patch v8-0001 comments that I missed yesterday.

======
src/test/subscription/t/011_generated.pl

1.
Are the PRIMARY KEY qualifiers needed for the new tab2, tab3 tables? I
don't think so.

~~~

2.
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
+is( $result, qq(4|8
+5|10), 'confirm generated columns ARE replicated when the
subscriber-side column is not generated');
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
+
+$node_publisher->wait_for_catchup('sub3');
+
+$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
+is( $result, qq(4|24
+5|25), 'confirm generated columns are NOT replicated when the
subscriber-side column is also generated');
+

It would be prudent to do explicit "SELECT a,b" instead of "SELECT *",
for readability and to avoid any surprises.

======
Kind Regards,
Peter Smith.
Fujitsu Australia.

  48. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-21T03:33:16Z

    Hi, here are some review comments for patch v9-0002.

======
src/backend/replication/logical/relation.c

1. logicalrep_rel_open

- if (attr->attisdropped)
+ if (attr->attisdropped ||
+ (!MySubscription->includegencols && attr->attgenerated))

You replied to my question from the previous review [1, #2] as follows:
In case 'include_generated_columns' is 'true'. column list in
remoterel will have an entry for generated columns. So, in this case
if we skip every attr->attgenerated, we will get a missing column
error.

~

TBH, the reason seems very subtle to me. Perhaps that
"(!MySubscription->includegencols && attr->attgenerated))" condition
should be coded as a separate "if", so then you can include a comment
similar to your answer, to explain it.

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:

NITPICK - punctuation in function comment
NITPICK - add/reword some more comments
NITPICK - rearrange comments to be closer to the code they are commenting

~

2. make_copy_attnamelist.

+ /*
+ * Construct column list for COPY.
+ */
+ for (int i = 0; i < rel->remoterel.natts; i++)
+ {
+ if(!gencollist[i])
+ attnamelist = lappend(attnamelist,
+   makeString(rel->remoterel.attnames[i]));
+ }

IIUC isn't this assuming that the attribute number (aka column order)
is the same on the subscriber side (e.g. gencollist idx) and on the
publisher side (e.g. remoterel.attnames[i]).  AFAIK logical
replication does not require this ordering must be match like that,
therefore I am suspicious this new logic is accidentally imposing new
unwanted assumptions/restrictions. I had asked the same question
before [1-#4] about this code, but there was no reply.

Ideally, there would be more test cases for when the columns
(including the generated ones) are all in different orders on the
pub/sub tables.

~~~

3. General - varnames.

It would help with understanding if the 'attgenlist' variables in all
these functions are re-named to make it very clear that this is
referring to the *remote* (publisher-side) table genlist, not the
subscriber table genlist.

~~~

4.
+ int i = 0;
+
  appendStringInfoString(&cmd, "COPY (SELECT ");
- for (int i = 0; i < lrel.natts; i++)
+ foreach_ptr(ListCell, l, attnamelist)
  {
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, quote_identifier(strVal(l)));
+ if (i < attnamelist->length - 1)
  appendStringInfoString(&cmd, ", ");
+ i++;
  }

4a.
I think the purpose of the new macros is to avoid using ListCell, and
also 'l' is an unhelpful variable name. Shouldn't this code be more
like:
foreach_node(String, att_name, attnamelist)

~

4b.
The code can be far simpler if you just put the comma (", ") always
up-front except the *first* iteration, so you can avoid checking the
list length every time. For example:

if (i++)
  appendStringInfoString(&cmd, ", ");

======
src/test/subscription/t/011_generated.pl

5. General.

Hmm. This patch 0002 included many formatting changes to tables tab2
and tab3 and subscriptions sub2 and sub3 but they do not belong here.
The proper formatting for those needs to be done back in patch 0001
where they were introduced. Patch 0002 should just concentrate only on
the new stuff for patch 0002.

~

6. CREATE TABLES would be better in pairs

IMO it will be better if the matching CREATE TABLE for pub and sub are
kept together, instead of separating them by doing all pub then all
sub. I previously made the same comment for patch 0001, so maybe it
will be addressed next time...

~

7. SELECT *

+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab4 ORDER BY a");

It will be prudent to do explicit "SELECT a,b,c" instead of "SELECT
*", for readability and so there are no surprises.

======

99.
Please also refer to my attached nitpicks diff for numerous cosmetic
changes, and apply if you agree.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPtAsEc3PEB1KUk1kFF5tcCrDCCTcbboougO29vP1B4E2Q%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  49. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-21T05:17:08Z

    Hi Shubham/Shlok.

FYI,  my patch describing the current PG17 behaviour of logical
replication of generated columns was recently pushed [1].

Note that this will have some impact on your patch set. e.g. You are
changing the current replication behaviour, so the "Generated Columns"
section note will now need to be modified by your patches.

======
[1] https://github.com/postgres/postgres/commit/7a089f6e6a14ca3a5dc8822c393c6620279968b9
[2]

Kind Regards,
Peter Smith.
Fujitsu Australia

  50. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-21T07:20:33Z

    Hi, Here are some review comments for patch v9-0003

======
Commit Message

/fix/fixes/

======
1.
General. Is tablesync enough?

I don't understand why is the patch only concerned about tablesync?
Does it make sense to prevent VIRTUAL column replication during
tablesync, if you aren't also going to prevent VIRTUAL columns from
normal logical replication (e.g. when copy_data = false)? Or is this
already handled somewhere?

~~~

2.
General. Missing test.

Add some test cases to verify behaviour is different for STORED versus
VIRTUAL generated columns

======
src/sgml/ref/create_subscription.sgml

NITPICK - consider rearranging as shown in my nitpicks diff
NITPICK - use <literal> sgml markup for STORED

======
src/backend/replication/logical/tablesync.c

3.
- if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
- walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
- !MySubscription->includegencols)
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) < 170000)
+ {
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
  appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }
+ else if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000)
+ {
+ if(MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ else
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }

IMO this logic is too tricky to remain uncommented -- please add some comments.
Also, it seems somewhat complex. I think you can achieve the same more simply:

SUGGESTION

if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
{
  bool gencols_allowed = walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000
    && MySubscription->includegencols;
  if (gencols_allowed)
  {
    /* Replication of generated cols is supported, but not VIRTUAL cols. */
    appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
  }
  else
  {
    /* Replication of generated cols is not supported. */
    appendStringInfo(&cmd, " AND a.attgenerated = ''");
  }
}

======

99.
Please refer also to my attached nitpick diffs and apply those if you agree.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  51. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-21T07:38:22Z

    Hi Shubham/Shlok.

FYI, there is some other documentation page that mentions generated
column replication messages.

This page [1] says:
"Next, the following message part appears for each column included in
the publication (except generated columns):"

But, with the introduction of your new feature, I think that the
"except generated columns" wording is not strictly correct anymore.
IOW that docs page needs updating but AFAICT your patches have not
addressed this yet.

======
[1] https://www.postgresql.org/docs/17/protocol-logicalrep-message-formats.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  52. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-22T18:06:00Z

    On Thu, Jun 20, 2024 at 9:03 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for v8-0001.
>
> ======
> Commit message.
>
> 1.
> It seems like the patch name was accidentally omitted, so it became a
> mess when it defaulted to the 1st paragraph of the commit message.
>
> ======
> contrib/test_decoding/test_decoding.c
>
> 2.
> + data->include_generated_columns = true;
>
> I previously posted a comment [1, #4] that this should default to
> false; IMO it is unintuitive for the test_decoding to have an
> *opposite* default behaviour compared to CREATE SUBSCRIPTION.
>
> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> NITPICK - remove the inconsistent blank line in SGML
>
> ======
> src/backend/commands/subscriptioncmds.c
>
> 3.
> +#define SUBOPT_include_generated_columns 0x00010000
>
> I previously posted a comment [1, #6] that this should be UPPERCASE,
> but it is not yet fixed.
>
> ======
> src/bin/psql/describe.c
>
> NITPICK - move and reword the bogus comment
>
> ~
>
> 4.
> + if (pset.sversion >= 170000)
> + appendPQExpBuffer(&buf,
> + ", subincludegencols AS \"%s\"\n",
> + gettext_noop("include_generated_columns"));
>
> 4a.
> For consistency with every other parameter, that column title should
> be written in words "Include generated columns" (not
> "include_generated_columns").
>
> ~
>
> 4b.
> IMO this new column belongs with the other subscription parameter
> columns (e.g. put it ahead of the "Conninfo" column).
>
> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - fixed a comment
>
> 5.
> IMO, it would be better for readability if all the matching CREATE
> TABLE for publisher and subscriber are kept together, instead of the
> current code which is creating all publisher tables and then creating
> all subscriber tables.
>
> ~~~
>
> 6.
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
> +is( $result, qq(4|8
> +5|10), 'confirm generated columns ARE replicated when the
> subscriber-side column is not generated');
> +
> ...
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
> +is( $result, qq(4|24
> +5|25), 'confirm generated columns are NOT replicated when the
> subscriber-side column is also generated');
> +
>
> 6a.
> These SELECT all need ORDER BY to protect against the SELECT *
> returning rows in some unexpected order.
>
> ~
>
> 6b.
> IMO there should be more comments here to explain how you can tell the
> column was NOT replicated. E.g. it is because the result value of 'b'
> is the subscriber-side computed value (which you made deliberately
> different to the publisher-side computed value).
>
> ======
>
> 99.
> Please also refer to the attached nitpicks top-up patch for minor
> cosmetic stuff.

All the comments are handled.

The attached Patch contains all the suggested changes.

Thanks and Regards,
Shubham Khanna.

  53. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-22T18:08:57Z

    On Fri, Jun 21, 2024 at 8:23 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham, here are some more patch v8-0001 comments that I missed yesterday.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> 1.
> Are the PRIMARY KEY qualifiers needed for the new tab2, tab3 tables? I
> don't think so.
>
> ~~~
>
> 2.
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2");
> +is( $result, qq(4|8
> +5|10), 'confirm generated columns ARE replicated when the
> subscriber-side column is not generated');
> +
> +$node_publisher->safe_psql('postgres', "INSERT INTO tab3 VALUES (4), (5)");
> +
> +$node_publisher->wait_for_catchup('sub3');
> +
> +$result = $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3");
> +is( $result, qq(4|24
> +5|25), 'confirm generated columns are NOT replicated when the
> subscriber-side column is also generated');
> +
>
> It would be prudent to do explicit "SELECT a,b" instead of "SELECT *",
> for readability and to avoid any surprises.

Both the comments are handled.

Patch v9-0001 contains all the changes required. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2B6kwOGmn5MsRaTmciJDxtvNsyszMoPXV62OGPGzkxrCg%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  54. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-06-23T04:58:15Z

    Hi Shubham,

Thanks for sharing new patch! You shared as v9, but it should be v10, right?
Also, since there are no commitfest entry, I registered [1]. You can rename the
title based on the needs. Currently CFbot said OK.

Anyway, below are my comments.

01. General
Your patch contains unnecessary changes. Please remove all of them. E.g., 

```
 						 " s.subpublications,\n");
-
```
And
```
 		appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
-							 " s.subenabled,\n");
+							" s.subenabled,\n");
```

02. General
Again, please run the pgindent/pgperltidy.

03. test_decoding
Previously I suggested to the default value of to be include_generated_columns
should be true, so you modified like that. However, Peter suggested opposite
opinion [3] and you just revised accordingly. I think either way might be okay, but
at least you must clarify the reason why you preferred to set default to false and
changed accordingly.

04. decoding_into_rel.sql
According to the comment atop this file, this test should insert result to a table.
But added case does not - we should put them at another place. I.e., create another
file.

05. decoding_into_rel.sql
```
+-- when 'include-generated-columns' is not set
```
Can you clarify the expected behavior as a comment?

06. getSubscriptions
```
+	else
+		appendPQExpBufferStr(query,
+						" false AS subincludegencols,\n");
```
I think the comma is not needed.
Also, this error meant that you did not test to use pg_dump for instances prior PG16.
Please verify whether we can dump subscriptions and restore them accordingly.

[1]: https://commitfest.postgresql.org/48/5068/
[2]: https://www.postgresql.org/message-id/OSBPR01MB25529997E012DEABA8E15A02F5E52%40OSBPR01MB2552.jpnprd01.prod.outlook.com
[3]: https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.gmail.com

Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/

  55. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-24T02:51:26Z

    Hi, here are some patch v9-0001 comments.

I saw Kuroda-san has already posted comments for this patch so there
may be some duplication here.

======
GENERAL

1.
The later patches 0002 etc are checking to support only STORED
gencols. But, doesn't that restriction belong in this patch 0001 so
VIRTUAL columns are not decoded etc in the first place... (??)

~~~

2.
The "Generated Columns" docs mentioned in my previous review comment
[2] should be modified by this 0001 patch.

~~~

3.
I think the "Message Format" page mentioned in my previous review
comment [3] should be modified by this 0001 patch.

======
Commit message

4.
The patch name is still broken as previously mentioned [1, #1]

======
doc/src/sgml/protocol.sgml

5.
Should this docs be referring to STORED generated columns, instead of
just generated columns?

======
doc/src/sgml/ref/create_subscription.sgml

6.
Should this be docs referring to STORED generated columns, instead of
just generated columns?

======
src/bin/pg_dump/pg_dump.c

getSubscriptions:
NITPICK - tabs
NITPICK - patch removed a blank line it should not be touching
NITPICK = patch altered indents it should not be touching
NITPICK - a missing blank line that was previously present

7.
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols,\n");

There is an unwanted comma here.

~

dumpSubscription
NITPICK - patch altered indents it should not be touching

======
src/bin/pg_dump/pg_dump.h

NITPICK - unnecessary blank line

======
src/bin/psql/describe.c

describeSubscriptions
NITPICK - bad indentation

8.
In my previous review [1, #4b] I suggested this new column should be
in a different order (e.g. adjacent to the other ones ahead of
'Conninfo'), but this is not yet addressed.

======
src/test/subscription/t/011_generated.pl

NITPICK - missing space in comment
NITPICK - misleading "because" wording in the comment

======

99.
See also my attached nitpicks diff, for cosmetic issues. Please apply
whatever you agree with.

======
[1] My v8-0001 review -
https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPvsRWq9t2tEErt5ZWZCVpNFVZjfZ_owqfdjOhh4yXb_3Q%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAHut%2BPsHsT3V1wQ5uoH9ynbmWn4ZQqOe34X%2Bg37LSi7sgE_i2g%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia.

  56. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-24T12:56:03Z

    On Fri, 21 Jun 2024 at 09:03, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some review comments for patch v9-0002.
>
> ======
> src/backend/replication/logical/relation.c
>
> 1. logicalrep_rel_open
>
> - if (attr->attisdropped)
> + if (attr->attisdropped ||
> + (!MySubscription->includegencols && attr->attgenerated))
>
> You replied to my question from the previous review [1, #2] as follows:
> In case 'include_generated_columns' is 'true'. column list in
> remoterel will have an entry for generated columns. So, in this case
> if we skip every attr->attgenerated, we will get a missing column
> error.
>
> ~
>
> TBH, the reason seems very subtle to me. Perhaps that
> "(!MySubscription->includegencols && attr->attgenerated))" condition
> should be coded as a separate "if", so then you can include a comment
> similar to your answer, to explain it.
Fixed

> ======
> src/backend/replication/logical/tablesync.c
>
> make_copy_attnamelist:
>
> NITPICK - punctuation in function comment
> NITPICK - add/reword some more comments
> NITPICK - rearrange comments to be closer to the code they are commenting
Applied the changes

> ~
>
> 2. make_copy_attnamelist.
>
> + /*
> + * Construct column list for COPY.
> + */
> + for (int i = 0; i < rel->remoterel.natts; i++)
> + {
> + if(!gencollist[i])
> + attnamelist = lappend(attnamelist,
> +   makeString(rel->remoterel.attnames[i]));
> + }
>
> IIUC isn't this assuming that the attribute number (aka column order)
> is the same on the subscriber side (e.g. gencollist idx) and on the
> publisher side (e.g. remoterel.attnames[i]).  AFAIK logical
> replication does not require this ordering must be match like that,
> therefore I am suspicious this new logic is accidentally imposing new
> unwanted assumptions/restrictions. I had asked the same question
> before [1-#4] about this code, but there was no reply.
>
> Ideally, there would be more test cases for when the columns
> (including the generated ones) are all in different orders on the
> pub/sub tables.
'gencollist' is set according to the remoterel
+           gencollist[attnum] = true;
where attnum is the attribute number of the corresponding column on remote rel.

I have also added the tests to confirm the behaviour

> ~~~
>
> 3. General - varnames.
>
> It would help with understanding if the 'attgenlist' variables in all
> these functions are re-named to make it very clear that this is
> referring to the *remote* (publisher-side) table genlist, not the
> subscriber table genlist.
Fixed

> ~~~
>
> 4.
> + int i = 0;
> +
>   appendStringInfoString(&cmd, "COPY (SELECT ");
> - for (int i = 0; i < lrel.natts; i++)
> + foreach_ptr(ListCell, l, attnamelist)
>   {
> - appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
> - if (i < lrel.natts - 1)
> + appendStringInfoString(&cmd, quote_identifier(strVal(l)));
> + if (i < attnamelist->length - 1)
>   appendStringInfoString(&cmd, ", ");
> + i++;
>   }
>
> 4a.
> I think the purpose of the new macros is to avoid using ListCell, and
> also 'l' is an unhelpful variable name. Shouldn't this code be more
> like:
> foreach_node(String, att_name, attnamelist)
>
> ~
>
> 4b.
> The code can be far simpler if you just put the comma (", ") always
> up-front except the *first* iteration, so you can avoid checking the
> list length every time. For example:
>
> if (i++)
>   appendStringInfoString(&cmd, ", ");
Fixed

> ======
> src/test/subscription/t/011_generated.pl
>
> 5. General.
>
> Hmm. This patch 0002 included many formatting changes to tables tab2
> and tab3 and subscriptions sub2 and sub3 but they do not belong here.
> The proper formatting for those needs to be done back in patch 0001
> where they were introduced. Patch 0002 should just concentrate only on
> the new stuff for patch 0002.
Fixed

> ~
>
> 6. CREATE TABLES would be better in pairs
>
> IMO it will be better if the matching CREATE TABLE for pub and sub are
> kept together, instead of separating them by doing all pub then all
> sub. I previously made the same comment for patch 0001, so maybe it
> will be addressed next time...
Fixed

> ~
>
> 7. SELECT *
>
> +$result =
> +  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab4 ORDER BY a");
>
> It will be prudent to do explicit "SELECT a,b,c" instead of "SELECT
> *", for readability and so there are no surprises.
Fixed

> ======
>
> 99.
> Please also refer to my attached nitpicks diff for numerous cosmetic
> changes, and apply if you agree.
Applied the changes.

> ======
> [1] https://www.postgresql.org/message-id/CAHut%2BPtAsEc3PEB1KUk1kFF5tcCrDCCTcbboougO29vP1B4E2Q%40mail.gmail.com

I have attached a v10 patch to address the comments:
v10-0001 - Not Modified
v10-0002 - Support replication of generated columns during initial sync.
v10-0003 - Fix behaviour for Virtual Generated Columns.

Thanks and Regards,
Shlok Kyal

  57. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-06-24T12:57:33Z

    On Fri, 21 Jun 2024 at 12:51, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, Here are some review comments for patch v9-0003
>
> ======
> Commit Message
>
> /fix/fixes/
Fixed

> ======
> 1.
> General. Is tablesync enough?
>
> I don't understand why is the patch only concerned about tablesync?
> Does it make sense to prevent VIRTUAL column replication during
> tablesync, if you aren't also going to prevent VIRTUAL columns from
> normal logical replication (e.g. when copy_data = false)? Or is this
> already handled somewhere?
I checked the behaviour during incremental changes. I saw during
decoding 'null' values are present for Virtual Generated Columns. I
made the relevant changes to not support replication of Virtual
generated columns.

> ~~~
>
> 2.
> General. Missing test.
>
> Add some test cases to verify behaviour is different for STORED versus
> VIRTUAL generated columns
I have not added the tests as it would give an error in cfbot.
I have added a TODO note for the same. This can be done once the
VIRTUAL generated columns patch is committted.

> ======
> src/sgml/ref/create_subscription.sgml
>
> NITPICK - consider rearranging as shown in my nitpicks diff
> NITPICK - use <literal> sgml markup for STORED
Fixed

> ======
> src/backend/replication/logical/tablesync.c
>
> 3.
> - if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
> - walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
> - !MySubscription->includegencols)
> + if (walrcv_server_version(LogRepWorkerWalRcvConn) < 170000)
> + {
> + if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
>   appendStringInfo(&cmd, " AND a.attgenerated = ''");
> + }
> + else if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000)
> + {
> + if(MySubscription->includegencols)
> + appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
> + else
> + appendStringInfo(&cmd, " AND a.attgenerated = ''");
> + }
>
> IMO this logic is too tricky to remain uncommented -- please add some comments.
> Also, it seems somewhat complex. I think you can achieve the same more simply:
>
> SUGGESTION
>
> if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000)
> {
>   bool gencols_allowed = walrcv_server_version(LogRepWorkerWalRcvConn) >= 170000
>     && MySubscription->includegencols;
>   if (gencols_allowed)
>   {
>     /* Replication of generated cols is supported, but not VIRTUAL cols. */
>     appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
>   }
>   else
>   {
>     /* Replication of generated cols is not supported. */
>     appendStringInfo(&cmd, " AND a.attgenerated = ''");
>   }
> }
Fixed

> ======
>
> 99.
> Please refer also to my attached nitpick diffs and apply those if you agree.
Applied the changes.

I have attached the updated patch v10 here in [1].
[1]: https://www.postgresql.org/message-id/CANhcyEUMCk6cCbw0vVZWo8FRd6ae9CmKG%3DgKP-9Q67jLn8HqtQ%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  58. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-25T06:25:50Z

    Here are some review comments for the patch v10-0002.

======
Commit Message

1.
Note that we don't copy columns when the subscriber-side column is also
generated. Those will be filled as normal with the subscriber-side computed or
default data.

~

Now this patch also introduced some errors etc, so I think that patch
comment should be written differently to explicitly spell out
behaviour of every combination, something like the below:

Summary

when (include_generated_column = true)

* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).

* publisher not-generated column => subscriber generated column: This
will give ERROR.

* publisher generated column => subscriber not-generated column: The
publisher generated column value is copied.

* publisher generated column => subscriber generated column: The
publisher generated column value is not copied. The subscriber
generated column will be filled with the subscriber-side computed or
default data.

when (include_generated_columns = false)

* publisher not-generated column => subscriber not-generated column:
This is just normal logical replication (not changed by this patch).

* publisher not-generated column => subscriber generated column: This
will give ERROR.

* publisher generated column => subscriber not-generated column: This
will replicate nothing. Publisher generate-column is not replicated.
The subscriber column will be filled with the subscriber-side default
data.

* publisher generated column => subscriber generated column:  This
will replicate nothing. Publisher generate-column is not replicated.
The subscriber generated column will be filled with the
subscriber-side computed or default data.

======
src/backend/replication/logical/relation.c

2.
logicalrep_rel_open:

I tested some of the "missing column" logic, and got the following results:

Scenario A:
PUB
test_pub=# create table t2(a int, b int);
test_pub=# create publication pub2 for table t2;
SUB
test_sub=# create table t2(a int, b int generated always as (a*2) stored);
test_sub=# create subscription sub2 connection 'dbname=test_pub'
publication pub2 with (include_generated_columns = false);
Result:
ERROR:  logical replication target relation "public.t2" is missing
replicated column: "b"

~

Scenario B:
PUB/SUB identical to above, but subscription sub2 created "with
(include_generated_columns = true);"
Result:
ERROR:  logical replication target relation "public.t2" has a
generated column "b" but corresponding column on source relation is
not a generated column

~~~

2a. Question

Why should we get 2 different error messages for what is essentially
the same problem according to whether the 'include_generated_columns'
is false or true? Isn't the 2nd error message the more correct and
useful one for scenarios like this involving generated columns?

Thoughts?

~

2b. Missing tests?

I also noticed there seems no TAP test for the current "missing
replicated column" message. IMO there should be a new test introduced
for this because the loop involved too much bms logic to go
untested...

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:
NITPICK - minor comment tweak
NITPICK - add some spaces after "if" code

3.
Should you pfree the gencollist at the bottom of this function when
you no longer need it, for tidiness?

~~~

4.
 static void
-fetch_remote_table_info(char *nspname, char *relname,
+fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
  LogicalRepRelation *lrel, List **qual)
 {
  WalRcvExecResult *res;
  StringInfoData cmd;
  TupleTableSlot *slot;
  Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
  Oid qualRow[] = {TEXTOID};
  bool isnull;
+ bool    *remotegenlist_res;

IMO the names 'remotegenlist' and 'remotegenlist_res' should be
swapped the other way around, because it is the function parameter
that is the "result", whereas the 'remotegenlist_res' is just the
local working var for it.

~~~

5. fetch_remote_table_info

Now walrcv_server_version(LogRepWorkerWalRcvConn) is used in multiple
places, I think it will be better to assign this to a 'server_version'
variable to be used everywhere instead of having multiple function
calls.

~~~

6.
  "SELECT a.attnum,"
  "       a.attname,"
  "       a.atttypid,"
- "       a.attnum = ANY(i.indkey)"
+ "       a.attnum = ANY(i.indkey),"
+ " a.attgenerated != ''"
  "  FROM pg_catalog.pg_attribute a"
  "  LEFT JOIN pg_catalog.pg_index i"
  "       ON (i.indexrelid = pg_get_replica_identity_index(%u))"
  " WHERE a.attnum > 0::pg_catalog.int2"
- "   AND NOT a.attisdropped %s"
+ "   AND NOT a.attisdropped", lrel->remoteid);
+
+ if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
+ walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+

If the server version is < PG12 then AFAIK there was no such thing as
"a.attgenerated", so shouldn't that SELECT " a.attgenerated != ''"
part also be guarded by some version checking condition like in the
WHERE? Otherwise won't it cause an ERROR for old servers?

~~~

7.
  /*
- * For non-tables and tables with row filters, we need to do COPY
- * (SELECT ...), but we can't just do SELECT * because we need to not
- * copy generated columns. For tables with any row filters, build a
- * SELECT query with OR'ed row filters for COPY.
+ * For non-tables and tables with row filters and when
+ * 'include_generated_columns' is specified as 'true', we need to do
+ * COPY (SELECT ...), as normal COPY of generated column is not
+ * supported. For tables with any row filters, build a SELECT query
+ * with OR'ed row filters for COPY.
  */

NITPICK. I felt this was not quite right. AFAIK the reasons for using
this COPY (SELECT ...) syntax is different for row-filters and
generated-columns. Anyway, I updated the comment slightly in my
nitpicks attachment. Please have a look at it to see if you agree with
the suggestions. Maybe I am wrong.

~~~

8.
- for (int i = 0; i < lrel.natts; i++)
+ foreach_ptr(String, att_name, attnamelist)

I'm not 100% sure, but isn't foreach_node the macro to use here,
rather than foreach_ptr?
======
src/test/subscription/t/011_generated.pl

9.
Please discuss with Shubham how to make all the tab1, tab2, tab3,
tab4, tab5, tab6 comments use the same kind of style/wording.
Currently, the patches 0001 and 0002 test comments are a bit
inconsistent.

~~~

10.
Related to above -- now that patch 0002 supports copy_data=true I
don't see why we need to test generated columns *both* for
copy_data=false and also for copy_data=true. IOW, is it really
necessary to have so many tables/tests? For example, I am thinking
some of those tests from patch 0001 can be re-used or just removed now
that copy_data=true works.

~~~

NITPICK - minor comment tweak

~~~

11.
For tab4 and tab6 I saw the initial sync and normal replication data
tests are all merged together, but I had expected to see the initial
sync and normal replication data tests separated so it would be
consistent with the earlier tab1, tab2, tab3 tests.

======

99.
Also, I have attached a nitpicks diff for some of the cosmetic review
comments mentioned above. Please apply whatever of these that you
agree with.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  59. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-06-25T13:19:21Z

    Dear Shlok,

Thanks for updating patches! Below are my comments, maybe only for 0002.

01. General

IIUC, we are not discussed why ALTER SUBSCRIPTION ... SET include_generated_columns
is prohibit. Previously, it seems okay because there are exclusive options. But now,
such restrictions are gone. Do you have a reason in your mind? It is just not considered
yet?

02. General

According to the doc, we allow to alter a column to non-generated one, by ALTER
TABLE ... ALTER COLUMN ... DROP EXPRESSION command. Not sure, what should be
when the command is executed on the subscriber while copying the data? Should
we continue the copy or restart? How do you think?

03. Tes tcode

IIUC, REFRESH PUBLICATION can also lead the table synchronization. Can you add
a test for that?

04. Test code (maybe for 0001)

Please test the combination with TABLE ... ALTER COLUMN ... DROP EXPRESSION command.

05. logicalrep_rel_open

```
+            /*
+             * In case 'include_generated_columns' is 'false', we should skip the
+             * check of missing attrs for generated columns.
+             * In case 'include_generated_columns' is 'true', we should check if
+             * corresponding column for the generated column in publication column
+             * list is present in the subscription table.
+             */
+            if (!MySubscription->includegencols && attr->attgenerated)
+            {
+                entry->attrmap->attnums[i] = -1;
+                continue;
+            }
```

This comment is not very clear to me, because here we do not skip anything.
Can you clarify the reason why attnums[i] is set to -1 and how will it be used?

06. make_copy_attnamelist

```
+    gencollist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
```

I think this array is too large. Can we reduce a size to (desc->natts * sizeof(bool))?
Also, the free'ing should be done.

07. make_copy_attnamelist

```
+    /* Loop to handle subscription table generated columns. */
+    for (int i = 0; i < desc->natts; i++)
```

IIUC, the loop is needed to find generated columns on the subscriber side, right?
Can you clarify as comment?

08. copy_table

```
+    /*
+     * Regular table with no row filter and 'include_generated_columns'
+     * specified as 'false' during creation of subscription.
+     */
```

I think this comment is not correct. After patching, all tablesync command becomes
like COPY (SELECT ...) if include_genereted_columns is set to true. Is it right?
Can we restrict only when the table has generated ones?

Best Regards,
Hayato Kuroda
FUJITSU LIMITED
https://www.fujitsu.com/

  60. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-26T02:36:07Z

    Hi Shlok. Here are my review comments for patch v10-0003

======
General.

1.
The patch has lots of conditions like:
if (att->attgenerated && (att->attgenerated !=
ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
 continue;

IMO these are hard to read. Although more verbose, please consider if
all those (for the sake of readability) would be better re-written
like below :

if (att->generated)
{
  if (!include_generated_columns)
    continue;

  if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
    continue;
}

======
contrib/test_decoding/test_decoding.c

tuple_to_stringinfo:

NITPICK = refactored the code and comments a bit here to make it easier
NITPICK - there is no need to mention "virtual". Instead, say we only
support STORED

======
src/backend/catalog/pg_publication.c

publication_translate_columns:

NITPICK - introduced variable 'att' to simplify this code

~

2.
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot use virtual generated column \"%s\" in publication
column list",
+    colname));

Is it better to avoid referring to "virtual" at all? Instead, consider
rearranging the wording to say something like "generated column \"%s\"
is not STORED so cannot be used in a publication column list"

~~~

pg_get_publication_tables:

NITPICK - split the condition code for readability

======
src/backend/replication/logical/relation.c

3. logicalrep_rel_open

+ if (attr->attgenerated && attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+

Isn't this missing some code to say "entry->attrmap->attnums[i] =
-1;", same as all the other nearby code is doing?

~~~

4.
I felt all the "generated column" logic should be kept together, so
this new condition should be combined with the other generated column
condition, like:

if (attr->attgenerated)
{
  /* comment... */
  if (!MySubscription->includegencols)
  {
    entry->attrmap->attnums[i] = -1;
    continue;
  }

  /* comment... */
  if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
  {
    entry->attrmap->attnums[i] = -1;
    continue;
  }
}

======
src/backend/replication/logical/tablesync.c

5.
+ if (gencols_allowed)
+ {
+ /* Replication of generated cols is supported, but not VIRTUAL cols. */
+ appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
+ }

Is it better here to use the ATTRIBUTE_GENERATED_VIRTUAL macro instead
of the hardwired 'v'? (Maybe add another TODO comment to revisit
this).

Alternatively, consider if it is more future-proof to rearrange so it
just says what *is* supported instead of what *isn't* supported:
e.g. "AND a.attgenerated IN ('', 's')"

======
src/test/subscription/t/011_generated.pl

NITPICK - some comments are missing the word "stored"
NITPICK - sometimes "col" should be plural "cols"
NITPICK = typo "GNERATED"

======

6.
In a previous review [1, comment #3] I mentioned that there should be
some docs updates on the "Logical Replication Message Formats" section
53.9. So, I expected patch 0001 would make some changes and then patch
0003 would have to update it again to say something about "STORED".
But all that is missing from the v10* patches.

======

99.
See also my nitpicks diff which is a top-up patch addressing all the
nitpick comments mentioned above. Please apply all of these that you
agree with.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPvQ8CLq-JysTTeRj4u5SC9vTVcx3AgwTHcPUEOh-UnKcQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, Jun 24, 2024 at 10:56 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> On Fri, 21 Jun 2024 at 09:03, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi, here are some review comments for patch v9-0002.
> >
> > ======
> > src/backend/replication/logical/relation.c
> >
> > 1. logicalrep_rel_open
> >
> > - if (attr->attisdropped)
> > + if (attr->attisdropped ||
> > + (!MySubscription->includegencols && attr->attgenerated))
> >
> > You replied to my question from the previous review [1, #2] as follows:
> > In case 'include_generated_columns' is 'true'. column list in
> > remoterel will have an entry for generated columns. So, in this case
> > if we skip every attr->attgenerated, we will get a missing column
> > error.
> >
> > ~
> >
> > TBH, the reason seems very subtle to me. Perhaps that
> > "(!MySubscription->includegencols && attr->attgenerated))" condition
> > should be coded as a separate "if", so then you can include a comment
> > similar to your answer, to explain it.
> Fixed
>
> > ======
> > src/backend/replication/logical/tablesync.c
> >
> > make_copy_attnamelist:
> >
> > NITPICK - punctuation in function comment
> > NITPICK - add/reword some more comments
> > NITPICK - rearrange comments to be closer to the code they are commenting
> Applied the changes
>
> > ~
> >
> > 2. make_copy_attnamelist.
> >
> > + /*
> > + * Construct column list for COPY.
> > + */
> > + for (int i = 0; i < rel->remoterel.natts; i++)
> > + {
> > + if(!gencollist[i])
> > + attnamelist = lappend(attnamelist,
> > +   makeString(rel->remoterel.attnames[i]));
> > + }
> >
> > IIUC isn't this assuming that the attribute number (aka column order)
> > is the same on the subscriber side (e.g. gencollist idx) and on the
> > publisher side (e.g. remoterel.attnames[i]).  AFAIK logical
> > replication does not require this ordering must be match like that,
> > therefore I am suspicious this new logic is accidentally imposing new
> > unwanted assumptions/restrictions. I had asked the same question
> > before [1-#4] about this code, but there was no reply.
> >
> > Ideally, there would be more test cases for when the columns
> > (including the generated ones) are all in different orders on the
> > pub/sub tables.
> 'gencollist' is set according to the remoterel
> +           gencollist[attnum] = true;
> where attnum is the attribute number of the corresponding column on remote rel.
>
> I have also added the tests to confirm the behaviour
>
> > ~~~
> >
> > 3. General - varnames.
> >
> > It would help with understanding if the 'attgenlist' variables in all
> > these functions are re-named to make it very clear that this is
> > referring to the *remote* (publisher-side) table genlist, not the
> > subscriber table genlist.
> Fixed
>
> > ~~~
> >
> > 4.
> > + int i = 0;
> > +
> >   appendStringInfoString(&cmd, "COPY (SELECT ");
> > - for (int i = 0; i < lrel.natts; i++)
> > + foreach_ptr(ListCell, l, attnamelist)
> >   {
> > - appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
> > - if (i < lrel.natts - 1)
> > + appendStringInfoString(&cmd, quote_identifier(strVal(l)));
> > + if (i < attnamelist->length - 1)
> >   appendStringInfoString(&cmd, ", ");
> > + i++;
> >   }
> >
> > 4a.
> > I think the purpose of the new macros is to avoid using ListCell, and
> > also 'l' is an unhelpful variable name. Shouldn't this code be more
> > like:
> > foreach_node(String, att_name, attnamelist)
> >
> > ~
> >
> > 4b.
> > The code can be far simpler if you just put the comma (", ") always
> > up-front except the *first* iteration, so you can avoid checking the
> > list length every time. For example:
> >
> > if (i++)
> >   appendStringInfoString(&cmd, ", ");
> Fixed
>
> > ======
> > src/test/subscription/t/011_generated.pl
> >
> > 5. General.
> >
> > Hmm. This patch 0002 included many formatting changes to tables tab2
> > and tab3 and subscriptions sub2 and sub3 but they do not belong here.
> > The proper formatting for those needs to be done back in patch 0001
> > where they were introduced. Patch 0002 should just concentrate only on
> > the new stuff for patch 0002.
> Fixed
>
> > ~
> >
> > 6. CREATE TABLES would be better in pairs
> >
> > IMO it will be better if the matching CREATE TABLE for pub and sub are
> > kept together, instead of separating them by doing all pub then all
> > sub. I previously made the same comment for patch 0001, so maybe it
> > will be addressed next time...
> Fixed
>
> > ~
> >
> > 7. SELECT *
> >
> > +$result =
> > +  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab4 ORDER BY a");
> >
> > It will be prudent to do explicit "SELECT a,b,c" instead of "SELECT
> > *", for readability and so there are no surprises.
> Fixed
>
> > ======
> >
> > 99.
> > Please also refer to my attached nitpicks diff for numerous cosmetic
> > changes, and apply if you agree.
> Applied the changes.
>
> > ======
> > [1] https://www.postgresql.org/message-id/CAHut%2BPtAsEc3PEB1KUk1kFF5tcCrDCCTcbboougO29vP1B4E2Q%40mail.gmail.com
>
> I have attached a v10 patch to address the comments:
> v10-0001 - Not Modified
> v10-0002 - Support replication of generated columns during initial sync.
> v10-0003 - Fix behaviour for Virtual Generated Columns.
>
> Thanks and Regards,
> Shlok Kyal

  61. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-26T12:12:24Z

    On Sun, Jun 23, 2024 at 10:28 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Hi Shubham,
>
> Thanks for sharing new patch! You shared as v9, but it should be v10, right?
> Also, since there are no commitfest entry, I registered [1]. You can rename the
> title based on the needs. Currently CFbot said OK.
>
> Anyway, below are my comments.
>
> 01. General
> Your patch contains unnecessary changes. Please remove all of them. E.g.,
>
> ```
>                                                  " s.subpublications,\n");
> -
> ```
> And
> ```
>                 appendPQExpBufferStr(query, " o.remote_lsn AS suboriginremotelsn,\n"
> -                                                        " s.subenabled,\n");
> +                                                       " s.subenabled,\n");
> ```
>
> 02. General
> Again, please run the pgindent/pgperltidy.
>
> 03. test_decoding
> Previously I suggested to the default value of to be include_generated_columns
> should be true, so you modified like that. However, Peter suggested opposite
> opinion [3] and you just revised accordingly. I think either way might be okay, but
> at least you must clarify the reason why you preferred to set default to false and
> changed accordingly.

I have set the default value as true in case of test_decoding. The
reason for this is even before the new feature implementation,
generated columns were getting selected.

> 04. decoding_into_rel.sql
> According to the comment atop this file, this test should insert result to a table.
> But added case does not - we should put them at another place. I.e., create another
> file.
>
> 05. decoding_into_rel.sql
> ```
> +-- when 'include-generated-columns' is not set
> ```
> Can you clarify the expected behavior as a comment?
>
> 06. getSubscriptions
> ```
> +       else
> +               appendPQExpBufferStr(query,
> +                                               " false AS subincludegencols,\n");
> ```
> I think the comma is not needed.
> Also, this error meant that you did not test to use pg_dump for instances prior PG16.
> Please verify whether we can dump subscriptions and restore them accordingly.
>
> [1]: https://commitfest.postgresql.org/48/5068/
> [2]: https://www.postgresql.org/message-id/OSBPR01MB25529997E012DEABA8E15A02F5E52%40OSBPR01MB2552.jpnprd01.prod.outlook.com
> [3]: https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.gmail.com

All the comments are handled.

The attached Patches contains all the suggested changes.

Thanks and Regards,
Shubham Khanna.

  62. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-26T12:16:33Z

    On Mon, Jun 24, 2024 at 8:21 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some patch v9-0001 comments.
>
> I saw Kuroda-san has already posted comments for this patch so there
> may be some duplication here.
>
> ======
> GENERAL
>
> 1.
> The later patches 0002 etc are checking to support only STORED
> gencols. But, doesn't that restriction belong in this patch 0001 so
> VIRTUAL columns are not decoded etc in the first place... (??)
>
> ~~~
>
> 2.
> The "Generated Columns" docs mentioned in my previous review comment
> [2] should be modified by this 0001 patch.
>
> ~~~
>
> 3.
> I think the "Message Format" page mentioned in my previous review
> comment [3] should be modified by this 0001 patch.
>
> ======
> Commit message
>
> 4.
> The patch name is still broken as previously mentioned [1, #1]
>
> ======
> doc/src/sgml/protocol.sgml
>
> 5.
> Should this docs be referring to STORED generated columns, instead of
> just generated columns?
>
> ======
> doc/src/sgml/ref/create_subscription.sgml
>
> 6.
> Should this be docs referring to STORED generated columns, instead of
> just generated columns?
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> getSubscriptions:
> NITPICK - tabs
> NITPICK - patch removed a blank line it should not be touching
> NITPICK = patch altered indents it should not be touching
> NITPICK - a missing blank line that was previously present
>
> 7.
> + else
> + appendPQExpBufferStr(query,
> + " false AS subincludegencols,\n");
>
> There is an unwanted comma here.
>
> ~
>
> dumpSubscription
> NITPICK - patch altered indents it should not be touching
>
> ======
> src/bin/pg_dump/pg_dump.h
>
> NITPICK - unnecessary blank line
>
> ======
> src/bin/psql/describe.c
>
> describeSubscriptions
> NITPICK - bad indentation
>
> 8.
> In my previous review [1, #4b] I suggested this new column should be
> in a different order (e.g. adjacent to the other ones ahead of
> 'Conninfo'), but this is not yet addressed.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - missing space in comment
> NITPICK - misleading "because" wording in the comment
>
> ======
>
> 99.
> See also my attached nitpicks diff, for cosmetic issues. Please apply
> whatever you agree with.
>
> ======
> [1] My v8-0001 review -
> https://www.postgresql.org/message-id/CAHut%2BPujrRQ63ju8P41tBkdjkQb4X9uEdLK_Wkauxum1MVUdfA%40mail.gmail.com
> [2] https://www.postgresql.org/message-id/CAHut%2BPvsRWq9t2tEErt5ZWZCVpNFVZjfZ_owqfdjOhh4yXb_3Q%40mail.gmail.com
> [3] https://www.postgresql.org/message-id/CAHut%2BPsHsT3V1wQ5uoH9ynbmWn4ZQqOe34X%2Bg37LSi7sgE_i2g%40mail.gmail.com

All the comments are handled.

I have attached the updated patch v11 here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJpS_XDkR6OrsmMZtCBZNxeYoCdENhC0%3Dbe0rLmNvhiQw%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  63. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-06-27T05:29:01Z

    >All the comments are handled.
>
> The attached Patches contain all the suggested changes.

v11-0003 patch was not getting applied, so here are the updated
patches for the same.

Thanks and Regards,
Shubham Khanna.

  64. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-06-27T09:10:59Z

    Hi, here are some patch v11-0001 comments.

(BTW, I had difficulty reviewing this because something seemed strange
with the changes this patch made to the test_decoding tests).

======
General

1. Patch name

Patch name does not need to quote 'logical replication'

~

2. test_decoding tests

Multiple test_decoding tests were failing for me. There is something
very suspicious about the unexplained changes the patch made to the
expected "binary.out" and "decoding_into_rel.out" etc. I REVERTED all
those changes in my nitpicks top-up to get everything working. Please
re-confirm that all the test_decoding tests are OK!

======
Commit Message

3.
Since you are including the example usage for test_decoding, I think
it's better to include the example usage of CREATE SUBSCRIPTION also.

======
contrib/test_decoding/expected/binary.out

4.
 SELECT 'init' FROM
pg_create_logical_replication_slot('regression_slot',
'test_decoding');
- ?column?
-----------
- init
-(1 row)
-
+ERROR:  replication slot "regression_slot" already exists

Huh? Why is this unrelated expected output changed by this patch?

The test_decoding test fails for me unless I REVERT this change!! See
my nitpicks diff.

======
.../expected/decoding_into_rel.out

5.
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
- ?column?
-----------
- stop
-(1 row)
-

Huh? Why is this unrelated expected output changed by this patch?

The test_decoding test fails for me unless I REVERT this change!! See
my nitpicks diff.

======
.../test_decoding/sql/decoding_into_rel.sql

6.
-SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');

Huh, Why does this patch change this code at all? I REVERTED this
change. See my nitpicks diff.

======
.../test_decoding/sql/generated_columns.sql

(see my nitpicks replacement file for this test)

7.
+-- test that we can insert the result of a 'include_generated_columns'
+-- into the tables created. That's really not a good idea in practical terms,
+-- but provides a nice test.

NITPICK - I didn't understand the point of this comment.  I updated
the comment according to my understanding.

~

NITPICK - The comment "when 'include-generated-columns' is not set
then columns will not be replicated" is the opposite of what the
result is. I changed this comment.

NITPICK - modified and unified wording of some of the other comments

NITPICK - changed some blank lines

======
contrib/test_decoding/test_decoding.c

8.
+ else if (strcmp(elem->defname, "include-generated-columns") == 0)
+ {
+ if (elem->arg == NULL)
+ data->include_generated_columns = true;

Is there any way to test that "elem->arg == NULL" in the
generated.sql? OTOH, if it is not possible to get here then is the
code even needed?

======
doc/src/sgml/ddl.sgml

9.
      <para>
-      Generated columns are skipped for logical replication and cannot be
-      specified in a <command>CREATE PUBLICATION</command> column list.
+      'include_generated_columns' option controls whether generated columns
+      should be included in the string representation of tuples during
+      logical decoding in PostgreSQL. The default is <literal>true</literal>.
      </para>

NITPICK - Use proper markdown instead of single quotes for the parameter.

NITPICK - I think this can be reworded slightly to provide a
cross-reference to the CREATE SUBSCRIPTION parameter for more details
(which means then we can avoid repeating details like the default
value here). PSA my nitpicks diff for an example of how I thought docs
should look.

======
doc/src/sgml/protocol.sgml

10.
+        The default is true.

No, it isn't. AFAIK you made the default behaviour true only for
'test_decoding', but the default for CREATE SUBSCRIPTION remains
*false* because that is the existing PG17 behaviour. And the default
for the 'include_generated_columns' in the protocol is *also* false to
match the CREATE SUBSCRIPTION default.

e.g. libpqwalreceiver.c only sets ", include_generated_columns 'true'"
when options->proto.logical.include_generated_columns
e.g. worker.c says: options->proto.logical.include_generated_columns =
MySubscription->includegencols;
e.g. subscriptioncmds.c sets default: opts->include_generated_columns = false;

(This confirmed my previous review expectation that using different
default behaviours for test_decoding and pgoutput would surely lead to
confusion)

~~~

11.
-     <para>
-      Next, the following message part appears for each column included in
-      the publication (except generated columns):
-     </para>
-

AFAIK you cannot just remove this entire paragraph because I thought
it was still relevant to talking about "... the following message
part". But, if you don't want to explain and cross-reference about
'include_generated_columns' then maybe it is OK just to remove the
"(except generated columns)" part?

======
src/test/subscription/t/011_generated.pl

NITPICK - comment typo /tab2/tab3/
NITPICK - remove some blank lines

~~~

12.
# the column was NOT replicated (the result value of 'b' is the
subscriber-side computed value)

NITPICK - I think this comment is wrong for the tab2 test because here
col 'b' IS replicated. I have added much more substantial test case
comments in the attached nitpicks diff. PSA.

======
src/test/subscription/t/031_column_list.pl

13.
NITPICK - IMO there is a missing word "list" in the comment. This bug
existed already on HEAD but since this patch is modifying this comment
I think we can also fix this in passing.

======
Kind Regards,
Peter Smith.
Fujitsu Australia.

  65. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-01T10:38:03Z

    On Thu, Jun 27, 2024 at 2:41 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some patch v11-0001 comments.
>
> (BTW, I had difficulty reviewing this because something seemed strange
> with the changes this patch made to the test_decoding tests).
>
> ======
> General
>
> 1. Patch name
>
> Patch name does not need to quote 'logical replication'
>
> ~
>
> 2. test_decoding tests
>
> Multiple test_decoding tests were failing for me. There is something
> very suspicious about the unexplained changes the patch made to the
> expected "binary.out" and "decoding_into_rel.out" etc. I REVERTED all
> those changes in my nitpicks top-up to get everything working. Please
> re-confirm that all the test_decoding tests are OK!
>
> ======
> Commit Message
>
> 3.
> Since you are including the example usage for test_decoding, I think
> it's better to include the example usage of CREATE SUBSCRIPTION also.
>
> ======
> contrib/test_decoding/expected/binary.out
>
> 4.
>  SELECT 'init' FROM
> pg_create_logical_replication_slot('regression_slot',
> 'test_decoding');
> - ?column?
> -----------
> - init
> -(1 row)
> -
> +ERROR:  replication slot "regression_slot" already exists
>
> Huh? Why is this unrelated expected output changed by this patch?
>
> The test_decoding test fails for me unless I REVERT this change!! See
> my nitpicks diff.
>
> ======
> .../expected/decoding_into_rel.out
>
> 5.
> -SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
> - ?column?
> -----------
> - stop
> -(1 row)
> -
>
> Huh? Why is this unrelated expected output changed by this patch?
>
> The test_decoding test fails for me unless I REVERT this change!! See
> my nitpicks diff.
>
> ======
> .../test_decoding/sql/decoding_into_rel.sql
>
> 6.
> -SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
> +SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
>
> Huh, Why does this patch change this code at all? I REVERTED this
> change. See my nitpicks diff.
>
> ======
> .../test_decoding/sql/generated_columns.sql
>
> (see my nitpicks replacement file for this test)
>
> 7.
> +-- test that we can insert the result of a 'include_generated_columns'
> +-- into the tables created. That's really not a good idea in practical terms,
> +-- but provides a nice test.
>
> NITPICK - I didn't understand the point of this comment.  I updated
> the comment according to my understanding.
>
> ~
>
> NITPICK - The comment "when 'include-generated-columns' is not set
> then columns will not be replicated" is the opposite of what the
> result is. I changed this comment.
>
> NITPICK - modified and unified wording of some of the other comments
>
> NITPICK - changed some blank lines
>
> ======
> contrib/test_decoding/test_decoding.c
>
> 8.
> + else if (strcmp(elem->defname, "include-generated-columns") == 0)
> + {
> + if (elem->arg == NULL)
> + data->include_generated_columns = true;
>
> Is there any way to test that "elem->arg == NULL" in the
> generated.sql? OTOH, if it is not possible to get here then is the
> code even needed?
>

Currently I could not find a case where the
'include_generated_columns' option is not specifying any value, but  I
was hesitant to remove this from here as the other options mentioned
follow the same rules. Thoughts?

> ======
> doc/src/sgml/ddl.sgml
>
> 9.
>       <para>
> -      Generated columns are skipped for logical replication and cannot be
> -      specified in a <command>CREATE PUBLICATION</command> column list.
> +      'include_generated_columns' option controls whether generated columns
> +      should be included in the string representation of tuples during
> +      logical decoding in PostgreSQL. The default is <literal>true</literal>.
>       </para>
>
> NITPICK - Use proper markdown instead of single quotes for the parameter.
>
> NITPICK - I think this can be reworded slightly to provide a
> cross-reference to the CREATE SUBSCRIPTION parameter for more details
> (which means then we can avoid repeating details like the default
> value here). PSA my nitpicks diff for an example of how I thought docs
> should look.
>
> ======
> doc/src/sgml/protocol.sgml
>
> 10.
> +        The default is true.
>
> No, it isn't. AFAIK you made the default behaviour true only for
> 'test_decoding', but the default for CREATE SUBSCRIPTION remains
> *false* because that is the existing PG17 behaviour. And the default
> for the 'include_generated_columns' in the protocol is *also* false to
> match the CREATE SUBSCRIPTION default.
>
> e.g. libpqwalreceiver.c only sets ", include_generated_columns 'true'"
> when options->proto.logical.include_generated_columns
> e.g. worker.c says: options->proto.logical.include_generated_columns =
> MySubscription->includegencols;
> e.g. subscriptioncmds.c sets default: opts->include_generated_columns = false;
>
> (This confirmed my previous review expectation that using different
> default behaviours for test_decoding and pgoutput would surely lead to
> confusion)
>
> ~~~
>
> 11.
> -     <para>
> -      Next, the following message part appears for each column included in
> -      the publication (except generated columns):
> -     </para>
> -
>
> AFAIK you cannot just remove this entire paragraph because I thought
> it was still relevant to talking about "... the following message
> part". But, if you don't want to explain and cross-reference about
> 'include_generated_columns' then maybe it is OK just to remove the
> "(except generated columns)" part?
>
> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - comment typo /tab2/tab3/
> NITPICK - remove some blank lines
>
> ~~~
>
> 12.
> # the column was NOT replicated (the result value of 'b' is the
> subscriber-side computed value)
>
> NITPICK - I think this comment is wrong for the tab2 test because here
> col 'b' IS replicated. I have added much more substantial test case
> comments in the attached nitpicks diff. PSA.
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 13.
> NITPICK - IMO there is a missing word "list" in the comment. This bug
> existed already on HEAD but since this patch is modifying this comment
> I think we can also fix this in passing.

All the comments are handled.

The attached Patches contain all the suggested changes.

Thanks and Regards,
Shubham Khanna.

  66. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-02T04:23:06Z

    On Mon, Jul 1, 2024 at 8:38 PM Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
>...
> > 8.
> > + else if (strcmp(elem->defname, "include-generated-columns") == 0)
> > + {
> > + if (elem->arg == NULL)
> > + data->include_generated_columns = true;
> >
> > Is there any way to test that "elem->arg == NULL" in the
> > generated.sql? OTOH, if it is not possible to get here then is the
> > code even needed?
> >
>
> Currently I could not find a case where the
> 'include_generated_columns' option is not specifying any value, but  I
> was hesitant to remove this from here as the other options mentioned
> follow the same rules. Thoughts?
>

If you do manage to find a scenario for this then I think a test for
it would be good. But, I agree that the code seems OK because now I
see it is the same pattern as similar nearby code.

~~~

Thanks for the updated patch. Here are some review comments for patch v13-0001.

======
.../expected/generated_columns.out

nitpicks (see generated_columns.sql)

======
.../test_decoding/sql/generated_columns.sql

nitpick - use plural /column/columns/
nitpick - use consistent wording in the comments
nitpick - IMO it is better to INSERT different values for each of the tests

======
doc/src/sgml/protocol.sgml

nitpick - I noticed that none of the other boolean parameters on this
page mention about a default, so maybe here we should do the same and
omit that information.

~~~

1.
-     <para>
-      Next, the following message part appears for each column included in
-      the publication (except generated columns):
-     </para>
-

In a previous review [1 comment #11] I wrote that you can't just
remove this paragraph because AFAIK it is still meaningful. A minimal
change might be to just remove the "(except generated columns)" part.
Alternatively, you could give a more detailed explanation mentioning
the include_generated_columns protocol parameter.

I provided some updated text for this paragraph in my NITPICKS top-up
patch, Please have a look at that for ideas.

======
src/backend/commands/subscriptioncmds.c

It looks like pg_indent needs to be run on this file.

======
src/include/catalog/pg_subscription.h

nitpick - comment /publish/Publish/ for consistency

======
src/include/replication/walreceiver.h

nitpick - comment /publish/Publish/ for consistency

======
src/test/regress/expected/subscription.out

nitpicks - (see subscription.sql)

======
src/test/regress/sql/subscription.sql

nitpick - combine the invalid option combinations test with all the
others (no special comment needed)
nitpick - rename subscription as 'regress_testsub2' same as all its peers.

======
src/test/subscription/t/011_generated.pl

nitpick - add/remove blank lines

======
src/test/subscription/t/031_column_list.pl

nitpick - rewording for a comment. This issue was not strictly caused
by this patch, but since you are modifying the same comment we can fix
this in passing.

======
99.
Please also see the attached top-up patch for all those nitpicks
identified above.

======
[1] v11-0001 review
https://www.postgresql.org/message-id/CAHut%2BPv45gB4cV%2BSSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  67. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-02T05:29:20Z

    Hi Shubham,

As you can see, most of my recent review comments for patch 0001 are
only cosmetic nitpicks. But, there is still one long-unanswered design
question from a month ago [1, #G.2]

A lot of the patch code of pgoutput.c and proto.c and logicalproto.h
is related to the introduction and passing everywhere of new
'include_generated_columns' function parameters. These same functions
are also always passing "BitMapSet *columns" representing the
publication column list.

My question was about whether we can't make use of the existing BMS
parameter instead of introducing all the new API parameters.

The idea might go something like this:

* If 'include_generated_columns' option is specified true and if no
column list was already specified then perhaps the relentry->columns
can be used for a "dummy" column list that has everything including
all the generated columns.

* By doing this:
 -- you may be able to avoid passing the extra
'include_gernated_columns' everywhere
 -- you may be able to avoid checking for generated columns deeper in
the code (since it is already checked up-front when building the
column list BMS)

~~

I'm not saying this design idea is guaranteed to work, but it might be
worth considering, because if it does work then there is potential to
make the current 0001 patch significantly shorter.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPsuJfcaeg6zst%3D6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  68. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-04T10:28:35Z

    On Tue, 25 Jun 2024 at 11:56, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for the patch v10-0002.
>
> ======
> Commit Message
>
> 1.
> Note that we don't copy columns when the subscriber-side column is also
> generated. Those will be filled as normal with the subscriber-side computed or
> default data.
>
> ~
>
> Now this patch also introduced some errors etc, so I think that patch
> comment should be written differently to explicitly spell out
> behaviour of every combination, something like the below:
>
> Summary
>
> when (include_generated_column = true)
>
> * publisher not-generated column => subscriber not-generated column:
> This is just normal logical replication (not changed by this patch).
>
> * publisher not-generated column => subscriber generated column: This
> will give ERROR.
>
> * publisher generated column => subscriber not-generated column: The
> publisher generated column value is copied.
>
> * publisher generated column => subscriber generated column: The
> publisher generated column value is not copied. The subscriber
> generated column will be filled with the subscriber-side computed or
> default data.
>
> when (include_generated_columns = false)
>
> * publisher not-generated column => subscriber not-generated column:
> This is just normal logical replication (not changed by this patch).
>
> * publisher not-generated column => subscriber generated column: This
> will give ERROR.
>
> * publisher generated column => subscriber not-generated column: This
> will replicate nothing. Publisher generate-column is not replicated.
> The subscriber column will be filled with the subscriber-side default
> data.
>
> * publisher generated column => subscriber generated column:  This
> will replicate nothing. Publisher generate-column is not replicated.
> The subscriber generated column will be filled with the
> subscriber-side computed or default data.
Modified

> ======
> src/backend/replication/logical/relation.c
>
> 2.
> logicalrep_rel_open:
>
> I tested some of the "missing column" logic, and got the following results:
>
> Scenario A:
> PUB
> test_pub=# create table t2(a int, b int);
> test_pub=# create publication pub2 for table t2;
> SUB
> test_sub=# create table t2(a int, b int generated always as (a*2) stored);
> test_sub=# create subscription sub2 connection 'dbname=test_pub'
> publication pub2 with (include_generated_columns = false);
> Result:
> ERROR:  logical replication target relation "public.t2" is missing
> replicated column: "b"
>
> ~
>
> Scenario B:
> PUB/SUB identical to above, but subscription sub2 created "with
> (include_generated_columns = true);"
> Result:
> ERROR:  logical replication target relation "public.t2" has a
> generated column "b" but corresponding column on source relation is
> not a generated column
>
> ~~~
>
> 2a. Question
>
> Why should we get 2 different error messages for what is essentially
> the same problem according to whether the 'include_generated_columns'
> is false or true? Isn't the 2nd error message the more correct and
> useful one for scenarios like this involving generated columns?
>
> Thoughts?
Did the modification to give same error in both cases

> ~
>
> 2b. Missing tests?
>
> I also noticed there seems no TAP test for the current "missing
> replicated column" message. IMO there should be a new test introduced
> for this because the loop involved too much bms logic to go
> untested...
Added the tests 004_sync.pl

> ======
> src/backend/replication/logical/tablesync.c
>
> make_copy_attnamelist:
> NITPICK - minor comment tweak
> NITPICK - add some spaces after "if" code
Applied the changes

> 3.
> Should you pfree the gencollist at the bottom of this function when
> you no longer need it, for tidiness?
Fixed

> ~~~
>
> 4.
>  static void
> -fetch_remote_table_info(char *nspname, char *relname,
> +fetch_remote_table_info(char *nspname, char *relname, bool **remotegenlist,
>   LogicalRepRelation *lrel, List **qual)
>  {
>   WalRcvExecResult *res;
>   StringInfoData cmd;
>   TupleTableSlot *slot;
>   Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
> - Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
> + Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID, BOOLOID};
>   Oid qualRow[] = {TEXTOID};
>   bool isnull;
> + bool    *remotegenlist_res;
>
> IMO the names 'remotegenlist' and 'remotegenlist_res' should be
> swapped the other way around, because it is the function parameter
> that is the "result", whereas the 'remotegenlist_res' is just the
> local working var for it.
Fixed

> ~~~
>
> 5. fetch_remote_table_info
>
> Now walrcv_server_version(LogRepWorkerWalRcvConn) is used in multiple
> places, I think it will be better to assign this to a 'server_version'
> variable to be used everywhere instead of having multiple function
> calls.
Fixed

> ~~~
>
> 6.
>   "SELECT a.attnum,"
>   "       a.attname,"
>   "       a.atttypid,"
> - "       a.attnum = ANY(i.indkey)"
> + "       a.attnum = ANY(i.indkey),"
> + " a.attgenerated != ''"
>   "  FROM pg_catalog.pg_attribute a"
>   "  LEFT JOIN pg_catalog.pg_index i"
>   "       ON (i.indexrelid = pg_get_replica_identity_index(%u))"
>   " WHERE a.attnum > 0::pg_catalog.int2"
> - "   AND NOT a.attisdropped %s"
> + "   AND NOT a.attisdropped", lrel->remoteid);
> +
> + if ((walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 &&
> + walrcv_server_version(LogRepWorkerWalRcvConn) <= 160000) ||
> + !MySubscription->includegencols)
> + appendStringInfo(&cmd, " AND a.attgenerated = ''");
> +
>
> If the server version is < PG12 then AFAIK there was no such thing as
> "a.attgenerated", so shouldn't that SELECT " a.attgenerated != ''"
> part also be guarded by some version checking condition like in the
> WHERE? Otherwise won't it cause an ERROR for old servers?
Fixed

> ~~~
>
> 7.
>   /*
> - * For non-tables and tables with row filters, we need to do COPY
> - * (SELECT ...), but we can't just do SELECT * because we need to not
> - * copy generated columns. For tables with any row filters, build a
> - * SELECT query with OR'ed row filters for COPY.
> + * For non-tables and tables with row filters and when
> + * 'include_generated_columns' is specified as 'true', we need to do
> + * COPY (SELECT ...), as normal COPY of generated column is not
> + * supported. For tables with any row filters, build a SELECT query
> + * with OR'ed row filters for COPY.
>   */
>
> NITPICK. I felt this was not quite right. AFAIK the reasons for using
> this COPY (SELECT ...) syntax is different for row-filters and
> generated-columns. Anyway, I updated the comment slightly in my
> nitpicks attachment. Please have a look at it to see if you agree with
> the suggestions. Maybe I am wrong.
Fixed

> ~~~
>
> 8.
> - for (int i = 0; i < lrel.natts; i++)
> + foreach_ptr(String, att_name, attnamelist)
>
> I'm not 100% sure, but isn't foreach_node the macro to use here,
> rather than foreach_ptr?
Fixed

> ======
> src/test/subscription/t/011_generated.pl
>
> 9.
> Please discuss with Shubham how to make all the tab1, tab2, tab3,
> tab4, tab5, tab6 comments use the same kind of style/wording.
> Currently, the patches 0001 and 0002 test comments are a bit
> inconsistent.
Fixed

> ~~~
>
> 10.
> Related to above -- now that patch 0002 supports copy_data=true I
> don't see why we need to test generated columns *both* for
> copy_data=false and also for copy_data=true. IOW, is it really
> necessary to have so many tables/tests? For example, I am thinking
> some of those tests from patch 0001 can be re-used or just removed now
> that copy_data=true works.
Fixed

> ~~~
>
> NITPICK - minor comment tweak
Fixed

> ~~~
>
> 11.
> For tab4 and tab6 I saw the initial sync and normal replication data
> tests are all merged together, but I had expected to see the initial
> sync and normal replication data tests separated so it would be
> consistent with the earlier tab1, tab2, tab3 tests.
Fixed

> ======
>
> 99.
> Also, I have attached a nitpicks diff for some of the cosmetic review
> comments mentioned above. Please apply whatever of these that you
> agree with.
Applied the relevant changes

I have attached a v14 to fix the comments.

Thanks and Regards,
Shlok Kyal

  69. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-04T10:31:31Z

    On Tue, 25 Jun 2024 at 18:49, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Shlok,
>
> Thanks for updating patches! Below are my comments, maybe only for 0002.
>
> 01. General
>
> IIUC, we are not discussed why ALTER SUBSCRIPTION ... SET include_generated_columns
> is prohibit. Previously, it seems okay because there are exclusive options. But now,
> such restrictions are gone. Do you have a reason in your mind? It is just not considered
> yet?
We donot support ALTER SUBSCRIPTION to alter
'include_generated_columns'. Suppose initially the user has a logical
replication setup. Publisher has
table t1 with columns (c1 int, c2 int generated always as (c1*2)) and
subscriber has table t1 with columns (c1 int, c2 int). And initially
'incude_generated_column' is true.
Now if we 'ALTER SUBSCRIPTION' to set 'include_generated_columns' as
false. Initial rows will have data for c2 on the subscriber table, but
will not have value after alter. This may be an inconsistent
behaviour.


> 02. General
>
> According to the doc, we allow to alter a column to non-generated one, by ALTER
> TABLE ... ALTER COLUMN ... DROP EXPRESSION command. Not sure, what should be
> when the command is executed on the subscriber while copying the data? Should
> we continue the copy or restart? How do you think?
COPY of data will happen in a single transaction, so if we execute
'ALTER TABLE ... ALTER COLUMN ... DROP EXPRESSION' command, It will
take place after the whole COPY command will finish. So I think it
will not create any issue.

> 03. Tes tcode
>
> IIUC, REFRESH PUBLICATION can also lead the table synchronization. Can you add
> a test for that?
Added

> 04. Test code (maybe for 0001)
>
> Please test the combination with TABLE ... ALTER COLUMN ... DROP EXPRESSION command.
Added

> 05. logicalrep_rel_open
>
> ```
> +            /*
> +             * In case 'include_generated_columns' is 'false', we should skip the
> +             * check of missing attrs for generated columns.
> +             * In case 'include_generated_columns' is 'true', we should check if
> +             * corresponding column for the generated column in publication column
> +             * list is present in the subscription table.
> +             */
> +            if (!MySubscription->includegencols && attr->attgenerated)
> +            {
> +                entry->attrmap->attnums[i] = -1;
> +                continue;
> +            }
> ```
>
> This comment is not very clear to me, because here we do not skip anything.
> Can you clarify the reason why attnums[i] is set to -1 and how will it be used?
This part of the code is removed to address some comments.

> 06. make_copy_attnamelist
>
> ```
> +    gencollist = palloc0(MaxTupleAttributeNumber * sizeof(bool));
> ```
>
> I think this array is too large. Can we reduce a size to (desc->natts * sizeof(bool))?
> Also, the free'ing should be done.
I have changed the name 'gencollist' to 'localgenlist' to make the
name more consistent. Also
size should be (rel->remoterel.natts * sizeof(bool)) as I am storing
if a column is generated like 'localgenlist[attnum] = true;'
where 'attnum' is corresponding attribute number on publisher side.

> 07. make_copy_attnamelist
>
> ```
> +    /* Loop to handle subscription table generated columns. */
> +    for (int i = 0; i < desc->natts; i++)
> ```
>
> IIUC, the loop is needed to find generated columns on the subscriber side, right?
> Can you clarify as comment?
Fixed

> 08. copy_table
>
> ```
> +    /*
> +     * Regular table with no row filter and 'include_generated_columns'
> +     * specified as 'false' during creation of subscription.
> +     */
> ```
>
> I think this comment is not correct. After patching, all tablesync command becomes
> like COPY (SELECT ...) if include_genereted_columns is set to true. Is it right?
> Can we restrict only when the table has generated ones?
Fixed

Please refer to v14 patch for the changes [1].


[1]: https://www.postgresql.org/message-id/CANhcyEW95M_usF1OJDudeejs0L0%2BYOEb%3DdXmt_4Hs-70%3DCXa-g%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  70. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-04T10:32:59Z

    On Wed, 26 Jun 2024 at 08:06, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shlok. Here are my review comments for patch v10-0003
>
> ======
> General.
>
> 1.
> The patch has lots of conditions like:
> if (att->attgenerated && (att->attgenerated !=
> ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
>  continue;
>
> IMO these are hard to read. Although more verbose, please consider if
> all those (for the sake of readability) would be better re-written
> like below :
>
> if (att->generated)
> {
>   if (!include_generated_columns)
>     continue;
>
>   if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
>     continue;
> }
Fixed

> ======
> contrib/test_decoding/test_decoding.c
>
> tuple_to_stringinfo:
>
> NITPICK = refactored the code and comments a bit here to make it easier
> NITPICK - there is no need to mention "virtual". Instead, say we only
> support STORED
Fixed

> ======
> src/backend/catalog/pg_publication.c
>
> publication_translate_columns:
>
> NITPICK - introduced variable 'att' to simplify this code
Fixed

> ~
>
> 2.
> + ereport(ERROR,
> + errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> + errmsg("cannot use virtual generated column \"%s\" in publication
> column list",
> +    colname));
>
> Is it better to avoid referring to "virtual" at all? Instead, consider
> rearranging the wording to say something like "generated column \"%s\"
> is not STORED so cannot be used in a publication column list"
Fixed

> ~~~
>
> pg_get_publication_tables:
>
> NITPICK - split the condition code for readability
Fixed

> ======
> src/backend/replication/logical/relation.c
>
> 3. logicalrep_rel_open
>
> + if (attr->attgenerated && attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
> + continue;
> +
>
> Isn't this missing some code to say "entry->attrmap->attnums[i] =
> -1;", same as all the other nearby code is doing?
Fixed

> ~~~
>
> 4.
> I felt all the "generated column" logic should be kept together, so
> this new condition should be combined with the other generated column
> condition, like:
>
> if (attr->attgenerated)
> {
>   /* comment... */
>   if (!MySubscription->includegencols)
>   {
>     entry->attrmap->attnums[i] = -1;
>     continue;
>   }
>
>   /* comment... */
>   if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
>   {
>     entry->attrmap->attnums[i] = -1;
>     continue;
>   }
> }
Fixed

> ======
> src/backend/replication/logical/tablesync.c
>
> 5.
> + if (gencols_allowed)
> + {
> + /* Replication of generated cols is supported, but not VIRTUAL cols. */
> + appendStringInfo(&cmd, " AND a.attgenerated != 'v'");
> + }
>
> Is it better here to use the ATTRIBUTE_GENERATED_VIRTUAL macro instead
> of the hardwired 'v'? (Maybe add another TODO comment to revisit
> this).
>
> Alternatively, consider if it is more future-proof to rearrange so it
> just says what *is* supported instead of what *isn't* supported:
> e.g. "AND a.attgenerated IN ('', 's')"
I feel we should use ATTRIBUTE_GENERATED_VIRTUAL macro. Added a TODO.

> ======
> src/test/subscription/t/011_generated.pl
>
> NITPICK - some comments are missing the word "stored"
> NITPICK - sometimes "col" should be plural "cols"
> NITPICK = typo "GNERATED"
Add the relevant changes.

> ======
>
> 6.
> In a previous review [1, comment #3] I mentioned that there should be
> some docs updates on the "Logical Replication Message Formats" section
> 53.9. So, I expected patch 0001 would make some changes and then patch
> 0003 would have to update it again to say something about "STORED".
> But all that is missing from the v10* patches.
>
> ======
Will fix in upcoming version

>
> 99.
> See also my nitpicks diff which is a top-up patch addressing all the
> nitpick comments mentioned above. Please apply all of these that you
> agree with.
Applied Relevant changes

Please refer v14 patch for the changes [1].


[1]: https://www.postgresql.org/message-id/CANhcyEW95M_usF1OJDudeejs0L0%2BYOEb%3DdXmt_4Hs-70%3DCXa-g%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  71. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-05T08:17:22Z

    Here are my review comments for v14-0002.

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:

nitpick - remove excessive parentheses in palloc0 call.

nitpick - Code is fine AFAICT except it's not immediately obvious
localgenlist is indexed by the *remote* attribute number. So I renamed
'attrnum' variable in my nitpicks diff. OTOH, if you think no change
is necessary, that is OK to (in that case maybe add a comment).

~~~

1. fetch_remote_table_info

+ if ((server_version >= 120000 && server_version <= 160000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");

Should this say < 180000 instead of <= 160000?

~~~

copy_table:

nitpick - uppercase in comment

nitpick - missing space after "if"

~~~

2. copy_table

+ attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
+
  /* Start copy on the publisher. */
  initStringInfo(&cmd);

- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /* check if remote column list has generated columns */
+ if(MySubscription->includegencols)
+ {
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if(remotegenlist[i])
+ {
+ remote_has_gencol = true;
+ break;
+ }
+ }
+ }
+

There is some subtle logic going on here:

For example, the comment here says "Check if the remote column list
has generated columns", and it then proceeds to iterate the remote
attributes checking the remotegenlist[i]. But the remotegenlist[] was
returned from a prior call to make_copy_attnamelist() and according to
the make_copy_attnamelist logic, it is NOT returning all remote
generated-cols in that list. Specifically, it is stripping some of
them -- "Do not include generated columns of the subscription table in
the [remotegenlist] column list.".

So, actually this loop seems to be only finding cases (setting
remote_has_gen = true) where the remote column is generated but the
match local column is *not* generated. Maybe this was the intended
logic all along but then certainly the comment should be improved to
describe it better.

~~~

3.
+ /*
+ * Regular table with no row filter and 'include_generated_columns'
+ * specified as 'false' during creation of subscription.
+ */
+ if (lrel.relkind == RELKIND_RELATION && qual == NIL && !remote_has_gencol)

nitpick - This comment also needs improving. For example, just because
remote_has_gencol is false, it does not follow that
'include_generated_columns' was specified as 'false' -- maybe the
parameter was 'true' but the table just had no generated columns
anyway... I've modified the comment already in my nitpicks diff, but
probably you can improve on that.

~

nitpick - "else" comment is modified slightly too. Please see the nitpicks diff.

~

4.
In hindsight, I felt your variable 'remote_has_gencol' was not
well-named because it is not for saying the remote table has a
generated column -- it is saying the remote table has a generated
column **that we have to copy**. So, rather it should be named
something like 'gencol_copy_needed' (but I didn't change this name in
the nitpick diffs...)

======
src/test/subscription/t/004_sync.pl

nitpick - changes to comment style to make the test case separations
much more obvious
nitpick - minor comment wording tweaks

5.
Here, you are confirming we get an ERROR when replicating from a
non-generated column to a generated column. But I think your patch
also added exactly that same test scenario in the 011_generated (as
the sub5 test). So, maybe this one here should be removed?

======
src/test/subscription/t/011_generated.pl

nitpick - comment wrapping at 80 chars
nitpick - add/remove blank lines for readability
nitpick - typo /subsriber/subscriber/
nitpick - prior to the ALTER test, tab6 is unsubscribed. So add
another test to verify its initial data
nitpick - sometimes the msg 'add a new table to existing publication'
is misplaced
nitpick - the tests for tab6 and tab5 were in opposite to the expected
order, so swapped them.

======
99.
Please see also the attached diff which implements all the nitpicks
described in this post.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  72. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-05T12:01:11Z

    On Tue, Jul 2, 2024 at 9:53 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Jul 1, 2024 at 8:38 PM Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> >...
> > > 8.
> > > + else if (strcmp(elem->defname, "include-generated-columns") == 0)
> > > + {
> > > + if (elem->arg == NULL)
> > > + data->include_generated_columns = true;
> > >
> > > Is there any way to test that "elem->arg == NULL" in the
> > > generated.sql? OTOH, if it is not possible to get here then is the
> > > code even needed?
> > >
> >
> > Currently I could not find a case where the
> > 'include_generated_columns' option is not specifying any value, but  I
> > was hesitant to remove this from here as the other options mentioned
> > follow the same rules. Thoughts?
> >
>
> If you do manage to find a scenario for this then I think a test for
> it would be good. But, I agree that the code seems OK because now I
> see it is the same pattern as similar nearby code.
>
> ~~~
>
> Thanks for the updated patch. Here are some review comments for patch v13-0001.
>
> ======
> .../expected/generated_columns.out
>
> nitpicks (see generated_columns.sql)
>
> ======
> .../test_decoding/sql/generated_columns.sql
>
> nitpick - use plural /column/columns/
> nitpick - use consistent wording in the comments
> nitpick - IMO it is better to INSERT different values for each of the tests
>
> ======
> doc/src/sgml/protocol.sgml
>
> nitpick - I noticed that none of the other boolean parameters on this
> page mention about a default, so maybe here we should do the same and
> omit that information.
>
> ~~~
>
> 1.
> -     <para>
> -      Next, the following message part appears for each column included in
> -      the publication (except generated columns):
> -     </para>
> -
>
> In a previous review [1 comment #11] I wrote that you can't just
> remove this paragraph because AFAIK it is still meaningful. A minimal
> change might be to just remove the "(except generated columns)" part.
> Alternatively, you could give a more detailed explanation mentioning
> the include_generated_columns protocol parameter.
>
> I provided some updated text for this paragraph in my NITPICKS top-up
> patch, Please have a look at that for ideas.
>
> ======
> src/backend/commands/subscriptioncmds.c
>
> It looks like pg_indent needs to be run on this file.
>
> ======
> src/include/catalog/pg_subscription.h
>
> nitpick - comment /publish/Publish/ for consistency
>
> ======
> src/include/replication/walreceiver.h
>
> nitpick - comment /publish/Publish/ for consistency
>
> ======
> src/test/regress/expected/subscription.out
>
> nitpicks - (see subscription.sql)
>
> ======
> src/test/regress/sql/subscription.sql
>
> nitpick - combine the invalid option combinations test with all the
> others (no special comment needed)
> nitpick - rename subscription as 'regress_testsub2' same as all its peers.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> nitpick - add/remove blank lines
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> nitpick - rewording for a comment. This issue was not strictly caused
> by this patch, but since you are modifying the same comment we can fix
> this in passing.
>
> ======
> 99.
> Please also see the attached top-up patch for all those nitpicks
> identified above.
>
> ======
> [1] v11-0001 review
> https://www.postgresql.org/message-id/CAHut%2BPv45gB4cV%2BSSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg%40mail.gmail.com

All the comments are handled.

The attached Patches contain all the suggested changes. Here, v15-0001
is modified to fix the comments, v15-0002 is not modified and v15-0003
is modified according to the changes in v15-0001 patch.
Thanks Shlok Kyal for modifying the v15-0003 Patch.


Thanks and Regards,
Shubham Khanna.

  73. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-05T12:05:33Z

    On Tue, Jul 2, 2024 at 10:59 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham,
>
> As you can see, most of my recent review comments for patch 0001 are
> only cosmetic nitpicks. But, there is still one long-unanswered design
> question from a month ago [1, #G.2]
>
> A lot of the patch code of pgoutput.c and proto.c and logicalproto.h
> is related to the introduction and passing everywhere of new
> 'include_generated_columns' function parameters. These same functions
> are also always passing "BitMapSet *columns" representing the
> publication column list.
>
> My question was about whether we can't make use of the existing BMS
> parameter instead of introducing all the new API parameters.
>
> The idea might go something like this:
>
> * If 'include_generated_columns' option is specified true and if no
> column list was already specified then perhaps the relentry->columns
> can be used for a "dummy" column list that has everything including
> all the generated columns.
>
> * By doing this:
>  -- you may be able to avoid passing the extra
> 'include_gernated_columns' everywhere
>  -- you may be able to avoid checking for generated columns deeper in
> the code (since it is already checked up-front when building the
> column list BMS)
>
> ~~
>
> I'm not saying this design idea is guaranteed to work, but it might be
> worth considering, because if it does work then there is potential to
> make the current 0001 patch significantly shorter.
>
> ======
> [1] https://www.postgresql.org/message-id/CAHut%2BPsuJfcaeg6zst%3D6PE5uyJv_UxVRHU3ck7W2aHb1uQYKng%40mail.gmail.com

I have fixed this issue in the latest Patches.

Please refer to the updated v15 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2B%3Dhn--ALJQvzzu7meX3LuO3tJKppDS7eO1BGvNFYBAbg%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  74. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-08T05:22:54Z

    Here are review comments for v15-0001

======
doc/src/sgml/ddl.sgml

nitpick - there was a comma (,) which should be a period (.)

======
.../libpqwalreceiver/libpqwalreceiver.c

1.
+ if (options->proto.logical.include_generated_columns &&
+ PQserverVersion(conn->streamConn) >= 170000)
+ appendStringInfoString(&cmd, ", include_generated_columns 'true'");
+

Should now say >= 180000

======
src/backend/replication/pgoutput/pgoutput.c

nitpick - comment wording for RelationSyncEntry.collist.

~~

2.
pgoutput_column_list_init:

I found the current logic to be quite confusing. I assume the code is
working OK, because AFAIK there are plenty of tests and they are all
passing, but the logic seems somewhat repetitive and there are also no
comments to explain it adding to my confusion.

IIUC, PRIOR TO THIS PATCH:

BMS field 'columns' represented the "columns of the column list" or it
was NULL if there was no publication column list (and it was also NULL
if the column list contained every column).

IIUC NOW, WITH THIS PATCH:

The BMS field 'columns' meaning is changed slightly to be something
like "columns to be replicated" or NULL if all columns are to be
replicated. This is almost the same thing except we are now handing
the generated columns up-front, so generated columns will or won't
appear in the BMS according to the "include_generated_columns"
parameter. See how this is all a bit subtle which is why copious new
comments are required to explain it...

So, although the test result evidence suggests this is working OK, I
have many questions/issues about it. Here are some to start with:

2a. It needs a lot more (summary and detailed) comments explaining the
logic now that the meaning is slightly different.

2b. What is the story with the FOR ALL TABLES case now? Previously,
there would always be NULL 'columns' for "FOR ALL TABLES" case -- the
comment still says so. But now you've tacked on a 2nd pass of
iterations to build the BMS outside of the "if (!pub->alltables)"
check. Is that OK?

2c. The following logic seemed unexpected:
- if (bms_num_members(cols) == nliveatts)
+ if (bms_num_members(cols) == nliveatts &&
+ data->include_generated_columns)
  {
  bms_free(cols);
  cols = NULL;
`
I had thought the above code would look different -- more like:
if (att->attgenerated && !data->include_generated_columns)
  continue;

nliveatts++;
...

2d. Was so much duplicated code necessary? It feels like the whole
"Get the number of live attributes." and assignment of cols to NULL
might be made common to both code paths.

2e. I'm beginning to question the pros/cons of the new BMS logic; I
had suggested trying this way (processing the generated columns
up-front in the BMS 'columns' list) to reduce patch code and simplify
all the subsequent API delegation of "include_generated_cloumns"
everywhere like it was in v14-0001. Indeed, that part was a success
and the patch is now smaller. But I don't like much that we've traded
reduced code overall for increased confusing code in that BMS
function. If all this BMS code can be refactored and commented to be
easier to understand then maybe all will be well, but if it can't then
maybe this BMS change was a bridge too far. I haven't given up on it
just yet, but I wonder what was your opinion about it, and do other
people have thoughts about whether this was the good direction to
take?

======
src/bin/pg_dump/pg_dump.c

3.
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subincludegencols\n");
+ else
+ appendPQExpBufferStr(query,
+ " false AS subincludegencols\n");

Should now say >= 180000

======
src/bin/psql/describe.c

4.
+ /* include_generated_columns is only supported in v18 and higher */
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+   ", subincludegencols AS \"%s\"\n",
+   gettext_noop("Include generated columns"));
+

Should now say >= 180000

======
src/include/catalog/pg_subscription.h

nitpick - let's make the comment the same as in WalRcvStreamOptions

======
src/include/replication/logicalproto.h

nitpick - extern for logicalrep_write_update should be unchanged by this patch

======
src/test/regress/sql/subscription.sql

nitpick = the comment "include_generated_columns and copy_data = true
are mutually exclusive" is not necessary because this all falls under
the existing comment "fail - invalid option combinations"

nitpick - let's explicitly put "copy_data = true" in the CREATE
SUBSCRIPTION to make it more obvious

======
99. Please also refer to the attached 'diffs' patch which implements
all of my nitpicks issues mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  75. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-08T07:50:26Z

    Hi Shlok, Here are some review comments for patch v15-0003.

======
src/backend/catalog/pg_publication.c

1. publication_translate_columns

The function comment says:
 * Translate a list of column names to an array of attribute numbers
 * and a Bitmapset with them; verify that each attribute is appropriate
 * to have in a publication column list (no system or generated attributes,
 * no duplicates).  Additional checks with replica identity are done later;
 * see pub_collist_contains_invalid_column.

That part about "[no] generated attributes" seems to have gone stale
-- e.g. not quite correct anymore. Should it say no VIRTUAL generated
attributes?

======
src/backend/replication/logical/proto.c

2. logicalrep_write_tuple and logicalrep_write_attrs

I thought all the code fragments like this:

+ if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+

don't need to be in the code anymore, because of the BitMapSet (BMS)
processing done to make the "column list" for publication where
disallowed generated cols should already be excluded from the BMS,
right?

So shouldn't all these be detected by the following statement:
if (!column_in_column_list(att->attnum, columns))
  continue;

======
src/backend/replication/logical/tablesync.c
3.
+ if(server_version >= 120000)
+ {
+ bool gencols_allowed = server_version >= 170000 &&
MySubscription->includegencols;
+
+ if (gencols_allowed)
+ {

Should say server_version >= 180000, instead of 170000

======
src/backend/replication/pgoutput/pgoutput.c

4. send_relation_and_attrs

(this is a similar comment for #2 above)

IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
process the generated columns up-front means there is no need to check
them again in code like this.

They should be discovered anyway in the subsequent check:
/* Skip this attribute if it's not present in the column list */
if (columns != NULL && !bms_is_member(att->attnum, columns))
  continue;

======
src/test/subscription/t/011_generated.pl

5.
AFAICT there are still multiple comments (e.g. for the "TEST tab<n>"
comments) where it still says "generated" instead of "stored
generated". I did not make a "nitpicks" diff for these because those
comments are inherited from the prior patch 0002 which still has
outstanding review comments on it too. Please just search/replace
them.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  76. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-08T12:03:14Z

    On Fri, 5 Jul 2024 at 13:47, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v14-0002.
>
> ======
> src/backend/replication/logical/tablesync.c
>
> 2. copy_table
>
> + attnamelist = make_copy_attnamelist(relmapentry, remotegenlist);
> +
>   /* Start copy on the publisher. */
>   initStringInfo(&cmd);
>
> - /* Regular table with no row filter */
> - if (lrel.relkind == RELKIND_RELATION && qual == NIL)
> + /* check if remote column list has generated columns */
> + if(MySubscription->includegencols)
> + {
> + for (int i = 0; i < relmapentry->remoterel.natts; i++)
> + {
> + if(remotegenlist[i])
> + {
> + remote_has_gencol = true;
> + break;
> + }
> + }
> + }
> +
>
> There is some subtle logic going on here:
>
> For example, the comment here says "Check if the remote column list
> has generated columns", and it then proceeds to iterate the remote
> attributes checking the remotegenlist[i]. But the remotegenlist[] was
> returned from a prior call to make_copy_attnamelist() and according to
> the make_copy_attnamelist logic, it is NOT returning all remote
> generated-cols in that list. Specifically, it is stripping some of
> them -- "Do not include generated columns of the subscription table in
> the [remotegenlist] column list.".
>
> So, actually this loop seems to be only finding cases (setting
> remote_has_gen = true) where the remote column is generated but the
> match local column is *not* generated. Maybe this was the intended
> logic all along but then certainly the comment should be improved to
> describe it better.

'remotegenlist' is actually constructed in function 'fetch_remote_table_info'
and it has an entry for every column in the column list specifying
whether a column is
generated or not.
In the function 'make_copy_attnamelist' we are not modifying the list.
So, I think the current comment would be sufficient. Thoughts?

> ======
> src/test/subscription/t/004_sync.pl
>
> nitpick - changes to comment style to make the test case separations
> much more obvious
> nitpick - minor comment wording tweaks
>
> 5.
> Here, you are confirming we get an ERROR when replicating from a
> non-generated column to a generated column. But I think your patch
> also added exactly that same test scenario in the 011_generated (as
> the sub5 test). So, maybe this one here should be removed?

For 0004_sync.pl, it is tested when 'include_generated_columns' is not
specified. Whereas for the test in 011_generated
'include_generated_columns = true' is specified.
I thought we should have a test for both cases to test if the error
message format is the same for both cases. Thoughts?

I have attached the patches and I have addressed the rest of the
comment and added changes in v16-0002. I have not modified the
v16-0001 patch.


Thanks and Regards,
Shlok Kyal

  77. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-08T12:04:38Z

    On Mon, 8 Jul 2024 at 13:20, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shlok, Here are some review comments for patch v15-0003.
>
> ======
> src/backend/catalog/pg_publication.c
>
> 1. publication_translate_columns
>
> The function comment says:
>  * Translate a list of column names to an array of attribute numbers
>  * and a Bitmapset with them; verify that each attribute is appropriate
>  * to have in a publication column list (no system or generated attributes,
>  * no duplicates).  Additional checks with replica identity are done later;
>  * see pub_collist_contains_invalid_column.
>
> That part about "[no] generated attributes" seems to have gone stale
> -- e.g. not quite correct anymore. Should it say no VIRTUAL generated
> attributes?
Yes, we should use VIRTUAL generated attributes, I have modified it.

> ======
> src/backend/replication/logical/proto.c
>
> 2. logicalrep_write_tuple and logicalrep_write_attrs
>
> I thought all the code fragments like this:
>
> + if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
> + continue;
> +
>
> don't need to be in the code anymore, because of the BitMapSet (BMS)
> processing done to make the "column list" for publication where
> disallowed generated cols should already be excluded from the BMS,
> right?
>
> So shouldn't all these be detected by the following statement:
> if (!column_in_column_list(att->attnum, columns))
>   continue;
The current BMS logic do not handle the Virtual Generated Columns.
There can be cases where we do not want a virtual generated column but
it would be present in BMS.
To address this I have added the above logic. I have added this logic
similar to the checks of 'attr->attisdropped'.

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 4. send_relation_and_attrs
>
> (this is a similar comment for #2 above)
>
> IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
> process the generated columns up-front means there is no need to check
> them again in code like this.
>
> They should be discovered anyway in the subsequent check:
> /* Skip this attribute if it's not present in the column list */
> if (columns != NULL && !bms_is_member(att->attnum, columns))
>   continue;
Same explanation as above.

I have addressed all the comments in v16-0003 patch. Please refer [1].
[1]: https://www.postgresql.org/message-id/CANhcyEXw%3DBFFVUqohWES9EPkdq-ZMC5QRBVQqQPzrO%3DQ7uzFQw%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  78. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-09T01:44:10Z

    Hi Shlok, Here are my review comments for v16-0002

======
src/backend/replication/logical/tablesync.c

1. fetch_remote_table_info

+ if ((server_version >= 120000 && server_version < 180000) ||
+ !MySubscription->includegencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");

I felt this condition was a bit complicated. it needs a comment to
explain that "attgenerated" has been supported only since >= PG12 and
'include_generated_columns' is supported only since >= PG18. The more
I look at this I think this is a bug. For example, what happens if the
server is *before* PG12 and include_generated_cols is false; won't it
then try to build SQL using the "attgenerated" column which will cause
an ERROR on the server?

IIRC this condition is already written properly in your patch 0003.
So, most of that 0003 condition refactoring should be done here in
patch 0002 instead.

~~~

2. copy_table

> > So, actually this loop seems to be only finding cases (setting
> > remote_has_gen = true) where the remote column is generated but the
> > match local column is *not* generated. Maybe this was the intended
> > logic all along but then certainly the comment should be improved to
> > describe it better.
>
> 'remotegenlist' is actually constructed in function 'fetch_remote_table_info'
> and it has an entry for every column in the column list specifying
> whether a column is
> generated or not.
> In the function 'make_copy_attnamelist' we are not modifying the list.
> So, I think the current comment would be sufficient. Thoughts?

Yes, I was mistaken thinking the list is "modified". OTOH, I still
feel the existing comment ("Check if remote column list has any
generated column") is misleading because the remote table might have
generated cols but we are not even interested in them if the
equivalent subscriber column is also generated. Please see nitpicks
diff, for my suggestion how to update this comment.

~~~

nitpick - add space after "if"

======
src/test/subscription/t/004_sync.pl

> > 5.
> > Here, you are confirming we get an ERROR when replicating from a
> > non-generated column to a generated column. But I think your patch
> > also added exactly that same test scenario in the 011_generated (as
> > the sub5 test). So, maybe this one here should be removed?
>
> For 0004_sync.pl, it is tested when 'include_generated_columns' is not
> specified. Whereas for the test in 011_generated
> 'include_generated_columns = true' is specified.
> I thought we should have a test for both cases to test if the error
> message format is the same for both cases. Thoughts?

3.
Sorry, I missed that there was a parameter flag difference. Anyway,
since the code-path to reach this error is the same regardless of the
'include_generated_columns' parameter value IMO having too many tests
might be overkill. YMMV.

Anyway, whether you decide to keep both test cases or not, I think all
testing related to generated column replication belongs in the new
001_generated.pl TAP file -- not here in 04_sync.pl
.
======
src/test/subscription/t/011_generated.pl

4. Untested scenarios for "missing col"?

I have seen (in 04_sync.pl) missing column test cases for:
- publisher not-generated col ==> subscriber missing column

Maybe I am mistaken, but I don't recall seeing any test cases for:
- publisher generated-col ==> subscriber missing col

Unless they are already done somewhere, I think this scenario should
be in 011_generated.pl. Furthermore, maybe it needs to be tested for
both include_generated_columns = true / false, because if the
parameter is false it should be OK, but if the parameter is true it
should give ERROR.

~~~

5.
-# publisher-side tab3 has generated col 'b' but subscriber-side tab3
has DIFFERENT COMPUTATION generated col 'b'.
+# tab3:
+# publisher-side tab3 has generated col 'b' but
+# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.

I think this change is only improving a comment that was introduced by
patch 0001. This all belongs back in patch 0001, then patch 0002 has
nothing to do here.

======
99.
Please also refer to the attached diffs patch which implements any
nitpicks mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  79. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-09T04:22:46Z

    Hi Shlok, here are my review comments for v16-0003.

======
src/backend/replication/logical/proto.c


On Mon, Jul 8, 2024 at 10:04 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> On Mon, 8 Jul 2024 at 13:20, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> >
> > 2. logicalrep_write_tuple and logicalrep_write_attrs
> >
> > I thought all the code fragments like this:
> >
> > + if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
> > + continue;
> > +
> >
> > don't need to be in the code anymore, because of the BitMapSet (BMS)
> > processing done to make the "column list" for publication where
> > disallowed generated cols should already be excluded from the BMS,
> > right?
> >
> > So shouldn't all these be detected by the following statement:
> > if (!column_in_column_list(att->attnum, columns))
> >   continue;
> The current BMS logic do not handle the Virtual Generated Columns.
> There can be cases where we do not want a virtual generated column but
> it would be present in BMS.
> To address this I have added the above logic. I have added this logic
> similar to the checks of 'attr->attisdropped'.
>

Hmm. I thought the BMS idea of patch 0001 is to discover what columns
should be replicated up-front. If they should not be replicated (e.g.
virtual generated columns cannot be) then they should never be in the
BMS.

So what you said ("There can be cases where we do not want a virtual
generated column but it would be present in BMS") should not be
happening. If that is happening then it sounds more like a bug in the
new BMS logic of pgoutput_column_list_init() function. In other words,
if what you say is true, then it seems like the current extra
conditions you have in patch 0004 are just a band-aid to cover a
problem of the BMS logic of patch 0001. Am I mistaken?

> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 4. send_relation_and_attrs
> >
> > (this is a similar comment for #2 above)
> >
> > IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
> > process the generated columns up-front means there is no need to check
> > them again in code like this.
> >
> > They should be discovered anyway in the subsequent check:
> > /* Skip this attribute if it's not present in the column list */
> > if (columns != NULL && !bms_is_member(att->attnum, columns))
> >   continue;
> Same explanation as above.

As above.

======
src/test/subscription/t/011_generated.pl

I'm not sure if you needed to say "STORED" generated cols for the
subscriber-side columns but anyway, whatever is done needs to be done
consistently. FYI, below you did *not* say STORED for subscriber-side
generated cols, but in other comments for subscriber-side generated
columns, you did say STORED.

# tab3:
# publisher-side tab3 has STORED generated col 'b' but
# subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.

~

# tab4:
# publisher-side tab4 has STORED generated cols 'b' and 'c' but
# subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
# where columns on publisher/subscriber are in a different order

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  80. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-09T22:51:55Z

    Hi Shubham/Shlok, I was thinking some more about the suggested new
BitMapSet (BMS) idea of patch 0001 that changes the 'columns' meaning
to include generated cols also where necessary.

I feel it is a bit risky to change lots of code without being 100%
confident it will still be in the final push. It's also going to make
the reviewing job harder if stuff gets added and then later removed.

IMO it might be better to revert all the patches (mostly 0001, but
also parts of subsequent patches) to their pre-BMS-change ~v14* state.
Then all the BMS "improvement" can be kept isolated in a new patch
0004.

Some more reasons to split this off into a separate patch are:

* The BMS change is essentially a redesign/cleanup of the code but is
nothing to do with the actual *functionality* of the new "generated
columns" feature.

* Apart from the BMS change I think the rest of the patches are nearly
stable now. So it might be good to get it all finished so the BMS
change can be tackled separately.

* By isolating the BMS change, then we will be able to see exactly
what is the code cost/benefit (e.g. removal of redundant code versus
adding new logic) which is part of the judgement to decide whether to
do it this way or not.

* By isolating the BMS change, then it makes it convenient for testing
before/after in case there are any performance concerns

* By isolating the BMS change, if some unexpected obstacle is
encountered that makes it unfeasible then we can just throw away patch
0004 and everything else (patches 0001,0002,0003) will still be good
to go.

Thoughts?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  81. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-11T06:12:58Z

    On Mon, Jul 8, 2024 at 10:53 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are review comments for v15-0001
>
> ======
> doc/src/sgml/ddl.sgml
>
> nitpick - there was a comma (,) which should be a period (.)
>
> ======
> .../libpqwalreceiver/libpqwalreceiver.c
>
> 1.
> + if (options->proto.logical.include_generated_columns &&
> + PQserverVersion(conn->streamConn) >= 170000)
> + appendStringInfoString(&cmd, ", include_generated_columns 'true'");
> +
>
> Should now say >= 180000
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> nitpick - comment wording for RelationSyncEntry.collist.
>
> ~~
>
> 2.
> pgoutput_column_list_init:
>
> I found the current logic to be quite confusing. I assume the code is
> working OK, because AFAIK there are plenty of tests and they are all
> passing, but the logic seems somewhat repetitive and there are also no
> comments to explain it adding to my confusion.
>
> IIUC, PRIOR TO THIS PATCH:
>
> BMS field 'columns' represented the "columns of the column list" or it
> was NULL if there was no publication column list (and it was also NULL
> if the column list contained every column).
>
> IIUC NOW, WITH THIS PATCH:
>
> The BMS field 'columns' meaning is changed slightly to be something
> like "columns to be replicated" or NULL if all columns are to be
> replicated. This is almost the same thing except we are now handing
> the generated columns up-front, so generated columns will or won't
> appear in the BMS according to the "include_generated_columns"
> parameter. See how this is all a bit subtle which is why copious new
> comments are required to explain it...
>
> So, although the test result evidence suggests this is working OK, I
> have many questions/issues about it. Here are some to start with:
>
> 2a. It needs a lot more (summary and detailed) comments explaining the
> logic now that the meaning is slightly different.
>
> 2b. What is the story with the FOR ALL TABLES case now? Previously,
> there would always be NULL 'columns' for "FOR ALL TABLES" case -- the
> comment still says so. But now you've tacked on a 2nd pass of
> iterations to build the BMS outside of the "if (!pub->alltables)"
> check. Is that OK?
>
> 2c. The following logic seemed unexpected:
> - if (bms_num_members(cols) == nliveatts)
> + if (bms_num_members(cols) == nliveatts &&
> + data->include_generated_columns)
>   {
>   bms_free(cols);
>   cols = NULL;
> `
> I had thought the above code would look different -- more like:
> if (att->attgenerated && !data->include_generated_columns)
>   continue;
>
> nliveatts++;
> ...
>
> 2d. Was so much duplicated code necessary? It feels like the whole
> "Get the number of live attributes." and assignment of cols to NULL
> might be made common to both code paths.
>
> 2e. I'm beginning to question the pros/cons of the new BMS logic; I
> had suggested trying this way (processing the generated columns
> up-front in the BMS 'columns' list) to reduce patch code and simplify
> all the subsequent API delegation of "include_generated_cloumns"
> everywhere like it was in v14-0001. Indeed, that part was a success
> and the patch is now smaller. But I don't like much that we've traded
> reduced code overall for increased confusing code in that BMS
> function. If all this BMS code can be refactored and commented to be
> easier to understand then maybe all will be well, but if it can't then
> maybe this BMS change was a bridge too far. I haven't given up on it
> just yet, but I wonder what was your opinion about it, and do other
> people have thoughts about whether this was the good direction to
> take?

I have created a separate patch(v17-0004) for this idea. Will address
this comment in the next version of patches.

> ======
> src/bin/pg_dump/pg_dump.c
>
> 3.
> + if (fout->remoteVersion >= 170000)
> + appendPQExpBufferStr(query,
> + " s.subincludegencols\n");
> + else
> + appendPQExpBufferStr(query,
> + " false AS subincludegencols\n");
>
> Should now say >= 180000
>
> ======
> src/bin/psql/describe.c
>
> 4.
> + /* include_generated_columns is only supported in v18 and higher */
> + if (pset.sversion >= 170000)
> + appendPQExpBuffer(&buf,
> +   ", subincludegencols AS \"%s\"\n",
> +   gettext_noop("Include generated columns"));
> +
>
> Should now say >= 180000
>
> ======
> src/include/catalog/pg_subscription.h
>
> nitpick - let's make the comment the same as in WalRcvStreamOptions
>
> ======
> src/include/replication/logicalproto.h
>
> nitpick - extern for logicalrep_write_update should be unchanged by this patch
>
> ======
> src/test/regress/sql/subscription.sql
>
> nitpick = the comment "include_generated_columns and copy_data = true
> are mutually exclusive" is not necessary because this all falls under
> the existing comment "fail - invalid option combinations"
>
> nitpick - let's explicitly put "copy_data = true" in the CREATE
> SUBSCRIPTION to make it more obvious
>
> ======
> 99. Please also refer to the attached 'diffs' patch which implements
> all of my nitpicks issues mentioned above.

The attached Patches contain all the suggested changes. Here, v17-0001
is modified to fix the comments, v17-0002 and v17-0003 are modified
according to the changes in v17-0001 patch and v17-0004 patch contains
the changes related to Bitmapset(BMS) idea that changes the 'columns'
meaning to include generated cols also where necessary.

Thanks and Regards,
Shubham Khanna.

  82. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-11T06:16:51Z

    On Wed, Jul 10, 2024 at 4:22 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham/Shlok, I was thinking some more about the suggested new
> BitMapSet (BMS) idea of patch 0001 that changes the 'columns' meaning
> to include generated cols also where necessary.
>
> I feel it is a bit risky to change lots of code without being 100%
> confident it will still be in the final push. It's also going to make
> the reviewing job harder if stuff gets added and then later removed.
>
> IMO it might be better to revert all the patches (mostly 0001, but
> also parts of subsequent patches) to their pre-BMS-change ~v14* state.
> Then all the BMS "improvement" can be kept isolated in a new patch
> 0004.
>
> Some more reasons to split this off into a separate patch are:
>
> * The BMS change is essentially a redesign/cleanup of the code but is
> nothing to do with the actual *functionality* of the new "generated
> columns" feature.
>
> * Apart from the BMS change I think the rest of the patches are nearly
> stable now. So it might be good to get it all finished so the BMS
> change can be tackled separately.
>
> * By isolating the BMS change, then we will be able to see exactly
> what is the code cost/benefit (e.g. removal of redundant code versus
> adding new logic) which is part of the judgement to decide whether to
> do it this way or not.
>
> * By isolating the BMS change, then it makes it convenient for testing
> before/after in case there are any performance concerns
>
> * By isolating the BMS change, if some unexpected obstacle is
> encountered that makes it unfeasible then we can just throw away patch
> 0004 and everything else (patches 0001,0002,0003) will still be good
> to go.

As suggested, I have created  a separate patch for the Bitmapset(BMS)
idea of patch 0001 that changes the 'columns' meaning to include
generated cols also where necessary.
Please refer to the updated v17 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjJ0gAUd62PvBRXCPYy2oTNZWEY-Qe8cBNzQaJPVMZCeGA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  83. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-12T06:43:15Z

    Hi Shubham.

Thanks for separating the new BMS 'columns' modification.

Here are my review comments for the latest patch v17-0001.

======

1. src/backend/replication/pgoutput/pgoutput.c

  /*
  * Columns included in the publication, or NULL if all columns are
  * included implicitly.  Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side.  Note that the attnums in this bitmap are not
  * shifted by FirstLowInvalidHeapAttributeNumber.
  */
  Bitmapset  *columns;
With this latest 0001 there is now no change to the original
interpretation of RelationSyncEntry BMS 'columns'. So, I think this
field comment should remain unchanged; i.e. it should be the same as
the current HEAD comment.

======
src/test/subscription/t/011_generated.pl

nitpick - comment changes for 'tab2' and 'tab3' to make them more consistent.

======
99.
Please refer to the attached diff patch which implements any nitpicks
described above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  84. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-15T02:38:20Z

    Hi, here are some review comments about patch v17-0003

======
1.
Missing a docs change?

Previously, (v16-0002) the patch included a change to
doc/src/sgml/protocol.sgml like below to say STORED generated instead
of just generated.

        <para>
-        Boolean option to enable generated columns. This option controls
-        whether generated columns should be included in the string
-        representation of tuples during logical decoding in PostgreSQL.
+        Boolean option to enable <literal>STORED</literal> generated columns.
+        This option controls whether <literal>STORED</literal>
generated columns
+        should be included in the string representation of tuples
during logical
+        decoding in PostgreSQL.
        </para>

Why is that v16 change no longer present in patch v17-0003?

======
src/backend/catalog/pg_publication.c

2.
Previously, (v16-0003) this patch included a change to clarify the
kind of generated cols that are allowed in a column list.

  * Translate a list of column names to an array of attribute numbers
  * and a Bitmapset with them; verify that each attribute is appropriate
- * to have in a publication column list (no system or generated attributes,
- * no duplicates).  Additional checks with replica identity are done later;
- * see pub_collist_contains_invalid_column.
+ * to have in a publication column list (no system or virtual generated
+ * attributes, no duplicates). Additional checks with replica identity
+ * are done later; see pub_collist_contains_invalid_column.

Why is that v16 change no longer present in patch v17-0003?

======
src/backend/replication/logical/tablesync.c

3. make_copy_attnamelist

- if (!attr->attgenerated)
+ if (attr->attgenerated != ATTRIBUTE_GENERATED_STORED)
  continue;

IIUC this logic is checking to make sure the subscriber-side table
column was not a generated column (because we don't replicate on top
of generated columns). So, does the distinction of STORED/VIRTUAL
really matter here?

~~~

fetch_remote_table_info:
nitpick - Should not change any spaces unrelated to the patch

======

send_relation_and_attrs:

- if (att->attgenerated && !include_generated_columns)
+ if (att->attgenerated && (att->attgenerated !=
ATTRIBUTE_GENERATED_STORED || !include_generated_columns))
  continue;

nitpick - It seems over-complicated. Conditions can be split so the
code fragment looks the same as in other places in this patch.

======
99.
Please see the attached diffs patch that implements any nitpicks
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  85. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-15T05:39:01Z

    Hi, I had a quick look at the patch v17-0004 which is the split-off
new BMS logic.

IIUC this 0004 is currently undergoing some refactoring and
cleaning-up, so I won't comment much about it except to give the
following observation below.

======
src/backend/replication/logical/proto.c.

I did not expect to see any code fragments that are still checking
generated columns like below:

logicalrep_write_tuple:

  if (att->attgenerated)
  {
- if (!include_generated_columns)
- continue;

  if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
  continue;
~

  if (att->attgenerated)
  {
- if (!include_generated_columns)
- continue;

  if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
  continue;

~~~

logicalrep_write_attrs:

  if (att->attgenerated)
  {
- if (!include_generated_columns)
- continue;

  if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
  continue;

~
if (att->attgenerated)
  {
- if (!include_generated_columns)
- continue;

  if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
  continue;
~~~


AFAIK, now checking support of generated columns will be done when the
BMS 'columns' is assigned, so the continuation code will be handled
like this:

if (!column_in_column_list(att->attnum, columns))
  continue;

======

BTW there is a subtle but significant difference in this 0004 patch.
IOW, we are introducing a difference between the list of published
columns VERSUS a publication column list. So please make sure that all
code comments are adjusted appropriately so they are not misleading by
calling these "column lists" still.

BEFORE: BMS 'columns'  means "columns of the column list" or NULL if
there was no publication column list
AFTER: BMS 'columns' means "columns to be replicated" or NULL if all
columns are to be replicated

======
Kind Regards,
Peter Smith.

  86. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-16T08:57:44Z

    On Tue, 9 Jul 2024 at 07:14, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shlok, Here are my review comments for v16-0002
>
> ======
> src/test/subscription/t/004_sync.pl
>
> > > 5.
> > > Here, you are confirming we get an ERROR when replicating from a
> > > non-generated column to a generated column. But I think your patch
> > > also added exactly that same test scenario in the 011_generated (as
> > > the sub5 test). So, maybe this one here should be removed?
> >
> > For 0004_sync.pl, it is tested when 'include_generated_columns' is not
> > specified. Whereas for the test in 011_generated
> > 'include_generated_columns = true' is specified.
> > I thought we should have a test for both cases to test if the error
> > message format is the same for both cases. Thoughts?
>
> 3.
> Sorry, I missed that there was a parameter flag difference. Anyway,
> since the code-path to reach this error is the same regardless of the
> 'include_generated_columns' parameter value IMO having too many tests
> might be overkill. YMMV.
>
> Anyway, whether you decide to keep both test cases or not, I think all
> testing related to generated column replication belongs in the new
> 001_generated.pl TAP file -- not here in 04_sync.pl
I have removed the test

> ======
> src/test/subscription/t/011_generated.pl
>
> 4. Untested scenarios for "missing col"?
>
> I have seen (in 04_sync.pl) missing column test cases for:
> - publisher not-generated col ==> subscriber missing column
>
> Maybe I am mistaken, but I don't recall seeing any test cases for:
> - publisher generated-col ==> subscriber missing col
>
> Unless they are already done somewhere, I think this scenario should
> be in 011_generated.pl. Furthermore, maybe it needs to be tested for
> both include_generated_columns = true / false, because if the
> parameter is false it should be OK, but if the parameter is true it
> should give ERROR.
 Have added the tests in 011_generated.pl

I have also addressed the remaining comments. Please find the updated
v18 patches

v18-0001 - Rebased the patch on HEAD
v18-0002 - Addressed the comments
v18-0003 - Addressed the comments
v18-0004- Rebased the patch

Thanks and Regards,
Shlok Kyal

  87. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-16T08:58:43Z

    On Tue, 9 Jul 2024 at 09:53, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shlok, here are my review comments for v16-0003.
>
> ======
> src/backend/replication/logical/proto.c
>
>
> On Mon, Jul 8, 2024 at 10:04 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > On Mon, 8 Jul 2024 at 13:20, Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > >
> > > 2. logicalrep_write_tuple and logicalrep_write_attrs
> > >
> > > I thought all the code fragments like this:
> > >
> > > + if (att->attgenerated && att->attgenerated != ATTRIBUTE_GENERATED_STORED)
> > > + continue;
> > > +
> > >
> > > don't need to be in the code anymore, because of the BitMapSet (BMS)
> > > processing done to make the "column list" for publication where
> > > disallowed generated cols should already be excluded from the BMS,
> > > right?
> > >
> > > So shouldn't all these be detected by the following statement:
> > > if (!column_in_column_list(att->attnum, columns))
> > >   continue;
> > The current BMS logic do not handle the Virtual Generated Columns.
> > There can be cases where we do not want a virtual generated column but
> > it would be present in BMS.
> > To address this I have added the above logic. I have added this logic
> > similar to the checks of 'attr->attisdropped'.
> >
>
> Hmm. I thought the BMS idea of patch 0001 is to discover what columns
> should be replicated up-front. If they should not be replicated (e.g.
> virtual generated columns cannot be) then they should never be in the
> BMS.
>
> So what you said ("There can be cases where we do not want a virtual
> generated column but it would be present in BMS") should not be
> happening. If that is happening then it sounds more like a bug in the
> new BMS logic of pgoutput_column_list_init() function. In other words,
> if what you say is true, then it seems like the current extra
> conditions you have in patch 0004 are just a band-aid to cover a
> problem of the BMS logic of patch 0001. Am I mistaken?
>
We have created a 0004 patch to use the BMS approach. It will be
addressed in the future 0004 patch.

> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 4. send_relation_and_attrs
> > >
> > > (this is a similar comment for #2 above)
> > >
> > > IIUC of the advantages of the BitMapSet (BMS) idea in patch 0001 to
> > > process the generated columns up-front means there is no need to check
> > > them again in code like this.
> > >
> > > They should be discovered anyway in the subsequent check:
> > > /* Skip this attribute if it's not present in the column list */
> > > if (columns != NULL && !bms_is_member(att->attnum, columns))
> > >   continue;
> > Same explanation as above.
>
> As above.
>
We have created a 0004 patch to use the BMS approach. It will be
addressed in the future 0004 patch.

> ======
> src/test/subscription/t/011_generated.pl
>
> I'm not sure if you needed to say "STORED" generated cols for the
> subscriber-side columns but anyway, whatever is done needs to be done
> consistently. FYI, below you did *not* say STORED for subscriber-side
> generated cols, but in other comments for subscriber-side generated
> columns, you did say STORED.
>
> # tab3:
> # publisher-side tab3 has STORED generated col 'b' but
> # subscriber-side tab3 has DIFFERENT COMPUTATION generated col 'b'.
>
> ~
>
> # tab4:
> # publisher-side tab4 has STORED generated cols 'b' and 'c' but
> # subscriber-side tab4 has non-generated col 'b', and generated-col 'c'
> # where columns on publisher/subscriber are in a different order
>
Fixed

Please find the updated patch v18-0003 patch at [1].

[1]: https://www.postgresql.org/message-id/CANhcyEW3LVJpRPScz6VBa%3DZipEMV7b-u76PDEALNcNDFURCYMA%40mail.gmail.com

Thanks and Regards,
Shok Kyal

  88. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-16T09:04:12Z

    On Mon, 15 Jul 2024 at 08:08, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some review comments about patch v17-0003

I have addressed the comments in v18-0003 patch [1].

[1]: https://www.postgresql.org/message-id/CANhcyEW3LVJpRPScz6VBa%3DZipEMV7b-u76PDEALNcNDFURCYMA%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  89. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-16T11:19:42Z

    On Fri, Jul 12, 2024 at 12:13 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham.
>
> Thanks for separating the new BMS 'columns' modification.
>
> Here are my review comments for the latest patch v17-0001.
>
> ======
>
> 1. src/backend/replication/pgoutput/pgoutput.c
>
>   /*
>   * Columns included in the publication, or NULL if all columns are
>   * included implicitly.  Note that the attnums in this bitmap are not
> + * publication and include_generated_columns option: other reasons should
> + * be checked at user side.  Note that the attnums in this bitmap are not
>   * shifted by FirstLowInvalidHeapAttributeNumber.
>   */
>   Bitmapset  *columns;
> With this latest 0001 there is now no change to the original
> interpretation of RelationSyncEntry BMS 'columns'. So, I think this
> field comment should remain unchanged; i.e. it should be the same as
> the current HEAD comment.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> nitpick - comment changes for 'tab2' and 'tab3' to make them more consistent.
>
> ======
> 99.
> Please refer to the attached diff patch which implements any nitpicks
> described above.

The attached Patches contain all the suggested changes.

v19-0001 - Addressed the comments.
v19-0002 - Rebased the Patch.
v19-0003 - Rebased the Patch.
v19-0004- Addressed all the comments related to Bitmapset(BMS).

Thanks and Regards,
Shubham Khanna.

  90. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-16T11:23:45Z

    On Mon, Jul 15, 2024 at 11:09 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, I had a quick look at the patch v17-0004 which is the split-off
> new BMS logic.
>
> IIUC this 0004 is currently undergoing some refactoring and
> cleaning-up, so I won't comment much about it except to give the
> following observation below.
>
> ======
> src/backend/replication/logical/proto.c.
>
> I did not expect to see any code fragments that are still checking
> generated columns like below:
>
> logicalrep_write_tuple:
>
>   if (att->attgenerated)
>   {
> - if (!include_generated_columns)
> - continue;
>
>   if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
>   continue;
> ~
>
>   if (att->attgenerated)
>   {
> - if (!include_generated_columns)
> - continue;
>
>   if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
>   continue;
>
> ~~~
>
> logicalrep_write_attrs:
>
>   if (att->attgenerated)
>   {
> - if (!include_generated_columns)
> - continue;
>
>   if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
>   continue;
>
> ~
> if (att->attgenerated)
>   {
> - if (!include_generated_columns)
> - continue;
>
>   if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
>   continue;
> ~~~
>
>
> AFAIK, now checking support of generated columns will be done when the
> BMS 'columns' is assigned, so the continuation code will be handled
> like this:
>
> if (!column_in_column_list(att->attnum, columns))
>   continue;
>
> ======
>
> BTW there is a subtle but significant difference in this 0004 patch.
> IOW, we are introducing a difference between the list of published
> columns VERSUS a publication column list. So please make sure that all
> code comments are adjusted appropriately so they are not misleading by
> calling these "column lists" still.
>
> BEFORE: BMS 'columns'  means "columns of the column list" or NULL if
> there was no publication column list
> AFTER: BMS 'columns' means "columns to be replicated" or NULL if all
> columns are to be replicated

I have addressed all the comments in v19-0004 patch.
Please refer to the updated v19-0004 Patch here in [1]. See [1] for
the changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2BR0cj%3Dz1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  91. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-18T05:17:16Z

    Hi Shubham, here are my review comments for patch v19-0001.

======
src/backend/replication/pgoutput/pgoutput.c

1.
  /*
  * Columns included in the publication, or NULL if all columns are
  * included implicitly.  Note that the attnums in this bitmap are not
+ * publication and include_generated_columns option: other reasons should
+ * be checked at user side.  Note that the attnums in this bitmap are not
  * shifted by FirstLowInvalidHeapAttributeNumber.
  */
  Bitmapset  *columns;
You replied [1] "The attached Patches contain all the suggested
changes." but as I previously commented [2, #1], since there is no
change to the interpretation of the 'columns' BMS caused by this
patch, then I expected this comment would be unchanged (i.e. same as
HEAD code). But this fix was missed in v19-0001.

OTOH, if you do think there was a reason to change the comment then
the above is still not good because "are not publication and
include_generated_columns option" wording doesn't make sense.

======
src/test/subscription/t/011_generated.pl

Observation -- I added (in nitpicks diffs) some more comments for
'tab1' (to make all comments consistent with the new tests added). But
when I was doing that I observed that tab1 and tab3 test scenarios are
very similar. It seems only the subscription parameter is not
specified (so 'include_generated_cols' default wll be tested). IIRC
the default for that parameter is "false", so tab1 is not really
testing that properly -- e.g. I thought maybe to test the default
parameter it's better the subscriber-side 'b' should be not-generated?
But doing that would make 'tab1' the same as 'tab2'. Anyway, something
seems amiss -- it seems either something is not tested or is duplicate
tested. Please revisit what the tab1 test intention was and make sure
we are doing the right thing for it...

======
99.
The attached nitpicks diff patch has some tweaked comments.

======
[1] https://www.postgresql.org/message-id/CAHv8Rj%2BR0cj%3Dz1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPtVfrbx0jb42LCmS%3D-LcMTtWxm%2BvhaoArkjg7Z0mvuXbg%40mail.gmail.com


Kind Regards,
Peter Smith.
Fujitsu Australia.

  92. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-18T08:24:39Z

    Hi, here are some review comments for v19-0002

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:
nitpick - tweak function comment
nitpick - tweak other comments

~~~

fetch_remote_table_info:
nitpick - add space after "if"
nitpick - removed a comment because logic is self-evident from the variable name

======
src/test/subscription/t/004_sync.pl

1.
This new test is not related to generated columns. IIRC, this is just
some test that we discovered missing during review of this thread. As
such, I think this change can be posted/patched separately from this
thread.

======
src/test/subscription/t/011_generated.pl

nitpick - change some comment wording to be more consistent with patch 0001.

======
99.
Please see the nitpicks diff attachment which implements any nitpicks
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  93. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-18T23:29:02Z

    Hi, here are some review comments for patch v19-0003

======
src/backend/catalog/pg_publication.c

1.
/*
 * Translate a list of column names to an array of attribute numbers
 * and a Bitmapset with them; verify that each attribute is appropriate
 * to have in a publication column list (no system or generated attributes,
 * no duplicates).  Additional checks with replica identity are done later;
 * see pub_collist_contains_invalid_column.
 *
 * Note that the attribute numbers are *not* offset by
 * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
 * is okay.
 */
static void
publication_translate_columns(Relation targetrel, List *columns,
  int *natts, AttrNumber **attrs)

~

I though the above comment ought to change: /or generated
attributes/or virtual generated attributes/

IIRC this was already addressed back in v16, but somehow that fix has
been lost (???).

======
src/backend/replication/logical/tablesync.c

fetch_remote_table_info:
nitpick - missing end space in this comment /* TODO: use
ATTRIBUTE_GENERATED_VIRTUAL*/

======

2.
(in patch v19-0001)
+# tab3:
+# publisher-side tab3 has generated col 'b'.
+# subscriber-side tab3 has generated col 'b', using a different computation.

(here, in patch v19-0003)
 # tab3:
-# publisher-side tab3 has generated col 'b'.
-# subscriber-side tab3 has generated col 'b', using a different computation.
+# publisher-side tab3 has stored generated col 'b' but
+# subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.

It has become difficult to review these TAP tests, particularly when
different patches are modifying the same comment. e.g. I post
suggestions to modify comments for patch 0001. Those get addressed OK,
only to vanish in subsequent patches like has happened in the above
example.

Really this patch 0003 was only supposed to add the word "stored", not
revert the entire comment to something from an earlier version. Please
take care that all comment changes are carried forward correctly from
one patch to the next.

======
Kind Regards,
Peter Smith.
Fujitsu Australia.

  94. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-19T06:01:01Z

    On Thu, 18 Jul 2024 at 13:55, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some review comments for v19-0002
> ======
> src/test/subscription/t/004_sync.pl
>
> 1.
> This new test is not related to generated columns. IIRC, this is just
> some test that we discovered missing during review of this thread. As
> such, I think this change can be posted/patched separately from this
> thread.
>
I have removed the test for this thread.

I have also addressed the remaining comments for v19-0002 patch.
Please find the latest patches.

v20-0001 - not modified
v20-0002 - Addressed the comments
v20-0003 - Addressed the comments
v20-0004 - Not modified

Thanks and Regards,
Shlok Kyal

  95. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2024-07-19T06:03:51Z

    On Fri, 19 Jul 2024 at 04:59, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some review comments for patch v19-0003
>
> ======
> src/backend/catalog/pg_publication.c
>
> 1.
> /*
>  * Translate a list of column names to an array of attribute numbers
>  * and a Bitmapset with them; verify that each attribute is appropriate
>  * to have in a publication column list (no system or generated attributes,
>  * no duplicates).  Additional checks with replica identity are done later;
>  * see pub_collist_contains_invalid_column.
>  *
>  * Note that the attribute numbers are *not* offset by
>  * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
>  * is okay.
>  */
> static void
> publication_translate_columns(Relation targetrel, List *columns,
>   int *natts, AttrNumber **attrs)
>
> ~
>
> I though the above comment ought to change: /or generated
> attributes/or virtual generated attributes/
>
> IIRC this was already addressed back in v16, but somehow that fix has
> been lost (???).
Modified the comment

> ======
> src/backend/replication/logical/tablesync.c
>
> fetch_remote_table_info:
> nitpick - missing end space in this comment /* TODO: use
> ATTRIBUTE_GENERATED_VIRTUAL*/
>
Fixed

> ======
>
> 2.
> (in patch v19-0001)
> +# tab3:
> +# publisher-side tab3 has generated col 'b'.
> +# subscriber-side tab3 has generated col 'b', using a different computation.
>
> (here, in patch v19-0003)
>  # tab3:
> -# publisher-side tab3 has generated col 'b'.
> -# subscriber-side tab3 has generated col 'b', using a different computation.
> +# publisher-side tab3 has stored generated col 'b' but
> +# subscriber-side tab3 has DIFFERENT COMPUTATION stored generated col 'b'.
>
> It has become difficult to review these TAP tests, particularly when
> different patches are modifying the same comment. e.g. I post
> suggestions to modify comments for patch 0001. Those get addressed OK,
> only to vanish in subsequent patches like has happened in the above
> example.
>
> Really this patch 0003 was only supposed to add the word "stored", not
> revert the entire comment to something from an earlier version. Please
> take care that all comment changes are carried forward correctly from
> one patch to the next.
Fixed

I have addressed the comment in v20-0003 patch. Please refer [1].

[1]: https://www.postgresql.org/message-id/CANhcyEUzUurrX38HGvG30gV92YDz6WmnnwNRYMVY4tiga-8KZg%40mail.gmail.com

Thanks and Regards,
Shlok Kyal

  96. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-22T23:23:45Z

    On Fri, Jul 19, 2024 at 4:01 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> On Thu, 18 Jul 2024 at 13:55, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi, here are some review comments for v19-0002
> > ======
> > src/test/subscription/t/004_sync.pl
> >
> > 1.
> > This new test is not related to generated columns. IIRC, this is just
> > some test that we discovered missing during review of this thread. As
> > such, I think this change can be posted/patched separately from this
> > thread.
> >
> I have removed the test for this thread.
>
> I have also addressed the remaining comments for v19-0002 patch.

Hi, I have no more review comments for patch v20-0002 at this time.

I saw that the above test was removed from this thread as suggested,
but I could not find that any new thread was started to propose this
valuable missing test.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  97. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-26T12:23:46Z

    On Thu, Jul 18, 2024 at 10:47 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham, here are my review comments for patch v19-0001.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 1.
>   /*
>   * Columns included in the publication, or NULL if all columns are
>   * included implicitly.  Note that the attnums in this bitmap are not
> + * publication and include_generated_columns option: other reasons should
> + * be checked at user side.  Note that the attnums in this bitmap are not
>   * shifted by FirstLowInvalidHeapAttributeNumber.
>   */
>   Bitmapset  *columns;
> You replied [1] "The attached Patches contain all the suggested
> changes." but as I previously commented [2, #1], since there is no
> change to the interpretation of the 'columns' BMS caused by this
> patch, then I expected this comment would be unchanged (i.e. same as
> HEAD code). But this fix was missed in v19-0001.
>
> OTOH, if you do think there was a reason to change the comment then
> the above is still not good because "are not publication and
> include_generated_columns option" wording doesn't make sense.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> Observation -- I added (in nitpicks diffs) some more comments for
> 'tab1' (to make all comments consistent with the new tests added). But
> when I was doing that I observed that tab1 and tab3 test scenarios are
> very similar. It seems only the subscription parameter is not
> specified (so 'include_generated_cols' default wll be tested). IIRC
> the default for that parameter is "false", so tab1 is not really
> testing that properly -- e.g. I thought maybe to test the default
> parameter it's better the subscriber-side 'b' should be not-generated?
> But doing that would make 'tab1' the same as 'tab2'. Anyway, something
> seems amiss -- it seems either something is not tested or is duplicate
> tested. Please revisit what the tab1 test intention was and make sure
> we are doing the right thing for it...
>
> ======
> 99.
> The attached nitpicks diff patch has some tweaked comments.
>
> ======
> [1] https://www.postgresql.org/message-id/CAHv8Rj%2BR0cj%3Dz1bTMAgQKQWx1EKvkMEnV9QsHGvOqTdnLUQi1A%40mail.gmail.com
> [2] https://www.postgresql.org/message-id/CAHut%2BPtVfrbx0jb42LCmS%3D-LcMTtWxm%2BvhaoArkjg7Z0mvuXbg%40mail.gmail.com

The attached Patches contain all the suggested changes.

v21-0001 - Addressed the comments.
v21-0002 - Added the TAP Tests for 011_generated.pl file and modified
the patch accordingly.
v21-0003 - Added the TAP Tests for 011_generated.pl file and modified
the patch accordingly.
v21-0004- Rebased the Patch.

Thanks and Regards,
Shubham Khanna.

  98. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-07-29T07:26:46Z

    Thanks for the patch updates.

Here are my review comments for v21-0001.

I think this patch is mostly OK now except there are still some
comments about the TAP test.

======
Commit Message

0.
Using Create Subscription:
CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION
pub1 WITH (include_generated_columns = true, copy_data = false)"

If you are going to give an example, I think a gen-to-nogen example
would be a better choice. That's because the gen-to-gen (as you have
here) is not going to replicate anything due to the subscriber-side
column being generated.

======
src/test/subscription/t/011_generated.pl

1.
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
* 22) STORED, c int)"
+);

The subscriber2 node was intended only for all the tables where we
need include_generated_columns to be true. Mostly that is the
combination tests. (tab_gen_to_nogen, tab_nogen_to_gen, etc) OTOH,
table 'tab1' already existed. I don't think we need to bother
subscribing to tab1 from subscriber2 because every combination is
already covered by the combination tests. Let's leave this one alone.


~~~

2.
Huh? Where is the "tab_nogen_to_gen" combination test that I sent to
you off-list?

~~~

3.
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED,
a int, b int)"
+);

Maybe you can test 'tab_order' on both subscription nodes but I think
it is overkill. IMO it is enough to test it on subscription2.

~~~

4.
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a
* 22) STORED)"
+);

Ditto above. Maybe you can test 'tab_order' on both subscription nodes
but I think it is overkill. IMO it is enough to test it on
subscription2.

~~~

5.
Don't forget to add initial data for the missing nogen_to_gen table/test.

~~~

6.
 $node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR ALL TABLES");
+ "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen,
tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen, tab_order");
+
 $node_subscriber->safe_psql('postgres',
  "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
 );

It is not a bad idea to reduce the number of publications as you have
done, but IMO jamming all the tables into 1 publication is too much
because it makes it less understandable instead of simpler.

How about this:
- leave the 'pub1' just for 'tab1'.
- have a 'pub_combo' for publication all the gen_to_nogen,
nogen_to_gen etc combination tests.
- and a 'pub_misc' for any other misc tables like tab_order.

~~~

7.
+#####################
 # Wait for initial sync of all subscriptions
+#####################

I think you should write a note here that you have deliberately set
copy_data = false because COPY and include_generated_columns are not
allowed at the same time for patch 0001. And that is why all expected
results on subscriber2 will be empty. Also, say this limitation will
be changed in patch 0002.

~~~

(I didn't yet check 011_generated.pl file results beyond this point...
I'll wait for v22-0001 to review further)

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  99. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-07-31T08:42:01Z

    On Mon, Jul 29, 2024 at 12:57 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Thanks for the patch updates.
>
> Here are my review comments for v21-0001.
>
> I think this patch is mostly OK now except there are still some
> comments about the TAP test.
>
> ======
> Commit Message
>
> 0.
> Using Create Subscription:
> CREATE SUBSCRIPTION sub2_gen_to_gen CONNECTION '$publisher_connstr' PUBLICATION
> pub1 WITH (include_generated_columns = true, copy_data = false)"
>
> If you are going to give an example, I think a gen-to-nogen example
> would be a better choice. That's because the gen-to-gen (as you have
> here) is not going to replicate anything due to the subscriber-side
> column being generated.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> 1.
> +$node_subscriber2->safe_psql('postgres',
> + "CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a
> * 22) STORED, c int)"
> +);
>
> The subscriber2 node was intended only for all the tables where we
> need include_generated_columns to be true. Mostly that is the
> combination tests. (tab_gen_to_nogen, tab_nogen_to_gen, etc) OTOH,
> table 'tab1' already existed. I don't think we need to bother
> subscribing to tab1 from subscriber2 because every combination is
> already covered by the combination tests. Let's leave this one alone.
>
>
> ~~~
>
> 2.
> Huh? Where is the "tab_nogen_to_gen" combination test that I sent to
> you off-list?
>
> ~~~
>
> 3.
> +$node_subscriber->safe_psql('postgres',
> + "CREATE TABLE tab_order (c int GENERATED ALWAYS AS (a * 22) STORED,
> a int, b int)"
> +);
>
> Maybe you can test 'tab_order' on both subscription nodes but I think
> it is overkill. IMO it is enough to test it on subscription2.
>
> ~~~
>
> 4.
> +$node_subscriber->safe_psql('postgres',
> + "CREATE TABLE tab_alter (a int, b int, c int GENERATED ALWAYS AS (a
> * 22) STORED)"
> +);
>
> Ditto above. Maybe you can test 'tab_order' on both subscription nodes
> but I think it is overkill. IMO it is enough to test it on
> subscription2.
>
> ~~~
>
> 5.
> Don't forget to add initial data for the missing nogen_to_gen table/test.
>
> ~~~
>
> 6.
>  $node_publisher->safe_psql('postgres',
> - "CREATE PUBLICATION pub1 FOR ALL TABLES");
> + "CREATE PUBLICATION pub1 FOR TABLE tab1, tab_gen_to_gen,
> tab_gen_to_nogen, tab_gen_to_missing, tab_missing_to_gen, tab_order");
> +
>  $node_subscriber->safe_psql('postgres',
>   "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
>  );
>
> It is not a bad idea to reduce the number of publications as you have
> done, but IMO jamming all the tables into 1 publication is too much
> because it makes it less understandable instead of simpler.
>
> How about this:
> - leave the 'pub1' just for 'tab1'.
> - have a 'pub_combo' for publication all the gen_to_nogen,
> nogen_to_gen etc combination tests.
> - and a 'pub_misc' for any other misc tables like tab_order.
>
> ~~~
>
> 7.
> +#####################
>  # Wait for initial sync of all subscriptions
> +#####################
>
> I think you should write a note here that you have deliberately set
> copy_data = false because COPY and include_generated_columns are not
> allowed at the same time for patch 0001. And that is why all expected
> results on subscriber2 will be empty. Also, say this limitation will
> be changed in patch 0002.
>
> ~~~
>
> (I didn't yet check 011_generated.pl file results beyond this point...
> I'll wait for v22-0001 to review further)

The attached Patches contain all the suggested changes.

v22-0001 - Addressed the comments.
v22-0002 - Rebased the Patch.
v22-0003 - Rebased the Patch.
v22-0004 - Rebased the Patch.

Thanks and Regards,
Shubham Khanna.

  100. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-01T08:32:21Z

    Hi, Here are my review comments for patch v22-0001

All comments now are only for the TAP test.

======
src/test/subscription/t/011_generated.pl

1. I added all new code for the missing combination test case
"gen-to-missing". See nitpicks diff.
- create a separate publication for this "tab_gen_to_missing" table
because the test gives subscription errors.
- for the initial data
- for the replicated data

~~~

2. I added sub1 and sub2 subscriptions for every combo test
(previously some were absent). See nitpicks diff.

~~~

3. There was a missing test case for nogen-to-gen combination, and
after experimenting with this I am getting a bit suspicious,

Currently, it seems that if a COPY is attempted then the error would
be like this:
2024-08-01 17:16:45.110 AEST [18942] ERROR:  column "b" is a generated column
2024-08-01 17:16:45.110 AEST [18942] DETAIL:  Generated columns cannot
be used in COPY.

OTOH, if a COPY is not attempted (e.g. copy_data = false) then patch
0001 allows replication to happen. And the generated value of the
subscriber "b" takes precedence.

I have included these tests in the nitpicks diff of patch 0001.

Those results weren't exactly what I was expecting.  That is why it is
so important to include *every* test combination in these TAP tests --
because unless we know how it works today, we won't know if we are
accidentally breaking the current behaviour with the other (0002,
0003) patches.

Please experiment in patches 0001 and 0002 using tab_nogen_to_gen more
to make sure the (new?) patch errors make sense and don't overstep by
giving ERRORs when they should not.

~~~~

Also, many other smaller issues/changes were done:

~~~

Creating tables:

nitpick - rearranged to keep all combo test SQLs in a consistent order
throughout this file
1/ gen-to-gen
2/ gen-to-nogen
3/ gen-to-missing
4/ missing-to-gen
5/ nogen-to-gen

nitpick - fixed the wrong comment for CREATE TABLE tab_nogen_to_gen.

nitpick - tweaked some CREATE TABLE comments for consistency.

nitpick - in the v22 patch many of the generated col 'b' use different
computations for every test. It makes it unnecessarily difficult to
read/review the expected results. So, I've made them all the same. Now
computation is "a * 2" on the publisher side, and "a * 22" on the
subscriber side.

~~~

Creating Publications and Subscriptions:


nitpick - added comment for all the CREATE PUBLICATION

nitpick - added comment for all the CREATE SUBSCRIPTION

nitpick - I moved the note about copy_data = false to where all the
node_subscriber2 subscriptions are created. Also, don't explicitly
refer to "patch 000" in the comment, because that will not make any
sense after getting pushed.

nitpick - I changed many subscriber names to consistently use "sub1"
or "sub2" within the name (this is the visual cue of which
node_subscriber<n> they are on). e.g.
/regress_sub_combo2/regress_sub2_combo/

~~~

Initial Sync tests:

nitpick - not sure if it is possible to do the initial data tests for
"nogen_to_gen" in the normal place. For now, it is just replaced by a
comment.
NOTE - Maybe this should be refactored later to put all the initial
data checks in one place. I'll think about this point more in the next
review.

~~~

nitpick - Changed cleanup I drop subscriptions before publications.

nitpick - remove the unnecessary blank line at the end.

======

Please see the attached diffs patch (apply it atop patch 0001) which
includes all the nipick changes mentioned above.

~~

BTW, For a quicker turnaround and less churning please consider just
posting the v23-0001 by itself instead of waiting to rebase all the
subsequent patches. When 0001 settles down some more then rebase the
others.

~~

Also, please run the indentation tool over this code ASAP.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  101. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-02T04:26:59Z

    On Thu, Aug 1, 2024 at 2:02 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, Here are my review comments for patch v22-0001
>
> All comments now are only for the TAP test.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> 1. I added all new code for the missing combination test case
> "gen-to-missing". See nitpicks diff.
> - create a separate publication for this "tab_gen_to_missing" table
> because the test gives subscription errors.
> - for the initial data
> - for the replicated data
>
> ~~~
>
> 2. I added sub1 and sub2 subscriptions for every combo test
> (previously some were absent). See nitpicks diff.
>
> ~~~
>
> 3. There was a missing test case for nogen-to-gen combination, and
> after experimenting with this I am getting a bit suspicious,
>
> Currently, it seems that if a COPY is attempted then the error would
> be like this:
> 2024-08-01 17:16:45.110 AEST [18942] ERROR:  column "b" is a generated column
> 2024-08-01 17:16:45.110 AEST [18942] DETAIL:  Generated columns cannot
> be used in COPY.
>
> OTOH, if a COPY is not attempted (e.g. copy_data = false) then patch
> 0001 allows replication to happen. And the generated value of the
> subscriber "b" takes precedence.
>
> I have included these tests in the nitpicks diff of patch 0001.
>
> Those results weren't exactly what I was expecting.  That is why it is
> so important to include *every* test combination in these TAP tests --
> because unless we know how it works today, we won't know if we are
> accidentally breaking the current behaviour with the other (0002,
> 0003) patches.
>
> Please experiment in patches 0001 and 0002 using tab_nogen_to_gen more
> to make sure the (new?) patch errors make sense and don't overstep by
> giving ERRORs when they should not.
>
> ~~~~
>
> Also, many other smaller issues/changes were done:
>
> ~~~
>
> Creating tables:
>
> nitpick - rearranged to keep all combo test SQLs in a consistent order
> throughout this file
> 1/ gen-to-gen
> 2/ gen-to-nogen
> 3/ gen-to-missing
> 4/ missing-to-gen
> 5/ nogen-to-gen
>
> nitpick - fixed the wrong comment for CREATE TABLE tab_nogen_to_gen.
>
> nitpick - tweaked some CREATE TABLE comments for consistency.
>
> nitpick - in the v22 patch many of the generated col 'b' use different
> computations for every test. It makes it unnecessarily difficult to
> read/review the expected results. So, I've made them all the same. Now
> computation is "a * 2" on the publisher side, and "a * 22" on the
> subscriber side.
>
> ~~~
>
> Creating Publications and Subscriptions:
>
>
> nitpick - added comment for all the CREATE PUBLICATION
>
> nitpick - added comment for all the CREATE SUBSCRIPTION
>
> nitpick - I moved the note about copy_data = false to where all the
> node_subscriber2 subscriptions are created. Also, don't explicitly
> refer to "patch 000" in the comment, because that will not make any
> sense after getting pushed.
>
> nitpick - I changed many subscriber names to consistently use "sub1"
> or "sub2" within the name (this is the visual cue of which
> node_subscriber<n> they are on). e.g.
> /regress_sub_combo2/regress_sub2_combo/
>
> ~~~
>
> Initial Sync tests:
>
> nitpick - not sure if it is possible to do the initial data tests for
> "nogen_to_gen" in the normal place. For now, it is just replaced by a
> comment.
> NOTE - Maybe this should be refactored later to put all the initial
> data checks in one place. I'll think about this point more in the next
> review.
>
> ~~~
>
> nitpick - Changed cleanup I drop subscriptions before publications.
>
> nitpick - remove the unnecessary blank line at the end.
>
> ======
>
> Please see the attached diffs patch (apply it atop patch 0001) which
> includes all the nipick changes mentioned above.
>
> ~~
>
> BTW, For a quicker turnaround and less churning please consider just
> posting the v23-0001 by itself instead of waiting to rebase all the
> subsequent patches. When 0001 settles down some more then rebase the
> others.
>
> ~~
>
> Also, please run the indentation tool over this code ASAP.
>
I have fixed all the comments. The attached Patch(v23-0001) contains
all the changes.

Thanks and Regards,
Shubham Khanna.

  102. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-05T02:40:01Z

    Hi Shubhab.

Here are some more review comments for the v23-0001.

======
011_generated.pl b/src/test/subscription/t/011_generated.pl

nitpick - renamed /regress_pub/regress_pub_tab1/ and
/regress_sub1/regress_sub1_tab1/
nitpick - typo /inital data /initial data/
nitpick - typo /snode_subscriber2/node_subscriber2/
nitpick - tweak the combo initial sync comments and messages
nitpick - /#Cleanup/# cleanup/
nitpick - tweak all the combo normal replication comments
nitpick - removed blank line at the end

~~~

1. Refactor tab_gen_to_missing initial sync tests.

I moved the tab_gen_to_missing initial sync for node_subscriber2 to be
back where all the other initial sync tests are done.
See the nitpicks patch file.

~~~

2. Refactor tab_nogen_to_gen initial sync tests

I moved all the tab_nogen_to_gen initial sync tests back to where the
other initial sync tests are done.
See the nitpicks patch file.

~~~

3. Added another test case:

Because the (current PG17) nogen-to-gen initial sync test case (with
copy_data=true) gives an ERROR, I have added another combination to
cover normal replication (e.g. using copy_data=false).
See the nitpicks patch file.

(This has exposed an inconsistency which IMO might be a PG17 bug. I
have included TAP test comments about this, and plan to post a
separate thread for it later).

~

4. GUC

Moving and adding more CREATE SUBSCRIPTION exceeded some default GUCs,
so extra configuration was needed.
See the nitpick patch file.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  103. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-05T03:44:41Z

    Hi,

Writing many new test case combinations has exposed a possible bug in
patch 0001.

In my previous post [1] there was questionable behaviour when
replicating from a normal (not generated) column on the publisher side
to a generated column on the subscriber side. Initially, I thought the
test might have exposed a possible PG17 bug, but now I think it has
really found a bug in patch 0001.

~~~

Previously (PG17) this would fail consistently both during COPY and
during normal replication.Now, patch 0001 has changed this behaviour
-- it is not always failing anymore.

The patch should not be impacting this existing behaviour. It only
introduces a new 'include_generated_columns', but since the publisher
side is not a generated column I do not expect there should be any
difference in behaviour for this test case. IMO the TAP test expected
results should be corrected for this scenario. And fix the bug.

Below is an example demonstrating PG17 behaviour.

======


Publisher:
----------

(notice column "b" is not generated)

test_pub=# CREATE TABLE tab_nogen_to_gen (a int, b int);
CREATE TABLE
test_pub=# INSERT INTO tab_nogen_to_gen VALUES (1,101),(2,102);
INSERT 0 2
test_pub=# CREATE PUBLICATION pub1 for TABLE tab_nogen_to_gen;
CREATE PUBLICATION
test_pub=#

Subscriber:
-----------

(notice corresponding column "b" is generated)

test_sub=# CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED
ALWAYS AS (a * 22) STORED);
CREATE TABLE
test_sub=#

Try to create a subscription. Notice we get the error: ERROR:  logical
replication target relation "public.tab_nogen_to_gen" is missing
replicated column: "b"

test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
PUBLICATION pub1;
2024-08-05 13:16:40.043 AEST [20957] WARNING:  subscriptions created
by regression test cases should have names starting with "regress_"
WARNING:  subscriptions created by regression test cases should have
names starting with "regress_"
NOTICE:  created replication slot "sub1" on publisher
CREATE SUBSCRIPTION
test_sub=# 2024-08-05 13:16:40.105 AEST [29258] LOG:  logical
replication apply worker for subscription "sub1" has started
2024-08-05 13:16:40.117 AEST [29260] LOG:  logical replication table
synchronization worker for subscription "sub1", table
"tab_nogen_to_gen" has started
2024-08-05 13:16:40.172 AEST [29260] ERROR:  logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:16:40.173 AEST [20039] LOG:  background worker "logical
replication tablesync worker" (PID 29260) exited with exit code 1
2024-08-05 13:16:45.187 AEST [29400] LOG:  logical replication table
synchronization worker for subscription "sub1", table
"tab_nogen_to_gen" has started
2024-08-05 13:16:45.285 AEST [29400] ERROR:  logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:16:45.286 AEST [20039] LOG:  background worker "logical
replication tablesync worker" (PID 29400) exited with exit code 1
...

Create the subscription again, but this time with copy_data = false

test_sub=# CREATE SUBSCRIPTION sub1_nocopy CONNECTION
'dbname=test_pub' PUBLICATION pub1 WITH (copy_data = false);
2024-08-05 13:22:57.719 AEST [20957] WARNING:  subscriptions created
by regression test cases should have names starting with "regress_"
WARNING:  subscriptions created by regression test cases should have
names starting with "regress_"
NOTICE:  created replication slot "sub1_nocopy" on publisher
CREATE SUBSCRIPTION
test_sub=# 2024-08-05 13:22:57.765 AEST [7012] LOG:  logical
replication apply worker for subscription "sub1_nocopy" has started

test_sub=#

~~~

Then insert data from the publisher to see what happens for normal replication.

test_pub=#
test_pub=# INSERT INTO tab_nogen_to_gen VALUES (3,103),(4,104);
INSERT 0 2

~~~

Notice the subscriber gets the same error as before: ERROR:  logical
replication target relation "public.tab_nogen_to_gen" is missing
replicated column: "b"

2024-08-05 13:25:14.897 AEST [20039] LOG:  background worker "logical
replication apply worker" (PID 10957) exited with exit code 1
2024-08-05 13:25:19.933 AEST [11095] LOG:  logical replication apply
worker for subscription "sub1_nocopy" has started
2024-08-05 13:25:19.966 AEST [11095] ERROR:  logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:25:19.966 AEST [11095] CONTEXT:  processing remote data
for replication origin "pg_16390" during message type "INSERT" in
transaction 742, finished at 0/1967BB0
2024-08-05 13:25:19.968 AEST [20039] LOG:  background worker "logical
replication apply worker" (PID 11095) exited with exit code 1
2024-08-05 13:25:24.917 AEST [11225] LOG:  logical replication apply
worker for subscription "sub1_nocopy" has started
2024-08-05 13:25:24.926 AEST [11225] ERROR:  logical replication
target relation "public.tab_nogen_to_gen" is missing replicated
column: "b"
2024-08-05 13:25:24.926 AEST [11225] CONTEXT:  processing remote data
for replication origin "pg_16390" during message type "INSERT" in
transaction 742, finished at 0/1967BB0
2024-08-05 13:25:24.927 AEST [20039] LOG:  background worker "logical
replication apply worker" (PID 11225) exited with exit code 1
...

======
[1] https://www.postgresql.org/message-id/CAHut%2BPvtT8fKOfvVYr4vANx_fr92vedas%2BZRbQxvMC097rks6w%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  104. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-07T05:26:16Z

    On Mon, Aug 5, 2024 at 8:10 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubhab.
>
> Here are some more review comments for the v23-0001.
>
> ======
> 011_generated.pl b/src/test/subscription/t/011_generated.pl
>
> nitpick - renamed /regress_pub/regress_pub_tab1/ and
> /regress_sub1/regress_sub1_tab1/
> nitpick - typo /inital data /initial data/
> nitpick - typo /snode_subscriber2/node_subscriber2/
> nitpick - tweak the combo initial sync comments and messages
> nitpick - /#Cleanup/# cleanup/
> nitpick - tweak all the combo normal replication comments
> nitpick - removed blank line at the end
>
> ~~~
>
> 1. Refactor tab_gen_to_missing initial sync tests.
>
> I moved the tab_gen_to_missing initial sync for node_subscriber2 to be
> back where all the other initial sync tests are done.
> See the nitpicks patch file.
>
> ~~~
>
> 2. Refactor tab_nogen_to_gen initial sync tests
>
> I moved all the tab_nogen_to_gen initial sync tests back to where the
> other initial sync tests are done.
> See the nitpicks patch file.
>
> ~~~
>
> 3. Added another test case:
>
> Because the (current PG17) nogen-to-gen initial sync test case (with
> copy_data=true) gives an ERROR, I have added another combination to
> cover normal replication (e.g. using copy_data=false).
> See the nitpicks patch file.
>
> (This has exposed an inconsistency which IMO might be a PG17 bug. I
> have included TAP test comments about this, and plan to post a
> separate thread for it later).
>
> ~
>
> 4. GUC
>
> Moving and adding more CREATE SUBSCRIPTION exceeded some default GUCs,
> so extra configuration was needed.
> See the nitpick patch file.
>

I have fixed all the comments. The attached Patch(v24-0001) contains
all the changes.

Thanks and Regards,
Shubham Khanna.

  105. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-07T05:37:20Z

    On Mon, Aug 5, 2024 at 9:15 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> Writing many new test case combinations has exposed a possible bug in
> patch 0001.
>
> In my previous post [1] there was questionable behaviour when
> replicating from a normal (not generated) column on the publisher side
> to a generated column on the subscriber side. Initially, I thought the
> test might have exposed a possible PG17 bug, but now I think it has
> really found a bug in patch 0001.
>
> ~~~
>
> Previously (PG17) this would fail consistently both during COPY and
> during normal replication.Now, patch 0001 has changed this behaviour
> -- it is not always failing anymore.
>
> The patch should not be impacting this existing behaviour. It only
> introduces a new 'include_generated_columns', but since the publisher
> side is not a generated column I do not expect there should be any
> difference in behaviour for this test case. IMO the TAP test expected
> results should be corrected for this scenario. And fix the bug.
>
> Below is an example demonstrating PG17 behaviour.
>
> ======
>
>
> Publisher:
> ----------
>
> (notice column "b" is not generated)
>
> test_pub=# CREATE TABLE tab_nogen_to_gen (a int, b int);
> CREATE TABLE
> test_pub=# INSERT INTO tab_nogen_to_gen VALUES (1,101),(2,102);
> INSERT 0 2
> test_pub=# CREATE PUBLICATION pub1 for TABLE tab_nogen_to_gen;
> CREATE PUBLICATION
> test_pub=#
>
> Subscriber:
> -----------
>
> (notice corresponding column "b" is generated)
>
> test_sub=# CREATE TABLE tab_nogen_to_gen (a int, b int GENERATED
> ALWAYS AS (a * 22) STORED);
> CREATE TABLE
> test_sub=#
>
> Try to create a subscription. Notice we get the error: ERROR:  logical
> replication target relation "public.tab_nogen_to_gen" is missing
> replicated column: "b"
>
> test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
> PUBLICATION pub1;
> 2024-08-05 13:16:40.043 AEST [20957] WARNING:  subscriptions created
> by regression test cases should have names starting with "regress_"
> WARNING:  subscriptions created by regression test cases should have
> names starting with "regress_"
> NOTICE:  created replication slot "sub1" on publisher
> CREATE SUBSCRIPTION
> test_sub=# 2024-08-05 13:16:40.105 AEST [29258] LOG:  logical
> replication apply worker for subscription "sub1" has started
> 2024-08-05 13:16:40.117 AEST [29260] LOG:  logical replication table
> synchronization worker for subscription "sub1", table
> "tab_nogen_to_gen" has started
> 2024-08-05 13:16:40.172 AEST [29260] ERROR:  logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:16:40.173 AEST [20039] LOG:  background worker "logical
> replication tablesync worker" (PID 29260) exited with exit code 1
> 2024-08-05 13:16:45.187 AEST [29400] LOG:  logical replication table
> synchronization worker for subscription "sub1", table
> "tab_nogen_to_gen" has started
> 2024-08-05 13:16:45.285 AEST [29400] ERROR:  logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:16:45.286 AEST [20039] LOG:  background worker "logical
> replication tablesync worker" (PID 29400) exited with exit code 1
> ...
>
> Create the subscription again, but this time with copy_data = false
>
> test_sub=# CREATE SUBSCRIPTION sub1_nocopy CONNECTION
> 'dbname=test_pub' PUBLICATION pub1 WITH (copy_data = false);
> 2024-08-05 13:22:57.719 AEST [20957] WARNING:  subscriptions created
> by regression test cases should have names starting with "regress_"
> WARNING:  subscriptions created by regression test cases should have
> names starting with "regress_"
> NOTICE:  created replication slot "sub1_nocopy" on publisher
> CREATE SUBSCRIPTION
> test_sub=# 2024-08-05 13:22:57.765 AEST [7012] LOG:  logical
> replication apply worker for subscription "sub1_nocopy" has started
>
> test_sub=#
>
> ~~~
>
> Then insert data from the publisher to see what happens for normal replication.
>
> test_pub=#
> test_pub=# INSERT INTO tab_nogen_to_gen VALUES (3,103),(4,104);
> INSERT 0 2
>
> ~~~
>
> Notice the subscriber gets the same error as before: ERROR:  logical
> replication target relation "public.tab_nogen_to_gen" is missing
> replicated column: "b"
>
> 2024-08-05 13:25:14.897 AEST [20039] LOG:  background worker "logical
> replication apply worker" (PID 10957) exited with exit code 1
> 2024-08-05 13:25:19.933 AEST [11095] LOG:  logical replication apply
> worker for subscription "sub1_nocopy" has started
> 2024-08-05 13:25:19.966 AEST [11095] ERROR:  logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:25:19.966 AEST [11095] CONTEXT:  processing remote data
> for replication origin "pg_16390" during message type "INSERT" in
> transaction 742, finished at 0/1967BB0
> 2024-08-05 13:25:19.968 AEST [20039] LOG:  background worker "logical
> replication apply worker" (PID 11095) exited with exit code 1
> 2024-08-05 13:25:24.917 AEST [11225] LOG:  logical replication apply
> worker for subscription "sub1_nocopy" has started
> 2024-08-05 13:25:24.926 AEST [11225] ERROR:  logical replication
> target relation "public.tab_nogen_to_gen" is missing replicated
> column: "b"
> 2024-08-05 13:25:24.926 AEST [11225] CONTEXT:  processing remote data
> for replication origin "pg_16390" during message type "INSERT" in
> transaction 742, finished at 0/1967BB0
> 2024-08-05 13:25:24.927 AEST [20039] LOG:  background worker "logical
> replication apply worker" (PID 11225) exited with exit code 1
>
This is an expected behaviour. The error message here is improvised.
This error is consistent and it is being handled in the 0002 patch.
Below are the logs for the same:
2024-08-07 10:47:45.977 IST [29756] LOG:  logical replication table
synchronization worker for subscription "sub1", table
"tab_nogen_to_gen" has started
2024-08-07 10:47:46.116 IST [29756] ERROR:  logical replication target
relation "public.tab_nogen_to_gen" has a generated column "b" but
corresponding column on source relation is not a generated column
0002 Patch needs to be applied to get rid of this error.

Thanks and Regards,
Shubham Khanna.

  106. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-07T08:00:37Z

    Hi Shubham,

Here are my review comments for patch v24-0001

I think the TAP tests have incorrect expected results for the nogen-to-gen case.

Whereas the HEAD code will cause "ERROR" for this test scenario, patch
0001 does not. IMO the behaviour should be unchanged for this scenario
which has no generated column on the publisher side. So it seems this
is a bug in patch 0001.

FYI, I have included "FIXME" comments in the attached top-up diff
patch to show which test cases I think are expecting wrong results.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  107. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-08T05:23:07Z

    On Wed, Aug 7, 2024 at 1:31 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham,
>
> Here are my review comments for patch v24-0001
>
> I think the TAP tests have incorrect expected results for the nogen-to-gen case.
>
> Whereas the HEAD code will cause "ERROR" for this test scenario, patch
> 0001 does not. IMO the behaviour should be unchanged for this scenario
> which has no generated column on the publisher side. So it seems this
> is a bug in patch 0001.
>
> FYI, I have included "FIXME" comments in the attached top-up diff
> patch to show which test cases I think are expecting wrong results.
>

Fixed all the comments. The attached Patch(v25-0001) contains all the changes.

Thanks and Regards,
Shubham Khanna.

  108. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-08T07:13:15Z

    Hi Shubham,

I think the v25-0001 patch only half-fixes the problems reported in my
v24-0001 review.

~

Background (from the commit message):
This commit enables support for the 'include_generated_columns' option
in logical replication, allowing the transmission of generated column
information and data alongside regular table changes.

~

The broken TAP test scenario in question is replicating from a
"not-generated" column to a "generated" column. As the generated
column is not on the publishing side, IMO the
'include_generated_columns' option should have zero effect here.

In other words, I expect this TAP test for 'include_generated_columns
= true' case should also be failing, as I wrote already yesterday:

+# FIXME
+# Since there is no generated column on the publishing side this should give
+# the same result as the previous test. -- e.g. something like:
+# ERROR:  logical replication target relation
"public.tab_nogen_to_gen" is missing
+# replicated column: "b"

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  109. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-08-10T14:23:04Z

    On Thu, 8 Aug 2024 at 10:53, Shubham Khanna <khannashubham1197@gmail.com> wrote:
>
> On Wed, Aug 7, 2024 at 1:31 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham,
> >
> > Here are my review comments for patch v24-0001
> >
> > I think the TAP tests have incorrect expected results for the nogen-to-gen case.
> >
> > Whereas the HEAD code will cause "ERROR" for this test scenario, patch
> > 0001 does not. IMO the behaviour should be unchanged for this scenario
> > which has no generated column on the publisher side. So it seems this
> > is a bug in patch 0001.
> >
> > FYI, I have included "FIXME" comments in the attached top-up diff
> > patch to show which test cases I think are expecting wrong results.
> >
>
> Fixed all the comments. The attached Patch(v25-0001) contains all the changes.

Few comments:
1) Can we add one test with replica identity full to show that
generated column is included in case of update operation with
test_decoding.

2) At the end of the file generated_columns.sql a newline is missing:
+-- when 'include-generated-columns' = '0' the generated column 'b'
values will not be replicated
+INSERT INTO gencoltable (a) VALUES (7), (8), (9);
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
'include-generated-columns', '0');
+
+DROP TABLE gencoltable;
+
+SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
\ No newline at end of file

3)
3.a)This can be changed:
+-- when 'include-generated-columns' is not set the generated column
'b' values will be replicated
+INSERT INTO gencoltable (a) VALUES (1), (2), (3);

to:
-- By default, 'include-generated-columns' is enabled, so the values
for the generated column 'b' will be replicated even if it is not
explicitly specified.

3.b) This can be changed:
-- when 'include-generated-columns' = '1' the generated column 'b'
values will be replicated
to:
-- when 'include-generated-columns' is enabled, the values of the
generated column 'b' will be replicated.

3.c) This can be changed:
-- when 'include-generated-columns' = '0' the generated column 'b'
values will not be replicated
to:
-- when 'include-generated-columns' is disabled, the values of the
generated column 'b' will not be replicated.

4) I did not see any test for dump, can we add one test for this.

Regards,
Vignesh

  110. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-15T07:39:30Z

    On Tue, Jul 23, 2024 at 9:23 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Jul 19, 2024 at 4:01 PM Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
> >
> > On Thu, 18 Jul 2024 at 13:55, Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi, here are some review comments for v19-0002
> > > ======
> > > src/test/subscription/t/004_sync.pl
> > >
> > > 1.
> > > This new test is not related to generated columns. IIRC, this is just
> > > some test that we discovered missing during review of this thread. As
> > > such, I think this change can be posted/patched separately from this
> > > thread.
> > >
> > I have removed the test for this thread.
> >
> > I have also addressed the remaining comments for v19-0002 patch.
>
> Hi, I have no more review comments for patch v20-0002 at this time.
>
> I saw that the above test was removed from this thread as suggested,
> but I could not find that any new thread was started to propose this
> valuable missing test.
>

I still did not find any new thread for adding the missing test case,
so I started one myself [1].

======
[1] https://www.postgresql.org/message-id/CAHut+PtX8P0EGhsk9p=hQGUHrzxeCSzANXSMKOvYiLX-EjdyNw@mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  111. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-16T04:34:46Z

    On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham,
>
> I think the v25-0001 patch only half-fixes the problems reported in my
> v24-0001 review.
>
> ~
>
> Background (from the commit message):
> This commit enables support for the 'include_generated_columns' option
> in logical replication, allowing the transmission of generated column
> information and data alongside regular table changes.
>
> ~
>
> The broken TAP test scenario in question is replicating from a
> "not-generated" column to a "generated" column. As the generated
> column is not on the publishing side, IMO the
> 'include_generated_columns' option should have zero effect here.
>
> In other words, I expect this TAP test for 'include_generated_columns
> = true' case should also be failing, as I wrote already yesterday:
>
> +# FIXME
> +# Since there is no generated column on the publishing side this should give
> +# the same result as the previous test. -- e.g. something like:
> +# ERROR:  logical replication target relation
> "public.tab_nogen_to_gen" is missing
> +# replicated column: "b"

I have fixed the given comments. The attached v26-0001 Patch contains
the required changes.

Thanks and Regards,
Shubham Khanna.

  112. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-08-16T09:17:39Z

    On Fri, 16 Aug 2024 at 10:04, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham,
> >
> > I think the v25-0001 patch only half-fixes the problems reported in my
> > v24-0001 review.
> >
> > ~
> >
> > Background (from the commit message):
> > This commit enables support for the 'include_generated_columns' option
> > in logical replication, allowing the transmission of generated column
> > information and data alongside regular table changes.
> >
> > ~
> >
> > The broken TAP test scenario in question is replicating from a
> > "not-generated" column to a "generated" column. As the generated
> > column is not on the publishing side, IMO the
> > 'include_generated_columns' option should have zero effect here.
> >
> > In other words, I expect this TAP test for 'include_generated_columns
> > = true' case should also be failing, as I wrote already yesterday:
> >
> > +# FIXME
> > +# Since there is no generated column on the publishing side this should give
> > +# the same result as the previous test. -- e.g. something like:
> > +# ERROR:  logical replication target relation
> > "public.tab_nogen_to_gen" is missing
> > +# replicated column: "b"
>
> I have fixed the given comments. The attached v26-0001 Patch contains
> the required changes.

Few comments:
1) There's no need to pass include_generated_columns in this case; we
can retrieve it from ctx->data instead:
@@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
 static void
 send_relation_and_attrs(Relation relation, TransactionId xid,
                                                LogicalDecodingContext *ctx,
-                                               Bitmapset *columns)
+                                               Bitmapset *columns,
bool include_generated_columns)
 {
        TupleDesc       desc = RelationGetDescr(relation);
        int                     i;
@@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation,
TransactionId xid,

2) Commit message:
If the subscriber-side column is also a generated column then this option
has no effect; the replicated data will be ignored and the subscriber
column will be filled as normal with the subscriber-side computed or
default data.

An error will occur in this case, so the message should be updated accordingly.

3) The current test is structured as follows: a) Create all required
tables b) Insert data into tables c) Create publications d) Create
subscriptions e) Perform inserts and verify
This approach can make reviewing and maintenance somewhat challenging.

Instead, could you modify it to: a) Create the required table for a
single test b) Insert data for this test c) Create the publication for
this test d) Create the subscriptions for this test e) Perform inserts
and verify f) Clean up

4) We can maintain the test as a separate 0002 patch, as it may need a
few rounds of review and final adjustments. Once it's fully completed,
we can merge it back in.

5) Once we create and drop publication/subscriptions for individual
tests, we won't need such extensive configuration; we should be able
to run them with default values:
+$node_publisher->append_conf(
+       'postgresql.conf',
+       "max_wal_senders = 20
+        max_replication_slots = 20");

Regards,
Vignesh

  113. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-16T11:10:33Z

    On Sat, Aug 10, 2024 at 7:53 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 8 Aug 2024 at 10:53, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, Aug 7, 2024 at 1:31 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Shubham,
> > >
> > > Here are my review comments for patch v24-0001
> > >
> > > I think the TAP tests have incorrect expected results for the nogen-to-gen case.
> > >
> > > Whereas the HEAD code will cause "ERROR" for this test scenario, patch
> > > 0001 does not. IMO the behaviour should be unchanged for this scenario
> > > which has no generated column on the publisher side. So it seems this
> > > is a bug in patch 0001.
> > >
> > > FYI, I have included "FIXME" comments in the attached top-up diff
> > > patch to show which test cases I think are expecting wrong results.
> > >
> >
> > Fixed all the comments. The attached Patch(v25-0001) contains all the changes.
>
> Few comments:
> 1) Can we add one test with replica identity full to show that
> generated column is included in case of update operation with
> test_decoding.
>
> 2) At the end of the file generated_columns.sql a newline is missing:
> +-- when 'include-generated-columns' = '0' the generated column 'b'
> values will not be replicated
> +INSERT INTO gencoltable (a) VALUES (7), (8), (9);
> +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
> NULL, 'include-xids', '0', 'skip-empty-xacts', '1',
> 'include-generated-columns', '0');
> +
> +DROP TABLE gencoltable;
> +
> +SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
> \ No newline at end of file
>
> 3)
> 3.a)This can be changed:
> +-- when 'include-generated-columns' is not set the generated column
> 'b' values will be replicated
> +INSERT INTO gencoltable (a) VALUES (1), (2), (3);
>
> to:
> -- By default, 'include-generated-columns' is enabled, so the values
> for the generated column 'b' will be replicated even if it is not
> explicitly specified.
>
> 3.b) This can be changed:
> -- when 'include-generated-columns' = '1' the generated column 'b'
> values will be replicated
> to:
> -- when 'include-generated-columns' is enabled, the values of the
> generated column 'b' will be replicated.
>
> 3.c) This can be changed:
> -- when 'include-generated-columns' = '0' the generated column 'b'
> values will not be replicated
> to:
> -- when 'include-generated-columns' is disabled, the values of the
> generated column 'b' will not be replicated.
>
> 4) I did not see any test for dump, can we add one test for this.

Fixed all the given comments. The attached Patch v27-0001 contains all
the required changes. Also, I have created a separate Patch (v27-0002)
for TAP Tests related to 'include-generated-columns'

Thanks and Regards,
Shubham Khanna.

  114. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-19T05:31:13Z

    Hi, Here are my review comments for v27-0001.

======
contrib/test_decoding/expected/generated_columns.out
contrib/test_decoding/sql/generated_columns.sql

+-- By default, 'include-generated-columns' is enabled, so the values
for the generated column 'b' will be replicated even if it is not
explicitly specified.

nit - The "default" is only like this for "test_decoding" (e.g., the
CREATE SUBSCRIPTION option is the opposite), so let's make the comment
clearer about that.
nit - Use sentence case in the comments.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  115. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-19T07:10:27Z

    Hi Shubham, here are my review comments for the TAP tests patch v27-0002

======
Commit message

Tap tests for 'include-generated-columns'

~

But, it's more than that-- these are the TAP tests for all
combinations of replication related to generated columns. i.e. both
with and without 'include_generated_columns' option enabled.

======
src/test/subscription/t/011_generated.pl

I was mistaken, thinking that the v27-0002 had already been refactored
according to Vignesh's last review but it is not done yet, so I am not
going to post detailed review comments until the restructuring is
completed.

~

OTOH, there are some problems I felt have crept into v26-0001 (TAP
test is same as v27-0002), so maybe try to also take care of them (see
below) in v28-0002.

In no particular order:

* I felt it is almost useless now to have the "combo" (
"regress_pub_combo")  publication. It used to have many tables when
you first created it but with every version posted it is publishing
less and less so now there are only 2 tables in it. Better to have a
specific publication for each table now and forget about "combos"

* The "TEST tab_gen_to_gen initial sync" seems to be not even checking
the table data. Why not? e.g. Even if you expect no data, you should
test for it.

* The "TEST tab_gen_to_gen replication" seems to be not even checking
the table data. Why not?

* Multiple XXX comments like "... it needs more study to determine if
the above result was actually correct, or a PG17 bug..." should be
removed. AFAIK we should well understand the expected results for all
combinations by now.

* The "TEST tab_order replication" is now getting an error saying
<missing replicated column: "c">, Now, that may now be the correct
error for this situation, but in that case, then I think the test is
not longer testing what it was intended to test (i.e. that column
order does not matter....) Probably the table definition needs
adjusting to make sure we are testing whenwe want to test, and not
just making some random scenario "PASS".

* The test "# TEST tab_alter" expected empty result also seems
unhelpful. It might be related to the previous bullet.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  116. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-22T04:52:03Z

    On Fri, Aug 16, 2024 at 2:47 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 16 Aug 2024 at 10:04, Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Shubham,
> > >
> > > I think the v25-0001 patch only half-fixes the problems reported in my
> > > v24-0001 review.
> > >
> > > ~
> > >
> > > Background (from the commit message):
> > > This commit enables support for the 'include_generated_columns' option
> > > in logical replication, allowing the transmission of generated column
> > > information and data alongside regular table changes.
> > >
> > > ~
> > >
> > > The broken TAP test scenario in question is replicating from a
> > > "not-generated" column to a "generated" column. As the generated
> > > column is not on the publishing side, IMO the
> > > 'include_generated_columns' option should have zero effect here.
> > >
> > > In other words, I expect this TAP test for 'include_generated_columns
> > > = true' case should also be failing, as I wrote already yesterday:
> > >
> > > +# FIXME
> > > +# Since there is no generated column on the publishing side this should give
> > > +# the same result as the previous test. -- e.g. something like:
> > > +# ERROR:  logical replication target relation
> > > "public.tab_nogen_to_gen" is missing
> > > +# replicated column: "b"
> >
> > I have fixed the given comments. The attached v26-0001 Patch contains
> > the required changes.
>
> Few comments:
> 1) There's no need to pass include_generated_columns in this case; we
> can retrieve it from ctx->data instead:
> @@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
>  static void
>  send_relation_and_attrs(Relation relation, TransactionId xid,
>                                                 LogicalDecodingContext *ctx,
> -                                               Bitmapset *columns)
> +                                               Bitmapset *columns,
> bool include_generated_columns)
>  {
>         TupleDesc       desc = RelationGetDescr(relation);
>         int                     i;
> @@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation,
> TransactionId xid,
>
> 2) Commit message:
> If the subscriber-side column is also a generated column then this option
> has no effect; the replicated data will be ignored and the subscriber
> column will be filled as normal with the subscriber-side computed or
> default data.
>
> An error will occur in this case, so the message should be updated accordingly.
>
> 3) The current test is structured as follows: a) Create all required
> tables b) Insert data into tables c) Create publications d) Create
> subscriptions e) Perform inserts and verify
> This approach can make reviewing and maintenance somewhat challenging.
>
> Instead, could you modify it to: a) Create the required table for a
> single test b) Insert data for this test c) Create the publication for
> this test d) Create the subscriptions for this test e) Perform inserts
> and verify f) Clean up
>
> 4) We can maintain the test as a separate 0002 patch, as it may need a
> few rounds of review and final adjustments. Once it's fully completed,
> we can merge it back in.
>
> 5) Once we create and drop publication/subscriptions for individual
> tests, we won't need such extensive configuration; we should be able
> to run them with default values:
> +$node_publisher->append_conf(
> +       'postgresql.conf',
> +       "max_wal_senders = 20
> +        max_replication_slots = 20");

Fixed all the given comments. The attached patches contain the
suggested changes.

Thanks and Regards,
Shubham Khanna.

  117. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-22T04:56:24Z

    On Mon, Aug 19, 2024 at 11:01 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, Here are my review comments for v27-0001.
>
> ======
> contrib/test_decoding/expected/generated_columns.out
> contrib/test_decoding/sql/generated_columns.sql
>
> +-- By default, 'include-generated-columns' is enabled, so the values
> for the generated column 'b' will be replicated even if it is not
> explicitly specified.
>
> nit - The "default" is only like this for "test_decoding" (e.g., the
> CREATE SUBSCRIPTION option is the opposite), so let's make the comment
> clearer about that.
> nit - Use sentence case in the comments.

I have addressed all the comments in the v-28-0001 Patch. Please refer
to the updated v28-0001 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjL7rkxk6qSroRPg5ZARWMdK2Nd4-QyYNeoc2vhBm3cdDg%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  118. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-08-22T04:58:16Z

    On Mon, Aug 19, 2024 at 12:40 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham, here are my review comments for the TAP tests patch v27-0002
>
> ======
> Commit message
>
> Tap tests for 'include-generated-columns'
>
> ~
>
> But, it's more than that-- these are the TAP tests for all
> combinations of replication related to generated columns. i.e. both
> with and without 'include_generated_columns' option enabled.
>
> ======
> src/test/subscription/t/011_generated.pl
>
> I was mistaken, thinking that the v27-0002 had already been refactored
> according to Vignesh's last review but it is not done yet, so I am not
> going to post detailed review comments until the restructuring is
> completed.
>
> ~
>
> OTOH, there are some problems I felt have crept into v26-0001 (TAP
> test is same as v27-0002), so maybe try to also take care of them (see
> below) in v28-0002.
>
> In no particular order:
>
> * I felt it is almost useless now to have the "combo" (
> "regress_pub_combo")  publication. It used to have many tables when
> you first created it but with every version posted it is publishing
> less and less so now there are only 2 tables in it. Better to have a
> specific publication for each table now and forget about "combos"
>
> * The "TEST tab_gen_to_gen initial sync" seems to be not even checking
> the table data. Why not? e.g. Even if you expect no data, you should
> test for it.
>
> * The "TEST tab_gen_to_gen replication" seems to be not even checking
> the table data. Why not?
>
> * Multiple XXX comments like "... it needs more study to determine if
> the above result was actually correct, or a PG17 bug..." should be
> removed. AFAIK we should well understand the expected results for all
> combinations by now.
>
> * The "TEST tab_order replication" is now getting an error saying
> <missing replicated column: "c">, Now, that may now be the correct
> error for this situation, but in that case, then I think the test is
> not longer testing what it was intended to test (i.e. that column
> order does not matter....) Probably the table definition needs
> adjusting to make sure we are testing whenwe want to test, and not
> just making some random scenario "PASS".
>
> * The test "# TEST tab_alter" expected empty result also seems
> unhelpful. It might be related to the previous bullet.

I have addressed all the comments in the v-28-0002 Patch. Please refer
to the updated v28-0002 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjL7rkxk6qSroRPg5ZARWMdK2Nd4-QyYNeoc2vhBm3cdDg%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  119. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-08-22T11:58:21Z

    On Thu, 22 Aug 2024 at 10:22, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Fri, Aug 16, 2024 at 2:47 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 16 Aug 2024 at 10:04, Shubham Khanna
> > <khannashubham1197@gmail.com> wrote:
> > >
> > > On Thu, Aug 8, 2024 at 12:43 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Shubham,
> > > >
> > > > I think the v25-0001 patch only half-fixes the problems reported in my
> > > > v24-0001 review.
> > > >
> > > > ~
> > > >
> > > > Background (from the commit message):
> > > > This commit enables support for the 'include_generated_columns' option
> > > > in logical replication, allowing the transmission of generated column
> > > > information and data alongside regular table changes.
> > > >
> > > > ~
> > > >
> > > > The broken TAP test scenario in question is replicating from a
> > > > "not-generated" column to a "generated" column. As the generated
> > > > column is not on the publishing side, IMO the
> > > > 'include_generated_columns' option should have zero effect here.
> > > >
> > > > In other words, I expect this TAP test for 'include_generated_columns
> > > > = true' case should also be failing, as I wrote already yesterday:
> > > >
> > > > +# FIXME
> > > > +# Since there is no generated column on the publishing side this should give
> > > > +# the same result as the previous test. -- e.g. something like:
> > > > +# ERROR:  logical replication target relation
> > > > "public.tab_nogen_to_gen" is missing
> > > > +# replicated column: "b"
> > >
> > > I have fixed the given comments. The attached v26-0001 Patch contains
> > > the required changes.
> >
> > Few comments:
> > 1) There's no need to pass include_generated_columns in this case; we
> > can retrieve it from ctx->data instead:
> > @@ -749,7 +764,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
> >  static void
> >  send_relation_and_attrs(Relation relation, TransactionId xid,
> >                                                 LogicalDecodingContext *ctx,
> > -                                               Bitmapset *columns)
> > +                                               Bitmapset *columns,
> > bool include_generated_columns)
> >  {
> >         TupleDesc       desc = RelationGetDescr(relation);
> >         int                     i;
> > @@ -766,7 +781,10 @@ send_relation_and_attrs(Relation relation,
> > TransactionId xid,
> >
> > 2) Commit message:
> > If the subscriber-side column is also a generated column then this option
> > has no effect; the replicated data will be ignored and the subscriber
> > column will be filled as normal with the subscriber-side computed or
> > default data.
> >
> > An error will occur in this case, so the message should be updated accordingly.
> >
> > 3) The current test is structured as follows: a) Create all required
> > tables b) Insert data into tables c) Create publications d) Create
> > subscriptions e) Perform inserts and verify
> > This approach can make reviewing and maintenance somewhat challenging.
> >
> > Instead, could you modify it to: a) Create the required table for a
> > single test b) Insert data for this test c) Create the publication for
> > this test d) Create the subscriptions for this test e) Perform inserts
> > and verify f) Clean up
> >
> > 4) We can maintain the test as a separate 0002 patch, as it may need a
> > few rounds of review and final adjustments. Once it's fully completed,
> > we can merge it back in.
> >
> > 5) Once we create and drop publication/subscriptions for individual
> > tests, we won't need such extensive configuration; we should be able
> > to run them with default values:
> > +$node_publisher->append_conf(
> > +       'postgresql.conf',
> > +       "max_wal_senders = 20
> > +        max_replication_slots = 20");
>
> Fixed all the given comments. The attached patches contain the
> suggested changes.

Few comments:
1) This is already been covered in the first existing test case, may
be this can be removed:
# =============================================================================
# Testcase start: Subscriber table with a generated column (b) on the
# subscriber, where column (b) is not present on the publisher.

This existing test:
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
INSERT INTO tab1 (a) VALUES (1), (2), (3);
CREATE PUBLICATION pub1 FOR ALL TABLES;
));

$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a *
22) STORED, c int);
CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1;
));

2) Can we have this test verified with include_generated_columns =
true too like how others are done:
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';

$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
INSERT INTO tab1 (a) VALUES (1), (2), (3);
CREATE PUBLICATION pub1 FOR ALL TABLES;
));

$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE tab1 (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a *
22) STORED, c int);
CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1;
));

3) There is a typo in this comment:
3.a)  # Testcase start: Publisher table with a generated column (b)
and subscriber
# table a with regular column (b).

It should be:
# Testcase start: Publisher table with a generated column (b) and subscriber
# table with a regular column (b).

3.b) similarly here too:
# Testcase end: Publisher table with a generated column (b) and subscriber
# table a with regular column (b).

3.c) The comments are not consistent, sometimes mentioned as
column(b) and sometimes as column (b). We can keep it consistent.

Regards,
Vignesh

  120. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-08-23T10:56:24Z

    Hi Shubham,

I have reviewed v28* and posted updated v29 versions of patches 0001 and 0002.

If you are OK with these changes, the next task would be to pg_indent
them, then rebase the remaining patches (0003 etc.) and include those
with the next patchset version.

//////////

Patch v29-0001 changes:

nit - Fixed typo in comments.
nit - Removed an unnecessary format change for the unchanged
send_relation_and_attrs declaration.

//////////

Patch v29-0002 changes:

1.
Made fixes to address Vignesh's review comments [1].

2.
Added the missing test cases for tab_gen_to_gen, and tab_alter.

3.
Multiple other modifications include:
nit - Renamed the test database /test/test_igc_true/ because 'test'
was too vague.
nit - This patch does not need to change most of the existing 'tab1'
test. So we should not be reformating the existing test code for no
reason.
nit - I added a summary comment to describe the test combinations
nit - The "Testcase end" comments were unnecessary and prone to error,
so I removed them.
nit - Change comments /incremental sync/incremental replication/
nit - Added XXX notes about copy_data=false. These are reminders for
to change code in later TAP patches
nit - Rearranged test steps so the publisher does not do incremental
INSERT until all initial sync tests are done
nit - Added initial sync tests even if copy_data=false. This is for
completeness - these will be handled in a later TAP patch
nit - The table names are self-explanatory, so some of the test
"messages" were simplified

======
[1] https://www.postgresql.org/message-id/CALDaNm31LZQfeR8Vv1qNCOREGffvZbgGDrTp%3D3h%3DEHiHTEO2pQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  121. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-08-28T06:06:19Z

    On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, May 8, 2024 at 4:14 PM Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal
> > <dangwalrajendra888@gmail.com> wrote:
> > >
> > > Hi PG Hackers.
> > >
> > > We are interested in enhancing the functionality of the pgoutput plugin by adding support for generated columns.
> > > Could you please guide us on the necessary steps to achieve this? Additionally, do you have a platform for tracking such feature requests? Any insights or assistance you can provide on this matter would be greatly appreciated.
> >
> > The attached patch has the changes to support capturing generated
> > column data using ‘pgoutput’ and’ test_decoding’ plugin. Now if the
> > ‘include_generated_columns’ option is specified, the generated column
> > information and generated column data also will be sent.
>
> As Euler mentioned earlier, I think it's a decision not to replicate
> generated columns because we don't know the target table on the
> subscriber has the same expression and there could be locale issues
> even if it looks the same. I can see that a benefit of this proposal
> would be to save cost to compute generated column values if the user
> wants the target table on the subscriber to have exactly the same data
> as the publisher's one. Are there other benefits or use cases?
>

The cost is one but the other is the user may not want the data to be
different based on volatile functions like timeofday() or the table on
subscriber won't have the column marked as generated. Now, considering
such use cases, is providing a subscription-level option a good idea
as the patch is doing? I understand that this can serve the purpose
but it could also lead to having the same behavior for all the tables
in all the publications for a subscription which may or may not be
what the user expects. This could lead to some performance overhead
(due to always sending generated columns for all the tables) for cases
where the user needs it only for a subset of tables.

I think we should consider it as a table-level option while defining
publication in some way. A few ideas could be: (a) We ask users to
explicitly mention the generated column in the columns list while
defining publication. This has a drawback such that users need to
specify the column list even when all columns need to be replicated.
(b) We can have some new syntax to indicate the same like: CREATE
PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
could be some challenges but we can at least investigate it.

Yet another idea is to keep this as a publication option
(include_generated_columns or publish_generated_columns) similar to
"publish_via_partition_root". Normally, "publish_via_partition_root"
is used when tables on either side have different partition
hierarchies which is somewhat the case here.

Thoughts?

-- 
With Regards,
Amit Kapila.

  122. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-08-29T03:13:31Z

    On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, May 8, 2024 at 4:14 PM Shubham Khanna
> > <khannashubham1197@gmail.com> wrote:
> > >
> > > On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal
> > > <dangwalrajendra888@gmail.com> wrote:
> > > >
> > > > Hi PG Hackers.
> > > >
> > > > We are interested in enhancing the functionality of the pgoutput plugin by adding support for generated columns.
> > > > Could you please guide us on the necessary steps to achieve this? Additionally, do you have a platform for tracking such feature requests? Any insights or assistance you can provide on this matter would be greatly appreciated.
> > >
> > > The attached patch has the changes to support capturing generated
> > > column data using ‘pgoutput’ and’ test_decoding’ plugin. Now if the
> > > ‘include_generated_columns’ option is specified, the generated column
> > > information and generated column data also will be sent.
> >
> > As Euler mentioned earlier, I think it's a decision not to replicate
> > generated columns because we don't know the target table on the
> > subscriber has the same expression and there could be locale issues
> > even if it looks the same. I can see that a benefit of this proposal
> > would be to save cost to compute generated column values if the user
> > wants the target table on the subscriber to have exactly the same data
> > as the publisher's one. Are there other benefits or use cases?
> >
>
> The cost is one but the other is the user may not want the data to be
> different based on volatile functions like timeofday()

Shouldn't the generation expression be immutable?

> or the table on
> subscriber won't have the column marked as generated.

Yeah, it would be another use case.

>  Now, considering
> such use cases, is providing a subscription-level option a good idea
> as the patch is doing? I understand that this can serve the purpose
> but it could also lead to having the same behavior for all the tables
> in all the publications for a subscription which may or may not be
> what the user expects. This could lead to some performance overhead
> (due to always sending generated columns for all the tables) for cases
> where the user needs it only for a subset of tables.

Yeah, it's a downside and I think it's less flexible. For example, if
users want to send both tables with generated columns and tables
without generated columns, they would have to create at least two
subscriptions. Also, they would have to include a different set of
tables to two publications.

>
> I think we should consider it as a table-level option while defining
> publication in some way. A few ideas could be: (a) We ask users to
> explicitly mention the generated column in the columns list while
> defining publication. This has a drawback such that users need to
> specify the column list even when all columns need to be replicated.
> (b) We can have some new syntax to indicate the same like: CREATE
> PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
> INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
> could be some challenges but we can at least investigate it.

I think we can create a publication for a single table, so what we can
do with this feature can be done also by the idea you described below.

> Yet another idea is to keep this as a publication option
> (include_generated_columns or publish_generated_columns) similar to
> "publish_via_partition_root". Normally, "publish_via_partition_root"
> is used when tables on either side have different partition
> hierarchies which is somewhat the case here.

It sounds more useful to me.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  123. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-08-29T06:16:35Z

    On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > As Euler mentioned earlier, I think it's a decision not to replicate
> > > generated columns because we don't know the target table on the
> > > subscriber has the same expression and there could be locale issues
> > > even if it looks the same. I can see that a benefit of this proposal
> > > would be to save cost to compute generated column values if the user
> > > wants the target table on the subscriber to have exactly the same data
> > > as the publisher's one. Are there other benefits or use cases?
> > >
> >
> > The cost is one but the other is the user may not want the data to be
> > different based on volatile functions like timeofday()
>
> Shouldn't the generation expression be immutable?
>

Yes, I missed that point.

> > or the table on
> > subscriber won't have the column marked as generated.
>
> Yeah, it would be another use case.
>

Right, apart from that I am not aware of other use cases. If they
have, I would request Euler or Rajendra to share any other use case.

> >  Now, considering
> > such use cases, is providing a subscription-level option a good idea
> > as the patch is doing? I understand that this can serve the purpose
> > but it could also lead to having the same behavior for all the tables
> > in all the publications for a subscription which may or may not be
> > what the user expects. This could lead to some performance overhead
> > (due to always sending generated columns for all the tables) for cases
> > where the user needs it only for a subset of tables.
>
> Yeah, it's a downside and I think it's less flexible. For example, if
> users want to send both tables with generated columns and tables
> without generated columns, they would have to create at least two
> subscriptions.
>

Agreed and that would consume more resources.

> Also, they would have to include a different set of
> tables to two publications.
>
> >
> > I think we should consider it as a table-level option while defining
> > publication in some way. A few ideas could be: (a) We ask users to
> > explicitly mention the generated column in the columns list while
> > defining publication. This has a drawback such that users need to
> > specify the column list even when all columns need to be replicated.
> > (b) We can have some new syntax to indicate the same like: CREATE
> > PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
> > INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
> > could be some challenges but we can at least investigate it.
>
> I think we can create a publication for a single table, so what we can
> do with this feature can be done also by the idea you described below.
>
> > Yet another idea is to keep this as a publication option
> > (include_generated_columns or publish_generated_columns) similar to
> > "publish_via_partition_root". Normally, "publish_via_partition_root"
> > is used when tables on either side have different partition
> > hierarchies which is somewhat the case here.
>
> It sounds more useful to me.
>

Fair enough. Let's see if anyone else has any preference among the
proposed methods or can think of a better way.

-- 
With Regards,
Amit Kapila.

  124. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-09T09:38:19Z

    On Thu, Aug 29, 2024 at 11:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > As Euler mentioned earlier, I think it's a decision not to replicate
> > > > generated columns because we don't know the target table on the
> > > > subscriber has the same expression and there could be locale issues
> > > > even if it looks the same. I can see that a benefit of this proposal
> > > > would be to save cost to compute generated column values if the user
> > > > wants the target table on the subscriber to have exactly the same data
> > > > as the publisher's one. Are there other benefits or use cases?
> > > >
> > >
> > > The cost is one but the other is the user may not want the data to be
> > > different based on volatile functions like timeofday()
> >
> > Shouldn't the generation expression be immutable?
> >
>
> Yes, I missed that point.
>
> > > or the table on
> > > subscriber won't have the column marked as generated.
> >
> > Yeah, it would be another use case.
> >
>
> Right, apart from that I am not aware of other use cases. If they
> have, I would request Euler or Rajendra to share any other use case.
>
> > >  Now, considering
> > > such use cases, is providing a subscription-level option a good idea
> > > as the patch is doing? I understand that this can serve the purpose
> > > but it could also lead to having the same behavior for all the tables
> > > in all the publications for a subscription which may or may not be
> > > what the user expects. This could lead to some performance overhead
> > > (due to always sending generated columns for all the tables) for cases
> > > where the user needs it only for a subset of tables.
> >
> > Yeah, it's a downside and I think it's less flexible. For example, if
> > users want to send both tables with generated columns and tables
> > without generated columns, they would have to create at least two
> > subscriptions.
> >
>
> Agreed and that would consume more resources.
>
> > Also, they would have to include a different set of
> > tables to two publications.
> >
> > >
> > > I think we should consider it as a table-level option while defining
> > > publication in some way. A few ideas could be: (a) We ask users to
> > > explicitly mention the generated column in the columns list while
> > > defining publication. This has a drawback such that users need to
> > > specify the column list even when all columns need to be replicated.
> > > (b) We can have some new syntax to indicate the same like: CREATE
> > > PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
> > > INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
> > > could be some challenges but we can at least investigate it.
> >
> > I think we can create a publication for a single table, so what we can
> > do with this feature can be done also by the idea you described below.
> >
> > > Yet another idea is to keep this as a publication option
> > > (include_generated_columns or publish_generated_columns) similar to
> > > "publish_via_partition_root". Normally, "publish_via_partition_root"
> > > is used when tables on either side have different partitions
> > > hierarchies which is somewhat the case here.
> >
> > It sounds more useful to me.
> >
>
> Fair enough. Let's see if anyone else has any preference among the
> proposed methods or can think of a better way.

I have fixed the current issue. I have added the option
'publish_generated_columns' to the publisher side and created the new
test cases accordingly.
The attached patches contain the desired changes.

Thanks and Regards,
Shubham Khanna.

  125. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-09T21:20:44Z

    On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Thu, Aug 29, 2024 at 11:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > As Euler mentioned earlier, I think it's a decision not to replicate
> > > > > generated columns because we don't know the target table on the
> > > > > subscriber has the same expression and there could be locale issues
> > > > > even if it looks the same. I can see that a benefit of this proposal
> > > > > would be to save cost to compute generated column values if the user
> > > > > wants the target table on the subscriber to have exactly the same data
> > > > > as the publisher's one. Are there other benefits or use cases?
> > > > >
> > > >
> > > > The cost is one but the other is the user may not want the data to be
> > > > different based on volatile functions like timeofday()
> > >
> > > Shouldn't the generation expression be immutable?
> > >
> >
> > Yes, I missed that point.
> >
> > > > or the table on
> > > > subscriber won't have the column marked as generated.
> > >
> > > Yeah, it would be another use case.
> > >
> >
> > Right, apart from that I am not aware of other use cases. If they
> > have, I would request Euler or Rajendra to share any other use case.
> >
> > > >  Now, considering
> > > > such use cases, is providing a subscription-level option a good idea
> > > > as the patch is doing? I understand that this can serve the purpose
> > > > but it could also lead to having the same behavior for all the tables
> > > > in all the publications for a subscription which may or may not be
> > > > what the user expects. This could lead to some performance overhead
> > > > (due to always sending generated columns for all the tables) for cases
> > > > where the user needs it only for a subset of tables.
> > >
> > > Yeah, it's a downside and I think it's less flexible. For example, if
> > > users want to send both tables with generated columns and tables
> > > without generated columns, they would have to create at least two
> > > subscriptions.
> > >
> >
> > Agreed and that would consume more resources.
> >
> > > Also, they would have to include a different set of
> > > tables to two publications.
> > >
> > > >
> > > > I think we should consider it as a table-level option while defining
> > > > publication in some way. A few ideas could be: (a) We ask users to
> > > > explicitly mention the generated column in the columns list while
> > > > defining publication. This has a drawback such that users need to
> > > > specify the column list even when all columns need to be replicated.
> > > > (b) We can have some new syntax to indicate the same like: CREATE
> > > > PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
> > > > INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
> > > > could be some challenges but we can at least investigate it.
> > >
> > > I think we can create a publication for a single table, so what we can
> > > do with this feature can be done also by the idea you described below.
> > >
> > > > Yet another idea is to keep this as a publication option
> > > > (include_generated_columns or publish_generated_columns) similar to
> > > > "publish_via_partition_root". Normally, "publish_via_partition_root"
> > > > is used when tables on either side have different partitions
> > > > hierarchies which is somewhat the case here.
> > >
> > > It sounds more useful to me.
> > >
> >
> > Fair enough. Let's see if anyone else has any preference among the
> > proposed methods or can think of a better way.
>
> I have fixed the current issue. I have added the option
> 'publish_generated_columns' to the publisher side and created the new
> test cases accordingly.
> The attached patches contain the desired changes.
>

Thank you for updating the patches. I have some comments:

Do we really need to add this option to test_decoding? I think it
would be good if this improves the test coverage. Otherwise, I'm not
sure we need this part. If we want to add it, I think it would be
better to have it in a separate patch.

---
+         <para>
+          If the publisher-side column is also a generated column
then this option
+          has no effect; the publisher column will be filled as normal with the
+          publisher-side computed or default data.
+         </para>

I don't understand this description. Why does this option have no
effect if the publisher-side column is a generated column?

---
+         <para>
+         This parameter can only be set <literal>true</literal> if
<literal>copy_data</literal> is
+         set to <literal>false</literal>.
+         </para>

If I understand this patch correctly, it doesn't disallow to set
copy_data to true when the publish_generated_columns option is
specified. But do we want to disallow it? I think it would be more
useful and understandable if we allow to use both
publish_generated_columns (publisher option) and copy_data (subscriber
option) at the same time.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  126. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-10T04:14:55Z

    On Tue, Sep 10, 2024 at 2:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
>
> Thank you for updating the patches. I have some comments:
>
> Do we really need to add this option to test_decoding?
>

I don't see any reason to have such an option in test_decoding,
otherwise, we need a separate option for each publication option. I
guess this is leftover of the previous subscriber-side approach.

> I think it
> would be good if this improves the test coverage. Otherwise, I'm not
> sure we need this part. If we want to add it, I think it would be
> better to have it in a separate patch.
>

Right.

> ---
> +         <para>
> +          If the publisher-side column is also a generated column
> then this option
> +          has no effect; the publisher column will be filled as normal with the
> +          publisher-side computed or default data.
> +         </para>
>
> I don't understand this description. Why does this option have no
> effect if the publisher-side column is a generated column?
>

Shouldn't it be subscriber-side?

I have one additional comment:
/*
- * If the publication is FOR ALL TABLES then it is treated the same as
- * if there are no column lists (even if other publications have a
- * list).
+ * If the publication is FOR ALL TABLES and include generated columns
+ * then it is treated the same as if there are no column lists (even
+ * if other publications have a list).
  */
- if (!pub->alltables)
+ if (!pub->alltables || !pub->pubgencolumns)

Why do we treat pubgencolumns at the same level as the FOR ALL TABLES
case? I thought that if the user has provided a column list, we only
need to publish the specified columns even when the
publish_generated_columns option is set.

-- 
With Regards,
Amit Kapila.

  127. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-10T08:19:37Z

    IIUC, previously there was a subscriber side option
'include_generated_columns', but now since v30* there is a publisher
side option 'publish_generated_columns'.

Fair enough, but in the v30* patches I can still see remnants of the
old name 'include_generated_columns' all over the place:
- in the commit message
- in the code (struct field names, param names etc)
- in the comments
- in the docs

If the decision is to call the new PUBLICATION option
'publish_generated_columns', then can't we please use that one name
*everywhere* -- e.g. replace all cases where any old name is still
lurking?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  128. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-11T03:24:33Z

    Here are a some more review comments for patch v30-0001.

======
src/sgml/ref/create_publication.sgml

1.
+         <para>
+          If the publisher-side column is also a generated column
then this option
+          has no effect; the publisher column will be filled as normal with the
+          publisher-side computed or default data.
+         </para>

It should say "subscriber-side"; not "publisher-side". The same was
already reported by Sawada-San [1].

~~~

2.
+         <para>
+         This parameter can only be set <literal>true</literal> if
<literal>copy_data</literal> is
+         set to <literal>false</literal>.
+         </para>

IMO this limitation should be addressed by patch 0001 like it was
already done in the previous patches (e.g. v22-0002). I think
Sawada-san suggested the same [1].

Anyway, 'copy_data' is not a PUBLICATION option, so the fact it is
mentioned like this without any reference to the SUBSCRIPTION seems
like a cut/paste error from the previous implementation.

======
src/backend/catalog/pg_publication.c

3. pub_collist_validate
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
-    colname));
-

Instead of just removing this ERROR entirely here, I thought it would
be more user-friendly to give a WARNING if the PUBLICATION's explicit
column list includes generated cols when the option
"publish_generated_columns" is false. This combination doesn't seem
like something a user would do intentionally, so just silently
ignoring it (like the current patch does) is likely going to give
someone unexpected results/grief.

======
src/backend/replication/logical/proto.c

4. logicalrep_write_tuple, and logicalrep_write_attrs:

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

Why aren't you also checking the new PUBLICATION option here and
skipping all gencols if the "publish_generated_columns" option is
false? Or is the BMS of pgoutput_column_list_init handling this case?
Maybe there should be an Assert for this?

======
src/backend/replication/pgoutput/pgoutput.c

5. send_relation_and_attrs

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

Same question as #4.

~~~

6. prepare_all_columns_bms and pgoutput_column_list_init

+ if (att->attgenerated && !pub->pubgencolumns)
+ cols = bms_del_member(cols, i + 1);

IIUC, the algorithm seems overly tricky filling the BMS with all
columns, before straight away conditionally removing the generated
columns. Can't it be refactored to assign all the correct columns
up-front, to avoid calling bms_del_member()?

======
src/bin/pg_dump/pg_dump.c

7. getPublications

IIUC, there is lots of missing SQL code here (for all older versions)
that should be saying "false AS pubgencolumns".
e.g. compare the SQL with how "false AS pubviaroot" is used.

======
src/bin/pg_dump/t/002_pg_dump.pl

8. Missing tests?

I expected to see a pg_dump test for this new PUBLICATION option.

======
src/test/regress/sql/publication.sql

9. Missing tests?

How about adding another test case that checks this new option must be
"Boolean"?

~~~

10. Missing tests?

--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;

(see my earlier comment #3)

IMO there should be another test case for a WARNING here if the user
attempts to include generated column 'd' in an explicit PUBLICATION
column list while the "publish_generated-columns" is false.

======
[1]  https://www.postgresql.org/message-id/CAD21AoA-tdTz0G-vri8KM2TXeFU8RCDsOpBXUBCgwkfokF7%3DjA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  129. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-11T07:06:10Z

    Hi Shubham,

Here are my general comments about the v30-0002 TAP test patch.

======

1.
As mentioned in a previous post [1] there are still several references
to the old 'include_generated_columns' option remaining in this patch.
They need replacing.

~~~

2.
+# Furthermore, all combinations are tested for publish_generated_columns=false
+# (see subscription sub1 of database 'postgres'), and
+# publish_generated_columns=true (see subscription sub2 of database
+# 'test_igc_true').

Those 'see subscription' notes and 'test_igc_true' are from the old
implementation. Those need fixing. BTW, 'test_pgc_true' is a better
name for the database now that the option name is changed.

In the previous implementation, the TAP test environment was:
- a common publication pub, on the 'postgres' database
- a subscription sub1 with option include_generated_columns=false, on
the 'postgres' database
- a subscription sub2 with option include_generated_columns=true, on
the 'test_igc_true' database

Now it is like:
- a publication pub1, on the 'postgres' database, with option
publish_generated_columns=false
- a publication pub2, on the 'postgres' database, with option
publish_generated_columns=true
- a subscription sub1, on the 'postgres' database for publication pub1
- a subscription sub2, on the 'test_pgc_true' database for publication pub2

It would be good to document that above convention because knowing how
the naming/numbering works makes it a lot easier to read the
subsequent test cases. Of course, it is really important to
name/number everything consistently otherwise these tests become hard
to follow.  AFAICT it is mostly OK, but the generated -> generated
publication should be called 'regress_pub2_gen_to_gen'

~~~

3.
+# Create table.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a *
2) STORED);
+ INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3);
+));
+
+# Create publication with publish_generated_columns=false.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub1_gen_to_nogen FOR TABLE
tab_gen_to_nogen WITH (publish_generated_columns = false)"
+);
+
+# Create table and subscription with copy_data=true.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_nogen (a int, b int);
+ CREATE SUBSCRIPTION regress_sub1_gen_to_nogen CONNECTION
'$publisher_connstr' PUBLICATION regress_pub1_gen_to_nogen WITH
(copy_data = true);
+));
+
+# Create publication with publish_generated_columns=true.
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION regress_pub2_gen_to_nogen FOR TABLE
tab_gen_to_nogen WITH (publish_generated_columns = true)"
+);
+

The code can be restructured to be simpler. Both publications are
always created on the 'postgres' database at the publisher node, so
let's just create them at the same time as the creating the publisher
table. It also makes readability much better e.g.

# Create table, and publications
$node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE tab_gen_to_nogen (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
INSERT INTO tab_gen_to_nogen (a) VALUES (1), (2), (3);
CREATE PUBLICATION regress_pub1_gen_to_nogen FOR TABLE
tab_gen_to_nogen WITH (publish_generated_columns = false);
CREATE PUBLICATION regress_pub2_gen_to_nogen FOR TABLE
tab_gen_to_nogen WITH (publish_generated_columns = true);
));

IFAICT this same simplification can be repeated multiple times in this TAP file.

~~

Similarly, it would be neater to combine DROP PUBLICATION's together too.

~~~

4.
Hopefully, the generated column 'copy_data' can be implemented again
soon for subscriptions, and then the initial sync tests here can be
properly implemented instead of the placeholders currently in patch
0002.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPuDJToG%3DV-ogTi9_6fnhhn2S0%2BsVRGPynhcf9mEh0Q%3DLA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  130. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-12T05:30:23Z

    Because this feature is now being implemented as a PUBLICATION option,
there is another scenario that might need consideration; I am thinking
about where the same table is published by multiple PUBLICATIONS (with
different option settings) that are subscribed by a single
SUBSCRIPTION.

e.g.1
-----
CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
-----

e.g.2
-----
CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
-----

Do you know if this case is supported? If yes, then which publication
option value wins?

The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
several publications in which the same table has been published with
different column lists are not supported."

Perhaps the user is supposed to deduce that the example above would
work OK if table 't1' has no generated cols. OTOH, if it did have
generated cols then the PUBLICATION column lists must be different and
therefore it is "not supported" (??).

I have not tried this to see what happens, but even if it behaves as
expected, there should probably be some comments/docs/tests for this
scenario to clarify it for the user.

Notice that "publish_via_partition_root" has a similar conundrum, but
in that case, the behaviour is documented in the CREATE PUBLICATION
docs [2]. So, maybe  "publish_generated_columns" should be documented
a bit like that.

======
[1] https://www.postgresql.org/docs/devel/sql-createsubscription.html
[2] https://www.postgresql.org/docs/devel/sql-createpublication.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  131. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-13T11:34:06Z

    On Tue, Sep 10, 2024 at 2:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Thu, Aug 29, 2024 at 11:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > >
> > > > > > As Euler mentioned earlier, I think it's a decision not to replicate
> > > > > > generated columns because we don't know the target table on the
> > > > > > subscriber has the same expression and there could be locale issues
> > > > > > even if it looks the same. I can see that a benefit of this proposal
> > > > > > would be to save cost to compute generated column values if the user
> > > > > > wants the target table on the subscriber to have exactly the same data
> > > > > > as the publisher's one. Are there other benefits or use cases?
> > > > > >
> > > > >
> > > > > The cost is one but the other is the user may not want the data to be
> > > > > different based on volatile functions like timeofday()
> > > >
> > > > Shouldn't the generation expression be immutable?
> > > >
> > >
> > > Yes, I missed that point.
> > >
> > > > > or the table on
> > > > > subscriber won't have the column marked as generated.
> > > >
> > > > Yeah, it would be another use case.
> > > >
> > >
> > > Right, apart from that I am not aware of other use cases. If they
> > > have, I would request Euler or Rajendra to share any other use case.
> > >
> > > > >  Now, considering
> > > > > such use cases, is providing a subscription-level option a good idea
> > > > > as the patch is doing? I understand that this can serve the purpose
> > > > > but it could also lead to having the same behavior for all the tables
> > > > > in all the publications for a subscription which may or may not be
> > > > > what the user expects. This could lead to some performance overhead
> > > > > (due to always sending generated columns for all the tables) for cases
> > > > > where the user needs it only for a subset of tables.
> > > >
> > > > Yeah, it's a downside and I think it's less flexible. For example, if
> > > > users want to send both tables with generated columns and tables
> > > > without generated columns, they would have to create at least two
> > > > subscriptions.
> > > >
> > >
> > > Agreed and that would consume more resources.
> > >
> > > > Also, they would have to include a different set of
> > > > tables to two publications.
> > > >
> > > > >
> > > > > I think we should consider it as a table-level option while defining
> > > > > publication in some way. A few ideas could be: (a) We ask users to
> > > > > explicitly mention the generated column in the columns list while
> > > > > defining publication. This has a drawback such that users need to
> > > > > specify the column list even when all columns need to be replicated.
> > > > > (b) We can have some new syntax to indicate the same like: CREATE
> > > > > PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
> > > > > INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
> > > > > could be some challenges but we can at least investigate it.
> > > >
> > > > I think we can create a publication for a single table, so what we can
> > > > do with this feature can be done also by the idea you described below.
> > > >
> > > > > Yet another idea is to keep this as a publication option
> > > > > (include_generated_columns or publish_generated_columns) similar to
> > > > > "publish_via_partition_root". Normally, "publish_via_partition_root"
> > > > > is used when tables on either side have different partitions
> > > > > hierarchies which is somewhat the case here.
> > > >
> > > > It sounds more useful to me.
> > > >
> > >
> > > Fair enough. Let's see if anyone else has any preference among the
> > > proposed methods or can think of a better way.
> >
> > I have fixed the current issue. I have added the option
> > 'publish_generated_columns' to the publisher side and created the new
> > test cases accordingly.
> > The attached patches contain the desired changes.
> >
>
> Thank you for updating the patches. I have some comments:
>
> Do we really need to add this option to test_decoding? I think it
> would be good if this improves the test coverage. Otherwise, I'm not
> sure we need this part. If we want to add it, I think it would be
> better to have it in a separate patch.
>

I have removed the option from the test_decoding file.

> ---
> +         <para>
> +          If the publisher-side column is also a generated column
> then this option
> +          has no effect; the publisher column will be filled as normal with the
> +          publisher-side computed or default data.
> +         </para>
>
> I don't understand this description. Why does this option have no
> effect if the publisher-side column is a generated column?
>

The documentation was incorrect. Currently, replicating from a
publisher table with a generated column to a subscriber table with a
generated column will result in an error. This has now been updated.

> ---
> +         <para>
> +         This parameter can only be set <literal>true</literal> if
> <literal>copy_data</literal> is
> +         set to <literal>false</literal>.
> +         </para>
>
> If I understand this patch correctly, it doesn't disallow to set
> copy_data to true when the publish_generated_columns option is
> specified. But do we want to disallow it? I think it would be more
> useful and understandable if we allow to use both
> publish_generated_columns (publisher option) and copy_data (subscriber
> option) at the same time.
>

Support for tablesync with generated columns was not included in the
initial patch, and this was reflected in the documentation. The
functionality for syncing generated column data has been introduced
with the 0002 patch.

The attached v31 patches contain the changes for the same. I won't be
posting the test patch for now. I will share it once this patch has
been stabilized.

Thanks and Regards,
Shubham Khanna.

  132. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-09-13T11:56:52Z

    On Tue, 10 Sept 2024 at 09:45, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Sep 10, 2024 at 2:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna
> > <khannashubham1197@gmail.com> wrote:
> > >
> >
> > Thank you for updating the patches. I have some comments:
> >
> > Do we really need to add this option to test_decoding?
> >
>
> I don't see any reason to have such an option in test_decoding,
> otherwise, we need a separate option for each publication option. I
> guess this is leftover of the previous subscriber-side approach.
>
> > I think it
> > would be good if this improves the test coverage. Otherwise, I'm not
> > sure we need this part. If we want to add it, I think it would be
> > better to have it in a separate patch.
> >
>
> Right.
>
> > ---
> > +         <para>
> > +          If the publisher-side column is also a generated column
> > then this option
> > +          has no effect; the publisher column will be filled as normal with the
> > +          publisher-side computed or default data.
> > +         </para>
> >
> > I don't understand this description. Why does this option have no
> > effect if the publisher-side column is a generated column?
> >
>
> Shouldn't it be subscriber-side?
>
> I have one additional comment:
> /*
> - * If the publication is FOR ALL TABLES then it is treated the same as
> - * if there are no column lists (even if other publications have a
> - * list).
> + * If the publication is FOR ALL TABLES and include generated columns
> + * then it is treated the same as if there are no column lists (even
> + * if other publications have a list).
>   */
> - if (!pub->alltables)
> + if (!pub->alltables || !pub->pubgencolumns)
>
> Why do we treat pubgencolumns at the same level as the FOR ALL TABLES
> case? I thought that if the user has provided a column list, we only
> need to publish the specified columns even when the
> publish_generated_columns option is set.

To handle cases where the publish_generated_columns option isn't
specified for all tables in a publication, the pubgencolumns check
needs to be performed. In such cases, we must create a column list
that excludes generated columns. This process involves:
a) Retrieving all columns for the table and adding them to the column
list. b) Iterating through this column list and removing generated
columns. c) Checking if the remaining column count matches the total
number of columns. If they match, set the relation entry's column list
to NULL, so we don’t need to check columns during data replication. If
they do not match, update the column list to include only the relevant
columns, allowing pgoutput to replicate data for these specific
columns.

This step is necessary because some tables in the publication may
include generated columns.
For tables where publish_generated_columns is set, the column list
will be set to NULL, eliminating the need for a column list check
during data publication.
However, modifying the column list based on publish_generated_columns
is not required, this is addressed in the v31 patch posted by Shubham
at [1].

[1] - https://www.postgresql.org/message-id/CAHv8Rj%2BinrG6EU0rpDJxih8mmYLhCUP6ouTAmMN2RDnT9tE_Gg%40mail.gmail.com

Regards,
Vignesh

  133. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-16T07:41:44Z

    On Fri, Sep 13, 2024 at 9:34 PM Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Tue, Sep 10, 2024 at 2:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna
> > <khannashubham1197@gmail.com> wrote:
> > >
> > > On Thu, Aug 29, 2024 at 11:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > > >
> > > > > > > As Euler mentioned earlier, I think it's a decision not to replicate
> > > > > > > generated columns because we don't know the target table on the
> > > > > > > subscriber has the same expression and there could be locale issues
> > > > > > > even if it looks the same. I can see that a benefit of this proposal
> > > > > > > would be to save cost to compute generated column values if the user
> > > > > > > wants the target table on the subscriber to have exactly the same data
> > > > > > > as the publisher's one. Are there other benefits or use cases?
> > > > > > >
> > > > > >
> > > > > > The cost is one but the other is the user may not want the data to be
> > > > > > different based on volatile functions like timeofday()
> > > > >
> > > > > Shouldn't the generation expression be immutable?
> > > > >
> > > >
> > > > Yes, I missed that point.
> > > >
> > > > > > or the table on
> > > > > > subscriber won't have the column marked as generated.
> > > > >
> > > > > Yeah, it would be another use case.
> > > > >
> > > >
> > > > Right, apart from that I am not aware of other use cases. If they
> > > > have, I would request Euler or Rajendra to share any other use case.
> > > >
> > > > > >  Now, considering
> > > > > > such use cases, is providing a subscription-level option a good idea
> > > > > > as the patch is doing? I understand that this can serve the purpose
> > > > > > but it could also lead to having the same behavior for all the tables
> > > > > > in all the publications for a subscription which may or may not be
> > > > > > what the user expects. This could lead to some performance overhead
> > > > > > (due to always sending generated columns for all the tables) for cases
> > > > > > where the user needs it only for a subset of tables.
> > > > >
> > > > > Yeah, it's a downside and I think it's less flexible. For example, if
> > > > > users want to send both tables with generated columns and tables
> > > > > without generated columns, they would have to create at least two
> > > > > subscriptions.
> > > > >
> > > >
> > > > Agreed and that would consume more resources.
> > > >
> > > > > Also, they would have to include a different set of
> > > > > tables to two publications.
> > > > >
> > > > > >
> > > > > > I think we should consider it as a table-level option while defining
> > > > > > publication in some way. A few ideas could be: (a) We ask users to
> > > > > > explicitly mention the generated column in the columns list while
> > > > > > defining publication. This has a drawback such that users need to
> > > > > > specify the column list even when all columns need to be replicated.
> > > > > > (b) We can have some new syntax to indicate the same like: CREATE
> > > > > > PUBLICATION pub1 FOR TABLE t1 INCLUDE GENERATED COLS, t2, t3, t4
> > > > > > INCLUDE ..., t5;. I haven't analyzed the feasibility of this, so there
> > > > > > could be some challenges but we can at least investigate it.
> > > > >
> > > > > I think we can create a publication for a single table, so what we can
> > > > > do with this feature can be done also by the idea you described below.
> > > > >
> > > > > > Yet another idea is to keep this as a publication option
> > > > > > (include_generated_columns or publish_generated_columns) similar to
> > > > > > "publish_via_partition_root". Normally, "publish_via_partition_root"
> > > > > > is used when tables on either side have different partitions
> > > > > > hierarchies which is somewhat the case here.
> > > > >
> > > > > It sounds more useful to me.
> > > > >
> > > >
> > > > Fair enough. Let's see if anyone else has any preference among the
> > > > proposed methods or can think of a better way.
> > >
> > > I have fixed the current issue. I have added the option
> > > 'publish_generated_columns' to the publisher side and created the new
> > > test cases accordingly.
> > > The attached patches contain the desired changes.
> > >
> >
> > Thank you for updating the patches. I have some comments:
> >
> > Do we really need to add this option to test_decoding? I think it
> > would be good if this improves the test coverage. Otherwise, I'm not
> > sure we need this part. If we want to add it, I think it would be
> > better to have it in a separate patch.
> >
>
> I have removed the option from the test_decoding file.
>
> > ---
> > +         <para>
> > +          If the publisher-side column is also a generated column
> > then this option
> > +          has no effect; the publisher column will be filled as normal with the
> > +          publisher-side computed or default data.
> > +         </para>
> >
> > I don't understand this description. Why does this option have no
> > effect if the publisher-side column is a generated column?
> >
>
> The documentation was incorrect. Currently, replicating from a
> publisher table with a generated column to a subscriber table with a
> generated column will result in an error. This has now been updated.
>
> > ---
> > +         <para>
> > +         This parameter can only be set <literal>true</literal> if
> > <literal>copy_data</literal> is
> > +         set to <literal>false</literal>.
> > +         </para>
> >
> > If I understand this patch correctly, it doesn't disallow to set
> > copy_data to true when the publish_generated_columns option is
> > specified. But do we want to disallow it? I think it would be more
> > useful and understandable if we allow to use both
> > publish_generated_columns (publisher option) and copy_data (subscriber
> > option) at the same time.
> >
>
> Support for tablesync with generated columns was not included in the
> initial patch, and this was reflected in the documentation. The
> functionality for syncing generated column data has been introduced
> with the 0002 patch.
>

Since nothing was said otherwise, I assumed my v30-0001 comments were
addressed in v31, but the new code seems to have quite a few of my
suggested changes missing. If you haven't addressed my review comments
for patch 0001 yet, please say so. OTOH, please give reasons for any
rejected comments.

> The attached v31 patches contain the changes for the same. I won't be
> posting the test patch for now. I will share it once this patch has
> been stabilized.

How can the patch become "stabilized" without associated tests to
verify the behaviour is not broken? e.g. I can write a stable function
that says 2+2=5.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  134. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-16T21:01:32Z

    On Wed, Sep 11, 2024 at 10:30 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Because this feature is now being implemented as a PUBLICATION option,
> there is another scenario that might need consideration; I am thinking
> about where the same table is published by multiple PUBLICATIONS (with
> different option settings) that are subscribed by a single
> SUBSCRIPTION.
>
> e.g.1
> -----
> CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> -----
>
> e.g.2
> -----
> CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> -----
>
> Do you know if this case is supported? If yes, then which publication
> option value wins?

I would expect these option values are processed with OR. That is, we
publish changes of the generated columns if at least one publication
sets publish_generated_columns to true. It seems to me that we treat
multiple row filters in the same way.

>
> The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
> several publications in which the same table has been published with
> different column lists are not supported."
>
> Perhaps the user is supposed to deduce that the example above would
> work OK if table 't1' has no generated cols. OTOH, if it did have
> generated cols then the PUBLICATION column lists must be different and
> therefore it is "not supported" (??).

With the patch, how should this feature work when users specify a
generated column to the column list and set publish_generated_column =
false, in the first place? raise an error (as we do today)? or always
send NULL?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  135. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-17T03:09:29Z

    On Tue, Sep 17, 2024 at 7:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Sep 11, 2024 at 10:30 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Because this feature is now being implemented as a PUBLICATION option,
> > there is another scenario that might need consideration; I am thinking
> > about where the same table is published by multiple PUBLICATIONS (with
> > different option settings) that are subscribed by a single
> > SUBSCRIPTION.
> >
> > e.g.1
> > -----
> > CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > -----
> >
> > e.g.2
> > -----
> > CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > -----
> >
> > Do you know if this case is supported? If yes, then which publication
> > option value wins?
>
> I would expect these option values are processed with OR. That is, we
> publish changes of the generated columns if at least one publication
> sets publish_generated_columns to true. It seems to me that we treat
> multiple row filters in the same way.
>

I thought that the option "publish_generated_columns" is more related
to "column lists" than "row filters".

Let's say table 't1' has columns 'a', 'b', 'c', 'gen1', 'gen2'.

Then:
PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
is equivalent to
PUBLICATION pub1 FOR TABLE t1(a,b,c,gen1,gen2);

And
PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
is equivalent to
PUBLICATION pub2 FOR TABLE t1(a,b,c);

So, I would expect this to fail because the SUBSCRIPTION docs say
"Subscriptions having several publications in which the same table has
been published with different column lists are not supported."

~~

Here's another example:
PUBLICATION pub3 FOR TABLE t1(a,b);
PUBLICATION pub4 FOR TABLE t1(c);

Won't it be strange (e.g. difficult to explain) why pub1 and pub2
table column lists are allowed to be combined in one subscription, but
pub3 and pub4 in one subscription are not supported due to the
different column lists?

> >
> > The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
> > several publications in which the same table has been published with
> > different column lists are not supported."
> >
> > Perhaps the user is supposed to deduce that the example above would
> > work OK if table 't1' has no generated cols. OTOH, if it did have
> > generated cols then the PUBLICATION column lists must be different and
> > therefore it is "not supported" (??).
>
> With the patch, how should this feature work when users specify a
> generated column to the column list and set publish_generated_column =
> false, in the first place? raise an error (as we do today)? or always
> send NULL?

For this scenario, I suggested (see [1] #3) that the code could give a
WARNING. As I wrote up-thread: This combination doesn't seem
like something a user would do intentionally, so just silently
ignoring it (which the current patch does) is likely going to give
someone unexpected results/grief.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPuaitgE4tu3nfaR%3DPCQEKjB%3DmpDtZ1aWkbwb%3DJZE8YvqQ%40mail.gmail.com

Kind Regards,
Peter Smith
Fujitsu Australia

  136. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-17T06:15:07Z

    On Mon, Sep 16, 2024 at 8:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> I thought that the option "publish_generated_columns" is more related
> to "column lists" than "row filters".
>
> Let's say table 't1' has columns 'a', 'b', 'c', 'gen1', 'gen2'.
>

> And
> PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> is equivalent to
> PUBLICATION pub2 FOR TABLE t1(a,b,c);

This makes sense to me as it preserves the current behavior.

> Then:
> PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> is equivalent to
> PUBLICATION pub1 FOR TABLE t1(a,b,c,gen1,gen2);

This also makes sense. It would also include future generated columns.

> So, I would expect this to fail because the SUBSCRIPTION docs say
> "Subscriptions having several publications in which the same table has
> been published with different column lists are not supported."

So I agree that it would raise an error if users subscribe to both
pub1 and pub2.

And looking back at your examples,

> > > e.g.1
> > > -----
> > > CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > -----
> > >
> > > e.g.2
> > > -----
> > > CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > -----

Both examples would not be supported.

> > >
> > > The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
> > > several publications in which the same table has been published with
> > > different column lists are not supported."
> > >
> > > Perhaps the user is supposed to deduce that the example above would
> > > work OK if table 't1' has no generated cols. OTOH, if it did have
> > > generated cols then the PUBLICATION column lists must be different and
> > > therefore it is "not supported" (??).
> >
> > With the patch, how should this feature work when users specify a
> > generated column to the column list and set publish_generated_column =
> > false, in the first place? raise an error (as we do today)? or always
> > send NULL?
>
> For this scenario, I suggested (see [1] #3) that the code could give a
> WARNING. As I wrote up-thread: This combination doesn't seem
> like something a user would do intentionally, so just silently
> ignoring it (which the current patch does) is likely going to give
> someone unexpected results/grief.

It gives a WARNING, and then publishes the specified generated column
data (even if publish_generated_column = false)? If so, it would mean
that specifying the generated column to the column list means to
publish its data regardless of the publish_generated_column parameter
value.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  137. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-17T06:34:09Z

    On Tue, Sep 17, 2024 at 4:15 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Sep 16, 2024 at 8:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > I thought that the option "publish_generated_columns" is more related
> > to "column lists" than "row filters".
> >
> > Let's say table 't1' has columns 'a', 'b', 'c', 'gen1', 'gen2'.
> >
>
> > And
> > PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > is equivalent to
> > PUBLICATION pub2 FOR TABLE t1(a,b,c);
>
> This makes sense to me as it preserves the current behavior.
>
> > Then:
> > PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > is equivalent to
> > PUBLICATION pub1 FOR TABLE t1(a,b,c,gen1,gen2);
>
> This also makes sense. It would also include future generated columns.
>
> > So, I would expect this to fail because the SUBSCRIPTION docs say
> > "Subscriptions having several publications in which the same table has
> > been published with different column lists are not supported."
>
> So I agree that it would raise an error if users subscribe to both
> pub1 and pub2.
>
> And looking back at your examples,
>
> > > > e.g.1
> > > > -----
> > > > CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > > -----
> > > >
> > > > e.g.2
> > > > -----
> > > > CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> > > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > > -----
>
> Both examples would not be supported.
>
> > > >
> > > > The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
> > > > several publications in which the same table has been published with
> > > > different column lists are not supported."
> > > >
> > > > Perhaps the user is supposed to deduce that the example above would
> > > > work OK if table 't1' has no generated cols. OTOH, if it did have
> > > > generated cols then the PUBLICATION column lists must be different and
> > > > therefore it is "not supported" (??).
> > >
> > > With the patch, how should this feature work when users specify a
> > > generated column to the column list and set publish_generated_column =
> > > false, in the first place? raise an error (as we do today)? or always
> > > send NULL?
> >
> > For this scenario, I suggested (see [1] #3) that the code could give a
> > WARNING. As I wrote up-thread: This combination doesn't seem
> > like something a user would do intentionally, so just silently
> > ignoring it (which the current patch does) is likely going to give
> > someone unexpected results/grief.
>
> It gives a WARNING, and then publishes the specified generated column
> data (even if publish_generated_column = false)? If so, it would mean
> that specifying the generated column to the column list means to
> publish its data regardless of the publish_generated_column parameter
> value.
>

No. I meant only it can give the WARNING to tell the user user  "Hey,
there is a conflict here because you said publish_generated_column=
false, but you also specified gencols in the column list".

But always it is the option "publish_generated_column" determines the
final publishing behaviour. So if it says
publish_generated_column=false then it would NOT publish generated
columns even if they are gencols in the column list. I think this
makes sense because when there is no column list specified then that
implicitly means "all columns" and the table might have some gencols,
but still 'publish_generated_columns' is what determines the
behaviour.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  138. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-17T07:43:36Z

    Here are some review comments for v31-0001 (for the docs only)

There may be some overlap here with some comments already made for
v30-0001 which are not yet addressed in v31-0001.

======
Commit message

1.
When introducing the 'publish_generated_columns' parameter, you must
also say this is a PUBLICATION parameter.

~~~

2.
With this enhancement, users can now include the 'include_generated_columns'
option when querying logical replication slots using either the pgoutput
plugin or the test_decoding plugin. This option, when set to 'true' or '1',
instructs the replication system to include generated column information
and data in the replication stream.

~

The above is stale information because it still refers to the old name
'include_generated_columns', and to test_decoding which was already
removed in this patch.

======
doc/src/sgml/ddl.sgml

3.
+      Generated columns may be skipped during logical replication
according to the
+      <command>CREATE PUBLICATION</command> option
+      <link linkend="sql-createpublication-params-with-include-generated-columns">
+      <literal>publish_generated_columns</literal></link>.

3a.
nit - The linkend is based on the old name instead of the new name.

3b.
nit - Better to call this a parameter instead of an option because
that is what the CREATE PUBLICATION docs call it.

======
doc/src/sgml/protocol.sgml

4.
+    <varlistentry>
+     <term>publish_generated_columns</term>
+      <listitem>
+       <para>
+        Boolean option to enable generated columns. This option controls
+        whether generated columns should be included in the string
+        representation of tuples during logical decoding in PostgreSQL.
+       </para>
+      </listitem>
+    </varlistentry>
+

Is this even needed anymore? Now that the implementation is using a
PUBLICATION parameter, isn't everything determined just by that
parameter? I don't see the reason why a protocol change is needed
anymore. And, if there is no protocol change needed, then this
documentation change is also not needed.

~~~~

5.
      <para>
-      Next, the following message part appears for each column included in
-      the publication (except generated columns):
+      Next, the following message parts appear for each column included in
+      the publication (generated columns are excluded unless the parameter
+      <link linkend="protocol-logical-replication-params">
+      <literal>publish_generated_columns</literal></link> specifies otherwise):
      </para>

Like the previous comment above, I think everything is now determined
by the PUBLICATION parameter. So maybe this should just be referring
to that instead.

======
doc/src/sgml/ref/create_publication.sgml

6.
+       <varlistentry
id="sql-createpublication-params-with-include-generated-columns">
+        <term><literal>publish_generated_columns</literal>
(<type>boolean</type>)</term>
+        <listitem>

nit - the ID is based on the old parameter name.

~

7.
+         <para>
+          This option is only available for replicating generated
column data from the publisher
+          to a regular, non-generated column in the subscriber.
+         </para>

IMO remove this paragraph. I really don't think you should be
mentioning the subscriber here at all. AFAIK this parameter is only
for determining if the generated column will be published or not. What
happens at the other end (e.g. logic whether it gets ignored or not by
the subscriber) is more like a matrix of behaviours that could be
documented in the "Logical Replication" section. But not here.

(I removed this in my nitpicks attachment)

~~~

8.
+         <para>
+         This parameter can only be set <literal>true</literal> if
<literal>copy_data</literal> is
+         set to <literal>false</literal>.
+         </para>

IMO remove this paragraph too. The user can create a PUBLICATION
before a SUBSCRIPTION even exists so to say it "can only be set..." is
not correct. Sure, your patch 0001 does not support the COPY of
generated columns but if you want to document that then it should be
documented in the CREATE SUBSCRIBER docs. But not here.

(I removed this in my nitpicks attachment)

TBH, it would be better if patches 0001 and 0002 were merged then you
can avoid all this. IIUC they were only separate in the first place
because 2 different people wrote them. It is not making reviews easier
with them split.

======

Please see the attachment which implements some of the nits above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  139. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-17T09:42:22Z

    Review comments for v31-0001.

(I tried to give only new comments, but there might be some overlap
with comments I previously made for v30-0001)

======
src/backend/catalog/pg_publication.c

1.
+
+ if (publish_generated_columns_given)
+ {
+ values[Anum_pg_publication_pubgencolumns - 1] =
BoolGetDatum(publish_generated_columns);
+ replaces[Anum_pg_publication_pubgencolumns - 1] = true;
+ }

nit - unnecessary whitespace above here.

======
src/backend/replication/pgoutput/pgoutput.c

2. prepare_all_columns_bms

+ /* Iterate the cols until generated columns are found. */
+ cols = bms_add_member(cols, i + 1);

How does the comment relate to the statement that follows it?

~~~

3.
+ * Skip generated column if pubgencolumns option was not
+ * specified.

nit - /pubgencolumns option/publish_generated_columns parameter/

======
src/bin/pg_dump/pg_dump.c

4.
getPublications:

nit - /i_pub_gencolumns/i_pubgencols/ (it's the same information but simpler)

======
src/bin/pg_dump/pg_dump.h

5.
+ bool pubgencolumns;
 } PublicationInfo;

nit - /pubgencolumns/pubgencols/ (it's the same information but simpler)

======
vsrc/bin/psql/describe.c

6.
  bool has_pubviaroot;
+ bool has_pubgencol;

nit - /has_pubgencol/has_pubgencols/ (plural consistency)

======
src/include/catalog/pg_publication.h

7.
+ /* true if generated columns data should be published */
+ bool pubgencolumns;
 } FormData_pg_publication;

nit - /pubgencolumns/pubgencols/ (it's the same information but simpler)

~~~

8.
+ bool pubgencolumns;
  PublicationActions pubactions;
 } Publication;

nit - /pubgencolumns/pubgencols/ (it's the same information but simpler)

======
src/test/regress/sql/publication.sql

9.
+-- Test the publication with or without 'PUBLISH_GENERATED_COLUMNS' parameter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION pub1 FOR ALL TABLES WITH (PUBLISH_GENERATED_COLUMNS=1);
+\dRp+ pub1
+
+CREATE PUBLICATION pub2 FOR ALL TABLES WITH (PUBLISH_GENERATED_COLUMNS=0);
+\dRp+ pub2

9a.
nit - Use lowercase for the parameters.

~

9b.
nit - Fix the comment to say what the test is actually doing:
"Test the publication 'publish_generated_columns' parameter enabled or disabled"

======
src/test/subscription/t/031_column_list.pl

10.
Later I think you should add another test here to cover the scenario
that I was discussing with Sawada-San -- e.g. when there are 2
publications for the same table subscribed by just 1 subscription but
having different values of the 'publish_generated_columns' for the
publications.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  140. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-18T03:28:08Z

    Hi, here are my review comments for patch v31-0002.

======

1. General.

IMO patches 0001 and 0002 should be merged when next posted. IIUC the
reason for the split was only because there were 2 different authors
but that seems to be not relevant anymore.

======
Commit message

2.
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'publish_generated_columns'
is true, we need to copy using the syntax:
'COPY (SELECT column_name FROM table_name) TO STDOUT'.

~

2a.
Should clarify that 'copy_data' is a SUBSCRIPTION parameter.

2b.
Should clarify that 'publish_generated_columns' is a PUBLICATION parameter.

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:

3.
- for (i = 0; i < rel->remoterel.natts; i++)
+ desc = RelationGetDescr(rel->localrel);
+ localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));

Each time I review this code I am tricked into thinking it is wrong to
use rel->remoterel.natts here for the localgenlist. AFAICT the code is
actually fine because you do not store *all* the subscriber gencols in
'localgenlist' -- you only store those with matching names on the
publisher table. It might be good if you could add an explanatory
comment about that to prevent any future doubts.

~~~

4.
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a
generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));

This error message has lots of good information. OTOH, I think when
copy_data=false the error would report the subscriber column just as
"missing", which is maybe less helpful. Perhaps that other
copy_data=false "missing" case can be improved to share the same error
message that you have here.

~~~

fetch_remote_table_info:

5.
IIUC, this logic needs to be more sophisticated to handle the case
that was being discussed earlier with Sawada-san [1]. e.g. when the
same table has gencols but there are multiple subscribed publications
where the 'publish_generated_columns' parameter differs.

Also, you'll need test cases for this scenario, because it is too
difficult to judge correctness just by visual inspection of the code.

~~~~

6.
nit - Change 'hasgencolpub' to 'has_pub_with_pubgencols' for
readability, and initialize it to 'false' to make it easy to use
later.

~~~

7.
- * Get column lists for each relation.
+ * Get column lists for each relation and check if any of the publication
+ * has generated column option.

and

+ /* Check if any of the publication has generated column option */
+ if (server_version >= 180000)

nit - tweak the comments to name the publication parameter properly.

~~~

8.
foreach(lc, MySubscription->publications)
{
if (foreach_current_index(lc) > 0)
appendStringInfoString(&pub_names, ", ");
appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
}

I know this is existing code, but shouldn't all this be done by using
the purpose-built function 'get_publications_str'

~~~

9.
+ ereport(ERROR,
+ errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch gencolumns information from publication list: %s",
+    pub_names.data));

and

+ errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("failed to fetch tuple for gencols from publication list: %s",
+    pub_names.data));

nit - /gencolumns information/generated column publication
information/ to make the errmsg more human-readable

~~~

10.
+ bool gencols_allowed = server_version >= 180000 && hasgencolpub;
+
+ if (!gencols_allowed)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");

Can the 'gencols_allowed' var be removed, and the condition just be
replaced with if (!has_pub_with_pubgencols)? It seems equivalent
unless I am mistaken.

======

Please refer to the attachment which implements some of the nits
mentioned above.

======
[1] https://www.postgresql.org/message-id/CAD21AoBun9crSWaxteMqyu8A_zme2ppa2uJvLJSJC2E3DJxQVA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  141. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-19T08:40:57Z

    On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > >
> > > As Euler mentioned earlier, I think it's a decision not to replicate
> > > generated columns because we don't know the target table on the
> > > subscriber has the same expression and there could be locale issues
> > > even if it looks the same. I can see that a benefit of this proposal
> > > would be to save cost to compute generated column values if the user
> > > wants the target table on the subscriber to have exactly the same data
> > > as the publisher's one. Are there other benefits or use cases?
> > >
> >
> > The cost is one but the other is the user may not want the data to be
> > different based on volatile functions like timeofday()
>
> Shouldn't the generation expression be immutable?
>
> > or the table on
> > subscriber won't have the column marked as generated.
>
> Yeah, it would be another use case.
>

While speaking with one of the decoding output plugin users, I learned
that this feature will be useful when replicating data to a
non-postgres database using the plugin output, especially when the
other database doesn't have a generated column concept.

-- 
With Regards,
Amit Kapila.

  142. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-19T09:32:25Z

    On Tue, Sep 17, 2024 at 12:04 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Sep 17, 2024 at 4:15 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Sep 16, 2024 at 8:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > I thought that the option "publish_generated_columns" is more related
> > > to "column lists" than "row filters".
> > >
> > > Let's say table 't1' has columns 'a', 'b', 'c', 'gen1', 'gen2'.
> > >
> >
> > > And
> > > PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > is equivalent to
> > > PUBLICATION pub2 FOR TABLE t1(a,b,c);
> >
> > This makes sense to me as it preserves the current behavior.
> >
> > > Then:
> > > PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > > is equivalent to
> > > PUBLICATION pub1 FOR TABLE t1(a,b,c,gen1,gen2);
> >
> > This also makes sense. It would also include future generated columns.
> >
> > > So, I would expect this to fail because the SUBSCRIPTION docs say
> > > "Subscriptions having several publications in which the same table has
> > > been published with different column lists are not supported."
> >
> > So I agree that it would raise an error if users subscribe to both
> > pub1 and pub2.
> >
> > And looking back at your examples,
> >
> > > > > e.g.1
> > > > > -----
> > > > > CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > > > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > > > -----
> > > > >
> > > > > e.g.2
> > > > > -----
> > > > > CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> > > > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > > > -----
> >
> > Both examples would not be supported.
> >
> > > > >
> > > > > The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
> > > > > several publications in which the same table has been published with
> > > > > different column lists are not supported."
> > > > >
> > > > > Perhaps the user is supposed to deduce that the example above would
> > > > > work OK if table 't1' has no generated cols. OTOH, if it did have
> > > > > generated cols then the PUBLICATION column lists must be different and
> > > > > therefore it is "not supported" (??).
> > > >
> > > > With the patch, how should this feature work when users specify a
> > > > generated column to the column list and set publish_generated_column =
> > > > false, in the first place? raise an error (as we do today)? or always
> > > > send NULL?
> > >
> > > For this scenario, I suggested (see [1] #3) that the code could give a
> > > WARNING. As I wrote up-thread: This combination doesn't seem
> > > like something a user would do intentionally, so just silently
> > > ignoring it (which the current patch does) is likely going to give
> > > someone unexpected results/grief.
> >
> > It gives a WARNING, and then publishes the specified generated column
> > data (even if publish_generated_column = false)?


I think that the column list should take priority and we should
publish the generated column if it is mentioned in  irrespective of
the option.

> > If so, it would mean
> > that specifying the generated column to the column list means to
> > publish its data regardless of the publish_generated_column parameter
> > value.
> >
>
> No. I meant only it can give the WARNING to tell the user user  "Hey,
> there is a conflict here because you said publish_generated_column=
> false, but you also specified gencols in the column list".
>

Users can use a publication like "create publication pub1 for table
t1(c1, c2), t2;" where they want t1's generated column to be published
but not for t2. They can specify the generated column name in the
column list of t1 in that case even though the rest of the tables
won't publish generated columns.

-- 
With Regards,
Amit Kapila.

  143. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-19T17:25:42Z

    On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Sep 17, 2024 at 12:04 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Tue, Sep 17, 2024 at 4:15 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, Sep 16, 2024 at 8:09 PM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > I thought that the option "publish_generated_columns" is more related
> > > > to "column lists" than "row filters".
> > > >
> > > > Let's say table 't1' has columns 'a', 'b', 'c', 'gen1', 'gen2'.
> > > >
> > >
> > > > And
> > > > PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > is equivalent to
> > > > PUBLICATION pub2 FOR TABLE t1(a,b,c);
> > >
> > > This makes sense to me as it preserves the current behavior.
> > >
> > > > Then:
> > > > PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > > > is equivalent to
> > > > PUBLICATION pub1 FOR TABLE t1(a,b,c,gen1,gen2);
> > >
> > > This also makes sense. It would also include future generated columns.
> > >
> > > > So, I would expect this to fail because the SUBSCRIPTION docs say
> > > > "Subscriptions having several publications in which the same table has
> > > > been published with different column lists are not supported."
> > >
> > > So I agree that it would raise an error if users subscribe to both
> > > pub1 and pub2.
> > >
> > > And looking back at your examples,
> > >
> > > > > > e.g.1
> > > > > > -----
> > > > > > CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> > > > > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > > > > -----
> > > > > >
> > > > > > e.g.2
> > > > > > -----
> > > > > > CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> > > > > > CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> > > > > > CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> > > > > > -----
> > >
> > > Both examples would not be supported.
> > >
> > > > > >
> > > > > > The CREATE SUBSCRIPTION docs [1] only says "Subscriptions having
> > > > > > several publications in which the same table has been published with
> > > > > > different column lists are not supported."
> > > > > >
> > > > > > Perhaps the user is supposed to deduce that the example above would
> > > > > > work OK if table 't1' has no generated cols. OTOH, if it did have
> > > > > > generated cols then the PUBLICATION column lists must be different and
> > > > > > therefore it is "not supported" (??).
> > > > >
> > > > > With the patch, how should this feature work when users specify a
> > > > > generated column to the column list and set publish_generated_column =
> > > > > false, in the first place? raise an error (as we do today)? or always
> > > > > send NULL?
> > > >
> > > > For this scenario, I suggested (see [1] #3) that the code could give a
> > > > WARNING. As I wrote up-thread: This combination doesn't seem
> > > > like something a user would do intentionally, so just silently
> > > > ignoring it (which the current patch does) is likely going to give
> > > > someone unexpected results/grief.
> > >
> > > It gives a WARNING, and then publishes the specified generated column
> > > data (even if publish_generated_column = false)?
>
>
> I think that the column list should take priority and we should
> publish the generated column if it is mentioned in  irrespective of
> the option.

Agreed.

>
> > > If so, it would mean
> > > that specifying the generated column to the column list means to
> > > publish its data regardless of the publish_generated_column parameter
> > > value.
> > >
> >
> > No. I meant only it can give the WARNING to tell the user user  "Hey,
> > there is a conflict here because you said publish_generated_column=
> > false, but you also specified gencols in the column list".
> >
>
> Users can use a publication like "create publication pub1 for table
> t1(c1, c2), t2;" where they want t1's generated column to be published
> but not for t2. They can specify the generated column name in the
> column list of t1 in that case even though the rest of the tables
> won't publish generated columns.

Agreed.

I think that users can use the publish_generated_column option when
they want to publish all generated columns, instead of specifying all
the columns in the column list. It's another advantage of this option
that it will also include the future generated columns.

Given that we publish the generated columns if they are mentioned in
the column list, can we separate the patch into two if it helps
reviews? One is to allow logical replication to publish generated
columns if they are explicitly mentioned in the column list. The
second patch is to introduce the publish_generated_columns option.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  144. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-19T22:46:01Z

    On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
...
> > I think that the column list should take priority and we should
> > publish the generated column if it is mentioned in  irrespective of
> > the option.
>
> Agreed.
>
> >
...
> >
> > Users can use a publication like "create publication pub1 for table
> > t1(c1, c2), t2;" where they want t1's generated column to be published
> > but not for t2. They can specify the generated column name in the
> > column list of t1 in that case even though the rest of the tables
> > won't publish generated columns.
>
> Agreed.
>
> I think that users can use the publish_generated_column option when
> they want to publish all generated columns, instead of specifying all
> the columns in the column list. It's another advantage of this option
> that it will also include the future generated columns.
>

OK. Let me give some examples below to help understand this idea.

Please correct me if these are incorrect.

======

Assuming these tables:

t1(a,b,gen1,gen2)
t2(c,d,gen1,gen2)

Examples, when publish_generated_columns=false:

CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
(publish_generated_columns=false)
t1 -> publishes a, b, gen2 (e.g. what column list says)
t2 -> publishes c, d

CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=false)
t1 -> publishes a, b
t2 -> publishes gen1 (e.g. what column list says)

CREATE PUBLICATION pub1 FOR t1, t2 WITH (publish_generated_columns=false)
t1 -> publishes a, b
t2 -> publishes c, d

CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns=false)
t1 -> publishes a, b
t2 -> publishes c, d

~~

Examples, when publish_generated_columns=true:

CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
(publish_generated_columns=true)
t1 -> publishes a, b, gen2 (e.g. what column list says)
t2 -> publishes c, d + ALSO gen1, gen2

CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
t1 -> publishes a, b + ALSO gen1, gen2
t2 -> publishes gen1 (e.g. what column list says)

CREATE PUBLICATION pub1 FOR t1, t2 WITH (publish_generated_columns=true)
t1 -> publishes a, b + ALSO gen1, gen2
t2 -> publishes c, d + ALSO gen1, gen2

CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns=true)
t1 -> publishes a, b + ALSO gen1, gen2
t2 -> publishes c, d + ALSO gen1, gen2

======

The idea LGTM, although now the parameter name
('publish_generated_columns') seems a bit misleading since sometimes
generated columns get published "irrespective of the option".

So, I think the original parameter name 'include_generated_columns'
might be better here because IMO "include" seems more like "add them
if they are not already specified", which is exactly what this idea is
doing.

Thoughts?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  145. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-20T04:25:56Z

    On Fri, Sep 20, 2024 at 4:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Users can use a publication like "create publication pub1 for table
> > > t1(c1, c2), t2;" where they want t1's generated column to be published
> > > but not for t2. They can specify the generated column name in the
> > > column list of t1 in that case even though the rest of the tables
> > > won't publish generated columns.
> >
> > Agreed.
> >
> > I think that users can use the publish_generated_column option when
> > they want to publish all generated columns, instead of specifying all
> > the columns in the column list. It's another advantage of this option
> > that it will also include the future generated columns.
> >
>
> OK. Let me give some examples below to help understand this idea.
>
> Please correct me if these are incorrect.
>
> Examples, when publish_generated_columns=true:
>
> CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> (publish_generated_columns=true)
> t1 -> publishes a, b, gen2 (e.g. what column list says)
> t2 -> publishes c, d + ALSO gen1, gen2
>
> CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
> t1 -> publishes a, b + ALSO gen1, gen2
> t2 -> publishes gen1 (e.g. what column list says)
>

These two could be controversial because one could expect that if
"publish_generated_columns=true" then publish generated columns
irrespective of whether they are mentioned in column_list. I am of the
opinion that column_list should take priority the results should be as
mentioned by you but let us see if anyone thinks otherwise.

>
> ======
>
> The idea LGTM, although now the parameter name
> ('publish_generated_columns') seems a bit misleading since sometimes
> generated columns get published "irrespective of the option".
>
> So, I think the original parameter name 'include_generated_columns'
> might be better here because IMO "include" seems more like "add them
> if they are not already specified", which is exactly what this idea is
> doing.
>

I still prefer 'publish_generated_columns' because it matches with
other publication option names. One can also deduce from
'include_generated_columns' that add all the generated columns even
when some of them are specified in column_list.

-- 
With Regards,
Amit Kapila.

  146. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-20T04:32:06Z

    On Thu, Sep 19, 2024 at 10:56 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
>
> Given that we publish the generated columns if they are mentioned in
> the column list, can we separate the patch into two if it helps
> reviews? One is to allow logical replication to publish generated
> columns if they are explicitly mentioned in the column list. The
> second patch is to introduce the publish_generated_columns option.
>

It sounds like a reasonable idea to me but I haven't looked at the
feasibility of the same. So, if it is possible without much effort, we
should split the patch as per your suggestion.

-- 
With Regards,
Amit Kapila.

  147. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-20T11:45:27Z

    On Wed, Sep 11, 2024 at 8:55 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are a some more review comments for patch v30-0001.
>
> ======
> src/sgml/ref/create_publication.sgml
>
> 1.
> +         <para>
> +          If the publisher-side column is also a generated column
> then this option
> +          has no effect; the publisher column will be filled as normal with the
> +          publisher-side computed or default data.
> +         </para>
>
> It should say "subscriber-side"; not "publisher-side". The same was
> already reported by Sawada-San [1].
>
> ~~~
>
> 2.
> +         <para>
> +         This parameter can only be set <literal>true</literal> if
> <literal>copy_data</literal> is
> +         set to <literal>false</literal>.
> +         </para>
>
> IMO this limitation should be addressed by patch 0001 like it was
> already done in the previous patches (e.g. v22-0002). I think
> Sawada-san suggested the same [1].
>
> Anyway, 'copy_data' is not a PUBLICATION option, so the fact it is
> mentioned like this without any reference to the SUBSCRIPTION seems
> like a cut/paste error from the previous implementation.
>
> ======
> src/backend/catalog/pg_publication.c
>
> 3. pub_collist_validate
> - if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> - ereport(ERROR,
> - errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> - errmsg("cannot use generated column \"%s\" in publication column list",
> -    colname));
> -
>
> Instead of just removing this ERROR entirely here, I thought it would
> be more user-friendly to give a WARNING if the PUBLICATION's explicit
> column list includes generated cols when the option
> "publish_generated_columns" is false. This combination doesn't seem
> like something a user would do intentionally, so just silently
> ignoring it (like the current patch does) is likely going to give
> someone unexpected results/grief.
>
> ======
> src/backend/replication/logical/proto.c
>
> 4. logicalrep_write_tuple, and logicalrep_write_attrs:
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
> Why aren't you also checking the new PUBLICATION option here and
> skipping all gencols if the "publish_generated_columns" option is
> false? Or is the BMS of pgoutput_column_list_init handling this case?
> Maybe there should be an Assert for this?
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 5. send_relation_and_attrs
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
> Same question as #4.
>
> ~~~
>
> 6. prepare_all_columns_bms and pgoutput_column_list_init
>
> + if (att->attgenerated && !pub->pubgencolumns)
> + cols = bms_del_member(cols, i + 1);
>
> IIUC, the algorithm seems overly tricky filling the BMS with all
> columns, before straight away conditionally removing the generated
> columns. Can't it be refactored to assign all the correct columns
> up-front, to avoid calling bms_del_member()?
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> 7. getPublications
>
> IIUC, there is lots of missing SQL code here (for all older versions)
> that should be saying "false AS pubgencolumns".
> e.g. compare the SQL with how "false AS pubviaroot" is used.
>
> ======
> src/bin/pg_dump/t/002_pg_dump.pl
>
> 8. Missing tests?
>
> I expected to see a pg_dump test for this new PUBLICATION option.
>
> ======
> src/test/regress/sql/publication.sql
>
> 9. Missing tests?
>
> How about adding another test case that checks this new option must be
> "Boolean"?
>
> ~~~
>
> 10. Missing tests?
>
> --- error: generated column "d" can't be in list
> +-- ok: generated columns can be in the list too
>  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
> +ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
>
> (see my earlier comment #3)
>
> IMO there should be another test case for a WARNING here if the user
> attempts to include generated column 'd' in an explicit PUBLICATION
> column list while the "publish_generated-columns" is false.
>
> ======
> [1]  https://www.postgresql.org/message-id/CAD21AoA-tdTz0G-vri8KM2TXeFU8RCDsOpBXUBCgwkfokF7%3DjA%40mail.gmail.com
>

I have fixed all the comments. The attached patches contain the desired changes.
Also the merging of 0001 and 0002 can be done once there are no
comments on the patch to help in reviewing.

Thanks and Regards,
Shubham Khanna.

  148. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-20T11:51:59Z

    On Tue, Sep 17, 2024 at 1:14 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for v31-0001 (for the docs only)
>
> There may be some overlap here with some comments already made for
> v30-0001 which are not yet addressed in v31-0001.
>
> ======
> Commit message
>
> 1.
> When introducing the 'publish_generated_columns' parameter, you must
> also say this is a PUBLICATION parameter.
>
> ~~~
>
> 2.
> With this enhancement, users can now include the 'include_generated_columns'
> option when querying logical replication slots using either the pgoutput
> plugin or the test_decoding plugin. This option, when set to 'true' or '1',
> instructs the replication system to include generated column information
> and data in the replication stream.
>
> ~
>
> The above is stale information because it still refers to the old name
> 'include_generated_columns', and to test_decoding which was already
> removed in this patch.
>
> ======
> doc/src/sgml/ddl.sgml
>
> 3.
> +      Generated columns may be skipped during logical replication
> according to the
> +      <command>CREATE PUBLICATION</command> option
> +      <link linkend="sql-createpublication-params-with-include-generated-columns">
> +      <literal>publish_generated_columns</literal></link>.
>
> 3a.
> nit - The linkend is based on the old name instead of the new name.
>
> 3b.
> nit - Better to call this a parameter instead of an option because
> that is what the CREATE PUBLICATION docs call it.
>
> ======
> doc/src/sgml/protocol.sgml
>
> 4.
> +    <varlistentry>
> +     <term>publish_generated_columns</term>
> +      <listitem>
> +       <para>
> +        Boolean option to enable generated columns. This option controls
> +        whether generated columns should be included in the string
> +        representation of tuples during logical decoding in PostgreSQL.
> +       </para>
> +      </listitem>
> +    </varlistentry>
> +
>
> Is this even needed anymore? Now that the implementation is using a
> PUBLICATION parameter, isn't everything determined just by that
> parameter? I don't see the reason why a protocol change is needed
> anymore. And, if there is no protocol change needed, then this
> documentation change is also not needed.
>
> ~~~~
>
> 5.
>       <para>
> -      Next, the following message part appears for each column included in
> -      the publication (except generated columns):
> +      Next, the following message parts appear for each column included in
> +      the publication (generated columns are excluded unless the parameter
> +      <link linkend="protocol-logical-replication-params">
> +      <literal>publish_generated_columns</literal></link> specifies otherwise):
>       </para>
>
> Like the previous comment above, I think everything is now determined
> by the PUBLICATION parameter. So maybe this should just be referring
> to that instead.
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 6.
> +       <varlistentry
> id="sql-createpublication-params-with-include-generated-columns">
> +        <term><literal>publish_generated_columns</literal>
> (<type>boolean</type>)</term>
> +        <listitem>
>
> nit - the ID is based on the old parameter name.
>
> ~
>
> 7.
> +         <para>
> +          This option is only available for replicating generated
> column data from the publisher
> +          to a regular, non-generated column in the subscriber.
> +         </para>
>
> IMO remove this paragraph. I really don't think you should be
> mentioning the subscriber here at all. AFAIK this parameter is only
> for determining if the generated column will be published or not. What
> happens at the other end (e.g. logic whether it gets ignored or not by
> the subscriber) is more like a matrix of behaviours that could be
> documented in the "Logical Replication" section. But not here.
>
> (I removed this in my nitpicks attachment)
>
> ~~~
>
> 8.
> +         <para>
> +         This parameter can only be set <literal>true</literal> if
> <literal>copy_data</literal> is
> +         set to <literal>false</literal>.
> +         </para>
>
> IMO remove this paragraph too. The user can create a PUBLICATION
> before a SUBSCRIPTION even exists so to say it "can only be set..." is
> not correct. Sure, your patch 0001 does not support the COPY of
> generated columns but if you want to document that then it should be
> documented in the CREATE SUBSCRIBER docs. But not here.
>
> (I removed this in my nitpicks attachment)
>
> TBH, it would be better if patches 0001 and 0002 were merged then you
> can avoid all this. IIUC they were only separate in the first place
> because 2 different people wrote them. It is not making reviews easier
> with them split.
>
> ======
>
> Please see the attachment which implements some of the nits above.
>

I have addressed all the comments in the v32-0001 Patch. Please refer
to the updated v32-0001 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjKkoaS1oMsFvPRFB9nPSVC5p_D4Kgq5XB9Y2B2xU7smbA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  149. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-20T11:53:50Z

    On Tue, Sep 17, 2024 at 3:12 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Review comments for v31-0001.
>
> (I tried to give only new comments, but there might be some overlap
> with comments I previously made for v30-0001)
>
> ======
> src/backend/catalog/pg_publication.c
>
> 1.
> +
> + if (publish_generated_columns_given)
> + {
> + values[Anum_pg_publication_pubgencolumns - 1] =
> BoolGetDatum(publish_generated_columns);
> + replaces[Anum_pg_publication_pubgencolumns - 1] = true;
> + }
>
> nit - unnecessary whitespace above here.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 2. prepare_all_columns_bms
>
> + /* Iterate the cols until generated columns are found. */
> + cols = bms_add_member(cols, i + 1);
>
> How does the comment relate to the statement that follows it?
>
> ~~~
>
> 3.
> + * Skip generated column if pubgencolumns option was not
> + * specified.
>
> nit - /pubgencolumns option/publish_generated_columns parameter/
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> 4.
> getPublications:
>
> nit - /i_pub_gencolumns/i_pubgencols/ (it's the same information but simpler)
>
> ======
> src/bin/pg_dump/pg_dump.h
>
> 5.
> + bool pubgencolumns;
>  } PublicationInfo;
>
> nit - /pubgencolumns/pubgencols/ (it's the same information but simpler)
>
> ======
> vsrc/bin/psql/describe.c
>
> 6.
>   bool has_pubviaroot;
> + bool has_pubgencol;
>
> nit - /has_pubgencol/has_pubgencols/ (plural consistency)
>
> ======
> src/include/catalog/pg_publication.h
>
> 7.
> + /* true if generated columns data should be published */
> + bool pubgencolumns;
>  } FormData_pg_publication;
>
> nit - /pubgencolumns/pubgencols/ (it's the same information but simpler)
>
> ~~~
>
> 8.
> + bool pubgencolumns;
>   PublicationActions pubactions;
>  } Publication;
>
> nit - /pubgencolumns/pubgencols/ (it's the same information but simpler)
>
> ======
> src/test/regress/sql/publication.sql
>
> 9.
> +-- Test the publication with or without 'PUBLISH_GENERATED_COLUMNS' parameter
> +SET client_min_messages = 'ERROR';
> +CREATE PUBLICATION pub1 FOR ALL TABLES WITH (PUBLISH_GENERATED_COLUMNS=1);
> +\dRp+ pub1
> +
> +CREATE PUBLICATION pub2 FOR ALL TABLES WITH (PUBLISH_GENERATED_COLUMNS=0);
> +\dRp+ pub2
>
> 9a.
> nit - Use lowercase for the parameters.
>
> ~
>
> 9b.
> nit - Fix the comment to say what the test is actually doing:
> "Test the publication 'publish_generated_columns' parameter enabled or disabled"
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 10.
> Later I think you should add another test here to cover the scenario
> that I was discussing with Sawada-San -- e.g. when there are 2
> publications for the same table subscribed by just 1 subscription but
> having different values of the 'publish_generated_columns' for the
> publications.
>

I have addressed all the comments in the v32-0001 Patch. Please refer
to the updated v32-0001 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjKkoaS1oMsFvPRFB9nPSVC5p_D4Kgq5XB9Y2B2xU7smbA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  150. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-20T11:55:57Z

    On Wed, Sep 18, 2024 at 8:58 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for patch v31-0002.
>
> ======
>
> 1. General.
>
> IMO patches 0001 and 0002 should be merged when next posted. IIUC the
> reason for the split was only because there were 2 different authors
> but that seems to be not relevant anymore.
>
> ======
> Commit message
>
> 2.
> When 'copy_data' is true, during the initial sync, the data is replicated from
> the publisher to the subscriber using the COPY command. The normal COPY
> command does not copy generated columns, so when 'publish_generated_columns'
> is true, we need to copy using the syntax:
> 'COPY (SELECT column_name FROM table_name) TO STDOUT'.
>
> ~
>
> 2a.
> Should clarify that 'copy_data' is a SUBSCRIPTION parameter.
>
> 2b.
> Should clarify that 'publish_generated_columns' is a PUBLICATION parameter.
>
> ======
> src/backend/replication/logical/tablesync.c
>
> make_copy_attnamelist:
>
> 3.
> - for (i = 0; i < rel->remoterel.natts; i++)
> + desc = RelationGetDescr(rel->localrel);
> + localgenlist = palloc0(rel->remoterel.natts * sizeof(bool));
>
> Each time I review this code I am tricked into thinking it is wrong to
> use rel->remoterel.natts here for the localgenlist. AFAICT the code is
> actually fine because you do not store *all* the subscriber gencols in
> 'localgenlist' -- you only store those with matching names on the
> publisher table. It might be good if you could add an explanatory
> comment about that to prevent any future doubts.
>
> ~~~
>
> 4.
> + if (!remotegenlist[remote_attnum])
> + ereport(ERROR,
> + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("logical replication target relation \"%s.%s\" has a
> generated column \"%s\" "
> + "but corresponding column on source relation is not a generated column",
> + rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
>
> This error message has lots of good information. OTOH, I think when
> copy_data=false the error would report the subscriber column just as
> "missing", which is maybe less helpful. Perhaps that other
> copy_data=false "missing" case can be improved to share the same error
> message that you have here.
>

This comment is still open. Will fix this in the next set of patches.

> ~~~
>
> fetch_remote_table_info:
>
> 5.
> IIUC, this logic needs to be more sophisticated to handle the case
> that was being discussed earlier with Sawada-san [1]. e.g. when the
> same table has gencols but there are multiple subscribed publications
> where the 'publish_generated_columns' parameter differs.
>
> Also, you'll need test cases for this scenario, because it is too
> difficult to judge correctness just by visual inspection of the code.
>
> ~~~~
>
> 6.
> nit - Change 'hasgencolpub' to 'has_pub_with_pubgencols' for
> readability, and initialize it to 'false' to make it easy to use
> later.
>
> ~~~
>
> 7.
> - * Get column lists for each relation.
> + * Get column lists for each relation and check if any of the publication
> + * has generated column option.
>
> and
>
> + /* Check if any of the publication has generated column option */
> + if (server_version >= 180000)
>
> nit - tweak the comments to name the publication parameter properly.
>
> ~~~
>
> 8.
> foreach(lc, MySubscription->publications)
> {
> if (foreach_current_index(lc) > 0)
> appendStringInfoString(&pub_names, ", ");
> appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
> }
>
> I know this is existing code, but shouldn't all this be done by using
> the purpose-built function 'get_publications_str'
>
> ~~~
>
> 9.
> + ereport(ERROR,
> + errcode(ERRCODE_CONNECTION_FAILURE),
> + errmsg("could not fetch gencolumns information from publication list: %s",
> +    pub_names.data));
>
> and
>
> + errcode(ERRCODE_UNDEFINED_OBJECT),
> + errmsg("failed to fetch tuple for gencols from publication list: %s",
> +    pub_names.data));
>
> nit - /gencolumns information/generated column publication
> information/ to make the errmsg more human-readable
>
> ~~~
>
> 10.
> + bool gencols_allowed = server_version >= 180000 && hasgencolpub;
> +
> + if (!gencols_allowed)
> + appendStringInfo(&cmd, " AND a.attgenerated = ''");
>
> Can the 'gencols_allowed' var be removed, and the condition just be
> replaced with if (!has_pub_with_pubgencols)? It seems equivalent
> unless I am mistaken.
>
> ======
>
> Please refer to the attachment which implements some of the nits
> mentioned above.
>
> ======
> [1] https://www.postgresql.org/message-id/CAD21AoBun9crSWaxteMqyu8A_zme2ppa2uJvLJSJC2E3DJxQVA%40mail.gmail.com
>

I have addressed the comments in the v32-0002 Patch. Please refer to
the updated v32-0002 Patch here in [1]. See [1] for the changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjKkoaS1oMsFvPRFB9nPSVC5p_D4Kgq5XB9Y2B2xU7smbA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  151. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-20T21:49:17Z

    On Thu, Sep 19, 2024 at 9:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 20, 2024 at 4:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Users can use a publication like "create publication pub1 for table
> > > > t1(c1, c2), t2;" where they want t1's generated column to be published
> > > > but not for t2. They can specify the generated column name in the
> > > > column list of t1 in that case even though the rest of the tables
> > > > won't publish generated columns.
> > >
> > > Agreed.
> > >
> > > I think that users can use the publish_generated_column option when
> > > they want to publish all generated columns, instead of specifying all
> > > the columns in the column list. It's another advantage of this option
> > > that it will also include the future generated columns.
> > >
> >
> > OK. Let me give some examples below to help understand this idea.
> >
> > Please correct me if these are incorrect.
> >
> > Examples, when publish_generated_columns=true:
> >
> > CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> > (publish_generated_columns=true)
> > t1 -> publishes a, b, gen2 (e.g. what column list says)
> > t2 -> publishes c, d + ALSO gen1, gen2
> >
> > CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
> > t1 -> publishes a, b + ALSO gen1, gen2
> > t2 -> publishes gen1 (e.g. what column list says)
> >
>
> These two could be controversial because one could expect that if
> "publish_generated_columns=true" then publish generated columns
> irrespective of whether they are mentioned in column_list. I am of the
> opinion that column_list should take priority the results should be as
> mentioned by you but let us see if anyone thinks otherwise.

I agree with Amit. We also publish t2's future generated column in the
first example and t1's future generated columns in the second example.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  152. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-22T22:39:53Z

    On Fri, Sep 20, 2024 at 2:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 20, 2024 at 4:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Users can use a publication like "create publication pub1 for table
> > > > t1(c1, c2), t2;" where they want t1's generated column to be published
> > > > but not for t2. They can specify the generated column name in the
> > > > column list of t1 in that case even though the rest of the tables
> > > > won't publish generated columns.
> > >
> > > Agreed.
> > >
> > > I think that users can use the publish_generated_column option when
> > > they want to publish all generated columns, instead of specifying all
> > > the columns in the column list. It's another advantage of this option
> > > that it will also include the future generated columns.
> > >
> >
> > OK. Let me give some examples below to help understand this idea.
> >
> > Please correct me if these are incorrect.
> >
> > Examples, when publish_generated_columns=true:
> >
> > CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> > (publish_generated_columns=true)
> > t1 -> publishes a, b, gen2 (e.g. what column list says)
> > t2 -> publishes c, d + ALSO gen1, gen2
> >
> > CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
> > t1 -> publishes a, b + ALSO gen1, gen2
> > t2 -> publishes gen1 (e.g. what column list says)
> >
>
> These two could be controversial because one could expect that if
> "publish_generated_columns=true" then publish generated columns
> irrespective of whether they are mentioned in column_list. I am of the
> opinion that column_list should take priority the results should be as
> mentioned by you but let us see if anyone thinks otherwise.
>
> >
> > ======
> >
> > The idea LGTM, although now the parameter name
> > ('publish_generated_columns') seems a bit misleading since sometimes
> > generated columns get published "irrespective of the option".
> >
> > So, I think the original parameter name 'include_generated_columns'
> > might be better here because IMO "include" seems more like "add them
> > if they are not already specified", which is exactly what this idea is
> > doing.
> >
>
> I still prefer 'publish_generated_columns' because it matches with
> other publication option names. One can also deduce from
> 'include_generated_columns' that add all the generated columns even
> when some of them are specified in column_list.
>

Fair point. Anyway, to avoid surprises it will be important for the
precedence rules to be documented clearly (probably with some
examples),

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  153. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-23T02:45:48Z

    On Sat, Sep 21, 2024 at 3:19 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Sep 19, 2024 at 9:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > >
> > > OK. Let me give some examples below to help understand this idea.
> > >
> > > Please correct me if these are incorrect.
> > >
> > > Examples, when publish_generated_columns=true:
> > >
> > > CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> > > (publish_generated_columns=true)
> > > t1 -> publishes a, b, gen2 (e.g. what column list says)
> > > t2 -> publishes c, d + ALSO gen1, gen2
> > >
> > > CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
> > > t1 -> publishes a, b + ALSO gen1, gen2
> > > t2 -> publishes gen1 (e.g. what column list says)
> > >
> >
> > These two could be controversial because one could expect that if
> > "publish_generated_columns=true" then publish generated columns
> > irrespective of whether they are mentioned in column_list. I am of the
> > opinion that column_list should take priority the results should be as
> > mentioned by you but let us see if anyone thinks otherwise.
>
> I agree with Amit. We also publish t2's future generated column in the
> first example and t1's future generated columns in the second example.
>

Right, it would be good to have at least one test that shows future
generated columns also get published wherever applicable (like where
column_list is not given and publish_generated_columns is true).

-- 
With Regards,
Amit Kapila.

  154. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-09-23T02:47:16Z

    On Mon, Sep 23, 2024 at 4:10 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Sep 20, 2024 at 2:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Sep 20, 2024 at 4:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Users can use a publication like "create publication pub1 for table
> > > > > t1(c1, c2), t2;" where they want t1's generated column to be published
> > > > > but not for t2. They can specify the generated column name in the
> > > > > column list of t1 in that case even though the rest of the tables
> > > > > won't publish generated columns.
> > > >
> > > > Agreed.
> > > >
> > > > I think that users can use the publish_generated_column option when
> > > > they want to publish all generated columns, instead of specifying all
> > > > the columns in the column list. It's another advantage of this option
> > > > that it will also include the future generated columns.
> > > >
> > >
> > > OK. Let me give some examples below to help understand this idea.
> > >
> > > Please correct me if these are incorrect.
> > >
> > > Examples, when publish_generated_columns=true:
> > >
> > > CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> > > (publish_generated_columns=true)
> > > t1 -> publishes a, b, gen2 (e.g. what column list says)
> > > t2 -> publishes c, d + ALSO gen1, gen2
> > >
> > > CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
> > > t1 -> publishes a, b + ALSO gen1, gen2
> > > t2 -> publishes gen1 (e.g. what column list says)
> > >
> >
> > These two could be controversial because one could expect that if
> > "publish_generated_columns=true" then publish generated columns
> > irrespective of whether they are mentioned in column_list. I am of the
> > opinion that column_list should take priority the results should be as
> > mentioned by you but let us see if anyone thinks otherwise.
> >
> > >
> > > ======
> > >
> > > The idea LGTM, although now the parameter name
> > > ('publish_generated_columns') seems a bit misleading since sometimes
> > > generated columns get published "irrespective of the option".
> > >
> > > So, I think the original parameter name 'include_generated_columns'
> > > might be better here because IMO "include" seems more like "add them
> > > if they are not already specified", which is exactly what this idea is
> > > doing.
> > >
> >
> > I still prefer 'publish_generated_columns' because it matches with
> > other publication option names. One can also deduce from
> > 'include_generated_columns' that add all the generated columns even
> > when some of them are specified in column_list.
> >
>
> Fair point. Anyway, to avoid surprises it will be important for the
> precedence rules to be documented clearly (probably with some
> examples),
>

Yeah, one or two examples would be good, but we can have a separate
doc patch that has clearly mentioned all the rules.

-- 
With Regards,
Amit Kapila.

  155. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-09-23T11:57:04Z

    On Thu, 12 Sept 2024 at 11:01, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Because this feature is now being implemented as a PUBLICATION option,
> there is another scenario that might need consideration; I am thinking
> about where the same table is published by multiple PUBLICATIONS (with
> different option settings) that are subscribed by a single
> SUBSCRIPTION.
>
> e.g.1
> -----
> CREATE PUBLICATION pub1 FOR TABLE t1 WITH (publish_generated_columns = true);
> CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> -----
>
> e.g.2
> -----
> CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns = true);
> CREATE PUBLICATION pub2 FOR TABLE t1 WITH (publish_generated_columns = false);
> CREATE SUBSCRIPTION sub ... PUBLICATIONS pub1,pub2;
> -----
>
> Do you know if this case is supported? If yes, then which publication
> option value wins?

I have verified the various scenarios discussed here and the patch
works as expected:
Test presetup:
-- publisher
CREATE TABLE t1 (a int PRIMARY KEY, b int, c int, gen1 int GENERATED
ALWAYS AS (a * 2) STORED, gen2 int GENERATED ALWAYS AS (a * 2)
STORED);
-- Subscriber
CREATE TABLE t1 (a int PRIMARY KEY, b int, c int, d int, e int);

Test1: Subscriber will have only non-generated columns a,b,c
replicated from publisher:
create publication pub1 for all tables with (
publish_generated_columns = false);
INSERT INTO t1 (a,b,c) VALUES (1,1,1);

--Subscriber will have only non-generated columns a,b,c replicated
from publisher:
subscriber=# select * from t1;
 a | b | c | d | e
---+---+---+---+---
 1 | 1 | 1 |   |
(1 row)

Test2: Subscriber will include generated columns a,b,c replicated from
publisher:
create publication pub1 for all tables with ( publish_generated_columns = true);
INSERT INTO t1 (a,b,c) VALUES (1,1,1);

-- Subscriber will include generated columns a,b,c replicated from publisher:
subscriber=# select * from t1;
 a | b | c | d | e
---+---+---+---+---
 1 | 1 | 1 | 2 | 2
(1 row)

Test3: Cannot have subscription subscribing to publication with
publish_generated_columns as true and false
-- publisher
create publication pub1 for all tables with (publish_generated_columns = false);
create publication pub2 for all tables with (publish_generated_columns = true);

-- subscriber
subscriber=# create subscription sub1 connection 'dbname=postgres
host=localhost port=5432' publication pub1,pub2;
ERROR:  cannot use different column lists for table "public.t1" in
different publications

Test4a: Warning thrown when a generated column is specified in column
list along with publish_generated_columns as false
-- publisher
postgres=# create publication pub1 for table t1(a,b,gen1) with (
publish_generated_columns = false);
WARNING:  specified generated column "gen1" in publication column list
for publication with publish_generated_columns as false
CREATE PUBLICATION

Regards,
Vignesh

  156. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-09-23T12:10:58Z

    On Fri, 20 Sept 2024 at 04:16, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> ...
> > > I think that the column list should take priority and we should
> > > publish the generated column if it is mentioned in  irrespective of
> > > the option.
> >
> > Agreed.
> >
> > >
> ...
> > >
> > > Users can use a publication like "create publication pub1 for table
> > > t1(c1, c2), t2;" where they want t1's generated column to be published
> > > but not for t2. They can specify the generated column name in the
> > > column list of t1 in that case even though the rest of the tables
> > > won't publish generated columns.
> >
> > Agreed.
> >
> > I think that users can use the publish_generated_column option when
> > they want to publish all generated columns, instead of specifying all
> > the columns in the column list. It's another advantage of this option
> > that it will also include the future generated columns.
> >
>
> OK. Let me give some examples below to help understand this idea.
>
> Please correct me if these are incorrect.
>
> ======
>
> Assuming these tables:
>
> t1(a,b,gen1,gen2)
> t2(c,d,gen1,gen2)
>
> Examples, when publish_generated_columns=false:
>
> CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> (publish_generated_columns=false)
> t1 -> publishes a, b, gen2 (e.g. what column list says)
> t2 -> publishes c, d
>
> CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=false)
> t1 -> publishes a, b
> t2 -> publishes gen1 (e.g. what column list says)
>
> CREATE PUBLICATION pub1 FOR t1, t2 WITH (publish_generated_columns=false)
> t1 -> publishes a, b
> t2 -> publishes c, d
>
> CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns=false)
> t1 -> publishes a, b
> t2 -> publishes c, d
>
> ~~
>
> Examples, when publish_generated_columns=true:
>
> CREATE PUBLICATION pub1 FOR t1(a,b,gen2), t2 WITH
> (publish_generated_columns=true)
> t1 -> publishes a, b, gen2 (e.g. what column list says)
> t2 -> publishes c, d + ALSO gen1, gen2
>
> CREATE PUBLICATION pub1 FOR t1, t2(gen1) WITH (publish_generated_columns=true)
> t1 -> publishes a, b + ALSO gen1, gen2
> t2 -> publishes gen1 (e.g. what column list says)
>
> CREATE PUBLICATION pub1 FOR t1, t2 WITH (publish_generated_columns=true)
> t1 -> publishes a, b + ALSO gen1, gen2
> t2 -> publishes c, d + ALSO gen1, gen2
>
> CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns=true)
> t1 -> publishes a, b + ALSO gen1, gen2
> t2 -> publishes c, d + ALSO gen1, gen2
>
> ======
>
> The idea LGTM, although now the parameter name
> ('publish_generated_columns') seems a bit misleading since sometimes
> generated columns get published "irrespective of the option".
>
> So, I think the original parameter name 'include_generated_columns'
> might be better here because IMO "include" seems more like "add them
> if they are not already specified", which is exactly what this idea is
> doing.
>
> Thoughts?

I have verified the various scenarios discussed here and the patch
works as expected with v32 version patch shared at [1]:

Test presetup:
-- publisher
CREATE TABLE t1 (a int PRIMARY KEY, b int, gen1 int GENERATED ALWAYS
AS (a * 2) STORED, gen2 int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE t2 (c int PRIMARY KEY, d int, gen1 int GENERATED ALWAYS
AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (d * 2) STORED);

-- subscriber
CREATE TABLE t1 (a int PRIMARY KEY, b int, gen1 int, gen2 int);
CREATE TABLE t2 (c int PRIMARY KEY, d int, gen1 int, gen1 int);

Test1: Publisher replicates the column list data including generated
columns even though publish_generated_columns option is false:
Publisher:
CREATE PUBLICATION pub1 FOR table t1, t2(gen1) WITH
(publish_generated_columns=false)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
--t1 -> publishes a, b
subscriber=# select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |      |
(1 row)

--t2 -> publishes gen1 (e.g. what column list says)
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
   |   |    2 |
(1 row)

Test2: Publisher does not replication gen column if
publish_generated_columns option is false
Publisher:
CREATE PUBLICATION pub1 FOR table t1, t2 WITH (publish_generated_columns=false)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
--t1 -> publishes a, b
subscriber=# select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |      |
(1 row)

-- t2 -> publishes c, d
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
 1 | 1 |      |
(1 row)

Test3: Publisher does not replication gen column if
publish_generated_columns option is false
Publisher:
CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns=false)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
--t1 -> publishes a, b
subscriber=# select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |      |
(1 row)

-- t2 -> publishes c, d
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
 1 | 1 |      |
(1 row)

Test4: Publisher publishes only the data of the columns specified in
column list skipping other generated/non-generated columns:
Publisher:
CREATE PUBLICATION pub1 FOR table t1(a,b,gen2), t2 WITH
(publish_generated_columns=true)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
-- t1 -> publishes a, b, gen2 (e.g. what column list says)
subscriber=#  select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |      |    2
(1 row)

-- t2 -> publishes c, d + ALSO gen1, gen2
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
 1 | 1 |    2 |    2
(1 row)


Test5: Publisher publishes only the data of the columns specified in
column list skipping other generated/non-generated columns:
Publisher:
CREATE PUBLICATION pub1 FOR table t1, t2(gen1) WITH
(publish_generated_columns=true)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
-- t1 -> publishes a, b + ALSO gen1, gen2
subscriber=#  select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |    2 |    2
(1 row)

-- t2 -> publishes gen1 (e.g. what column list says)
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
   |   |    2 |
(1 row)

Test6: Publisher replicates all columns if publish_generated_columns
is enabled without column list
Publisher:
CREATE PUBLICATION pub1 FOR  table t1, t2 WITH (publish_generated_columns=true)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
-- t1 -> publishes a, b + ALSO gen1, gen2
subscriber=#  select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |    2 |    2
(1 row)

-- t2 -> publishes c, d + ALSO gen1, gen2
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
 1 | 1 |    2 |    2
(1 row)

Test7: Publisher replicates all columns if publish_generated_columns
is enabled without column list
Publisher:
CREATE PUBLICATION pub1 FOR ALL TABLES WITH (publish_generated_columns=true)
insert into t1 values(1,1);
insert into t2 values(1,1);

Subscriber:
-- t1 -> publishes a, b + ALSO gen1, gen2
subscriber=#  select * from t1;
 a | b | gen1 | gen2
---+---+------+------
 1 | 1 |    2 |    2
(1 row)

-- t2 -> publishes c, d + ALSO gen1, gen2
subscriber=# select * from t2;
 c | d | gen1 | gen2
---+---+------+------
 1 | 1 |    2 |    2
(1 row)

[1] - https://www.postgresql.org/message-id/CAHv8RjKkoaS1oMsFvPRFB9nPSVC5p_D4Kgq5XB9Y2B2xU7smbA%40mail.gmail.com

Regards,
Vignesh

  157. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-09-23T12:16:45Z

    On Fri, 20 Sept 2024 at 17:15, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Wed, Sep 11, 2024 at 8:55 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are a some more review comments for patch v30-0001.
> >
> > ======
> > src/sgml/ref/create_publication.sgml
> >
> > 1.
> > +         <para>
> > +          If the publisher-side column is also a generated column
> > then this option
> > +          has no effect; the publisher column will be filled as normal with the
> > +          publisher-side computed or default data.
> > +         </para>
> >
> > It should say "subscriber-side"; not "publisher-side". The same was
> > already reported by Sawada-San [1].
> >
> > ~~~
> >
> > 2.
> > +         <para>
> > +         This parameter can only be set <literal>true</literal> if
> > <literal>copy_data</literal> is
> > +         set to <literal>false</literal>.
> > +         </para>
> >
> > IMO this limitation should be addressed by patch 0001 like it was
> > already done in the previous patches (e.g. v22-0002). I think
> > Sawada-san suggested the same [1].
> >
> > Anyway, 'copy_data' is not a PUBLICATION option, so the fact it is
> > mentioned like this without any reference to the SUBSCRIPTION seems
> > like a cut/paste error from the previous implementation.
> >
> > ======
> > src/backend/catalog/pg_publication.c
> >
> > 3. pub_collist_validate
> > - if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > - ereport(ERROR,
> > - errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > - errmsg("cannot use generated column \"%s\" in publication column list",
> > -    colname));
> > -
> >
> > Instead of just removing this ERROR entirely here, I thought it would
> > be more user-friendly to give a WARNING if the PUBLICATION's explicit
> > column list includes generated cols when the option
> > "publish_generated_columns" is false. This combination doesn't seem
> > like something a user would do intentionally, so just silently
> > ignoring it (like the current patch does) is likely going to give
> > someone unexpected results/grief.
> >
> > ======
> > src/backend/replication/logical/proto.c
> >
> > 4. logicalrep_write_tuple, and logicalrep_write_attrs:
> >
> > - if (att->attisdropped || att->attgenerated)
> > + if (att->attisdropped)
> >   continue;
> >
> > Why aren't you also checking the new PUBLICATION option here and
> > skipping all gencols if the "publish_generated_columns" option is
> > false? Or is the BMS of pgoutput_column_list_init handling this case?
> > Maybe there should be an Assert for this?
> >
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 5. send_relation_and_attrs
> >
> > - if (att->attisdropped || att->attgenerated)
> > + if (att->attisdropped)
> >   continue;
> >
> > Same question as #4.
> >
> > ~~~
> >
> > 6. prepare_all_columns_bms and pgoutput_column_list_init
> >
> > + if (att->attgenerated && !pub->pubgencolumns)
> > + cols = bms_del_member(cols, i + 1);
> >
> > IIUC, the algorithm seems overly tricky filling the BMS with all
> > columns, before straight away conditionally removing the generated
> > columns. Can't it be refactored to assign all the correct columns
> > up-front, to avoid calling bms_del_member()?
> >
> > ======
> > src/bin/pg_dump/pg_dump.c
> >
> > 7. getPublications
> >
> > IIUC, there is lots of missing SQL code here (for all older versions)
> > that should be saying "false AS pubgencolumns".
> > e.g. compare the SQL with how "false AS pubviaroot" is used.
> >
> > ======
> > src/bin/pg_dump/t/002_pg_dump.pl
> >
> > 8. Missing tests?
> >
> > I expected to see a pg_dump test for this new PUBLICATION option.
> >
> > ======
> > src/test/regress/sql/publication.sql
> >
> > 9. Missing tests?
> >
> > How about adding another test case that checks this new option must be
> > "Boolean"?
> >
> > ~~~
> >
> > 10. Missing tests?
> >
> > --- error: generated column "d" can't be in list
> > +-- ok: generated columns can be in the list too
> >  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
> > +ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
> >
> > (see my earlier comment #3)
> >
> > IMO there should be another test case for a WARNING here if the user
> > attempts to include generated column 'd' in an explicit PUBLICATION
> > column list while the "publish_generated-columns" is false.
> >
> > ======
> > [1]  https://www.postgresql.org/message-id/CAD21AoA-tdTz0G-vri8KM2TXeFU8RCDsOpBXUBCgwkfokF7%3DjA%40mail.gmail.com
> >
>
> I have fixed all the comments. The attached patches contain the desired changes.
> Also the merging of 0001 and 0002 can be done once there are no
> comments on the patch to help in reviewing.

The warning message appears to be incorrect. Even though
publish_generated_columns is set to true, the warning indicates that
it is false.
CREATE TABLE t1 (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
postgres=# CREATE PUBLICATION pub1 FOR table t1(gen1) WITH
(publish_generated_columns=true);
WARNING:  specified generated column "gen1" in publication column list
for publication with publish_generated_columns as false

Regards,
Vignesh

  158. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-09-23T12:39:45Z

    On Fri, 20 Sept 2024 at 17:15, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Wed, Sep 11, 2024 at 8:55 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> I have fixed all the comments. The attached patches contain the desired changes.
> Also the merging of 0001 and 0002 can be done once there are no
> comments on the patch to help in reviewing.

Few comments:
1) This commit  message seems wrong, currently irrespective of
publish_generated_columns, the column specified in column list take
preceedene:
When 'publish_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.

2) Since we have added pubgencols to pg_pubication.h we can specify
"Bump catversion" in the commit message.

3) In create publication column list/publish_generated_columns
documentation we should mention that if generated column is mentioned
in column list, generated columns mentioned in column list will be
replication irrespective of publish_generated_columns option.

4) This warning should be mentioned only if publish_generated_columns is false:
                if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
-                       ereport(ERROR,
+                       ereport(WARNING,

errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
-                                       errmsg("cannot use generated
column \"%s\" in publication column list",
+                                       errmsg("specified generated
column \"%s\" in publication column list for publication with
publish_generated_columns as false",
                                                   colname));

5) These tests are not required for this feature:
+       'ALTER PUBLICATION pub5 ADD TABLE test_table WHERE (col1 > 0);' => {
+               create_order => 51,
+               create_sql =>
+                 'ALTER PUBLICATION pub5 ADD TABLE
dump_test.test_table WHERE (col1 > 0);',
+               regexp => qr/^
+                       \QALTER PUBLICATION pub5 ADD TABLE ONLY
dump_test.test_table WHERE ((col1 > 0));\E
+                       /xm,
+               like => { %full_runs, section_post_data => 1, },
+               unlike => {
+                       exclude_dump_test_schema => 1,
+                       exclude_test_table => 1,
+               },
+       },
+
+       'ALTER PUBLICATION pub5 ADD TABLE test_second_table WHERE
(col2 = \'test\');'
+         => {
+               create_order => 52,
+               create_sql =>
+                 'ALTER PUBLICATION pub5 ADD TABLE
dump_test.test_second_table WHERE (col2 = \'test\');',
+               regexp => qr/^
+                       \QALTER PUBLICATION pub5 ADD TABLE ONLY
dump_test.test_second_table WHERE ((col2 = 'test'::text));\E
+                       /xm,
+               like => { %full_runs, section_post_data => 1, },
+               unlike => { exclude_dump_test_schema => 1, },
+         },

Regards,
Vignesh

  159. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-23T23:46:30Z

    Hi. Here are my review comments for v32-0001

You wrote: "I have addressed all the comments in the v32-0001 Patch.",
however, I found multiple old review comments not addressed. Please
give a reason if a comment is deliberately left out, otherwise, I will
assume they are omitted by accident and so keep repeating them.

There were also still some unanswered questions from previous reviews,
so I have reminded you about those again here.

======
Commit message

1.
This commit enables support for the 'publish_generated_columns' option
in logical replication, allowing the transmission of generated column
information and data alongside regular table changes. The option
'publish_generated_columns' is a PUBLICATION parameter.

~

That PUBLICATION info in the 2nd sentence would be easier to say in
the 1st sentence.
SUGGESTION:
This commit supports the transmission of generated column information
and data alongside regular table changes. This behaviour is controlled
by a new PUBLICATION parameter ('publish_generated_columns').

~~~

2.
When 'publish_generated_columns' is false, generated columns are not
replicated, even when present in a PUBLICATION col-list.

Hm. This contradicts the behaviour that Amit wanted, (e.g.
"column-list takes precedence"). So I am not sure if this patch is
already catering for the behaviour suggested by Amit or if that is yet
to come in v33. For now, I am assuming that 32* has not caught up with
the latest behaviour requirements, but that might be a wrong
assumption; perhaps it is only this commit message that is bogus.

~~~

3. General.

On the same subject, there is lots of code, like:

if (att->attgenerated && !pub->pubgencols)
continue;

I suspect that might not be quite what you want for the "column-list
takes precedence" behaviour, but I am not going to identify all those
during this review. It needs lots of combinations of column list tests
to verify it.

======
doc/src/sgml/ddl.sgml

4ab.
nit - Huh?? Not changed the linkend as told in a previous review [1-#3a]
nit - Huh?? Not changed to call this a "parameter" instead of an
"option" as told in a previous review [1-#3b]

======
doc/src/sgml/protocol.sgml

5.
-     <para>
-      Next, the following message part appears for each column included in
-      the publication (except generated columns):
-     </para>
-

nit -- Huh?? I don't think you can just remove this whole paragraph.
But, probably you can just remove the "except generated columns" part.
I posted this same comment [4 #11] 20 patch versions back.

======
doc/src/sgml/ref/create_publication.sgml

6abc.
nit - Huh?? Not changed the parameter ID as told in a previous review [1-#6]
nit - Huh?? Not removed paragraph "This option is only available..."
as told in a previous review. See [1-#7]
nit - Huh?? Not removed paragraph "This parameter can only be set" as
told in a previous review. See [1-#8]

======
src/backend/catalog/pg_publication.c

7.
  if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
+ ereport(WARNING,
  errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
+ errmsg("specified generated column \"%s\" in publication column list
for publication with publish_generated_columns as false",
     colname));

I did not understand how this WARNING can know
"publish_generated_columns as false"? Should the code be checking the
function parameter 'pubgencols'?

The errmsg also seemed a bit verbose. How about:
"specified generated column \"%s\" in publication column list when
publish_generated_columns = false"

======
src/backend/replication/logical/proto.c

8.
logicalrep_write_tuple:
logicalrep_write_attrs:

Reminder. I think I have multiple questions about this code from
previous reviews that may be still unanswered. See [2 #4]. Maybe when
you implement Amit's "column list takes precedence" behaviour then
this code is fine as-is (because the replication message might include
gencols or not-gecols regardless of the 'publish_generated_columns'
value). But I don't think that is the current implementation, so
something did not quite seem right. I am not sure. If you say it is
fine then I will believe it, but the question [2 #4] remains
unanswered.

======
src/backend/replication/pgoutput/pgoutput.c

9.
send_relation_and_attrs:

Reminder: Here is another question that was answered from [2 #5]. I
did not really trust it for the current implementation, but for the
"column list takes precedence" behaviour probably it will be ok.

~~~

10.
+/*
+ * Prepare new column list bitmap. This includes all the columns of the table.
+ */
+static Bitmapset *
+prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
+ TupleDesc desc)
+{

This function needs a better comment with more explanation about what
this is REALLY doing. e.g. it says "includes all columns of the
table", but tthe implementation is skipping generated cols, so clearly
it is not "all columns of the table".

~~~

11. pgoutput_column_list_init

TBH, I struggle to read the logic of this function. Rewriting some
parts, inverting some variables, and adding more commentary might help
a lot.

11a.
There are too many "negatives" (with ! operator and with the word "no"
in the variable).

e.g. code is written in a backward way like:
if (!pub_no_list)
cols = pub_collist_to_bitmapset(cols, cfdatum, entry->entry_cxt);
else
cols = prepare_all_columns_bms(data, entry, desc);

instead of what could have been said:
if (pub_rel_has_collist)
cols = pub_collist_to_bitmapset(cols, cfdatum, entry->entry_cxt);
else
cols = prepare_all_columns_bms(data, entry, desc);

~

11b.
- * If the publication is FOR ALL TABLES then it is treated the same as
- * if there are no column lists (even if other publications have a
- * list).
+ * If the publication is FOR ALL TABLES and include generated columns
+ * then it is treated the same as if there are no column lists (even
+ * if other publications have a list).
  */
- if (!pub->alltables)
+ if (!pub->alltables || !pub->pubgencols)

The code does not appear to match the comment ("If the publication is
FOR ALL TABLES and include generated columns"). If it did it should
look like "if (pub->alltables && pub->pubgencols)".

Also, should "and include generated column" be properly referring to
the new PUBLICATION parameter name?

Also, the comment is somewhat confusing. I saw in the thread Vignesh
wrote an explanation like "To handle cases where the
publish_generated_columns option isn't specified for all tables in a
publication, the pubgencolumns check needs to be performed. In such
cases, we must create a column list that excludes generated columns"
[3]. IMO that was clearer information so something similar should be
written in this code comment.
~

11c.
+ /* Build the column list bitmap in the per-entry context. */
+ if (!pub_no_list || !pub->pubgencols) /* when not null */

I don't know what "when not null" means here. Aren't those both
booleans? How can it be "null"?

======
src/bin/pg_dump/pg_dump.c

12. getPublications:

Huh?? The code has not changed to address an old review comment I had
posted to say there seem multiple code fragments missing that should
say "false AS pubgencols". Refer to [2 #7].

======
src/bin/pg_dump/t/002_pg_dump.pl

13.
'ALTER PUBLICATION pub5 ADD TABLE test_table WHERE (col1 > 0);' => {
+ create_order => 51,
+ create_sql =>
+   'ALTER PUBLICATION pub5 ADD TABLE dump_test.test_table WHERE (col1 > 0);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub5 ADD TABLE ONLY dump_test.test_table WHERE
((col1 > 0));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => {
+ exclude_dump_test_schema => 1,
+ exclude_test_table => 1,
+ },
+ },
+
+ 'ALTER PUBLICATION pub5 ADD TABLE test_second_table WHERE (col2 = \'test\');'
+   => {
+ create_order => 52,
+ create_sql =>
+   'ALTER PUBLICATION pub5 ADD TABLE dump_test.test_second_table
WHERE (col2 = \'test\');',
+ regexp => qr/^
+ \QALTER PUBLICATION pub5 ADD TABLE ONLY dump_test.test_second_table
WHERE ((col2 = 'test'::text));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+   },
+

It wasn't clear to me how these tests are related to the patch.
Shouldn't there instead be some ALTER tests for trying to modify the
'publish_generate_columns' parameter?

======
src/test/regress/expected/publication.out
src/test/regress/sql/publication.sql

14.
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR:  cannot use generated column "d" in publication column list
+WARNING:  specified generated column "d" in publication column list
for publication with publish_generated_columns as false

I think these tests for the WARNING scenario need to be a bit more
deliberate. This seems to have happened as a side-effect. For example,
I was expecting more testing like:

Comments about various combinations to say what you are doing and what
you are expecting:
- gencols in column list with publish_generated_columns=false, expecting WARNING
- gencols in column list with publish_generated_columns=true, NOT
expecting WARNING
- gencols in column list with publish_generated_columns=true, then
ALTER PUBLICATION setting publication_generate_columns=false,
expecting WARNING
- NO gencols in column list with publish_generated_columns=false, then
ALTER PUBLICATION to add gencols to column list, expecting WARNING

======
src/test/subscription/t/031_column_list.pl

15.
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
 # So, the publication having a column list except for those columns and a
-# publication without any column (aka all columns as part of the columns
+# publication without any column list (aka all columns as part of the column
 # list) are considered to have the same column list.
 $node_publisher->safe_psql(
  'postgres', qq(
  CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int
GENERATED ALWAYS AS (a + 1) STORED);
  ALTER TABLE test_mix_4 DROP COLUMN c;

- CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
- CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
+ CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 WITH
(publish_generated_columns = true);
+ CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4 WITH
(publish_generated_columns = false);

I felt the comment for this test ought to be saying something more
about what you are doing with the 'publish_generated_columns'
parameters and what behaviour it was expecting.

======
Please refer to the attachment which addresses some of the nit
comments mentioned above.

======
[1] my review of v31-0001:
https://www.postgresql.org/message-id/CAHut%2BPsv-neEP_ftvBUBahh%2BKCWw%2BqQMF9N3sGU3YHWPEzFH-Q%40mail.gmail.com
[2] my review of v30-0001:
https://www.postgresql.org/message-id/CAHut%2BPuaitgE4tu3nfaR%3DPCQEKjB%3DmpDtZ1aWkbwb%3DJZE8YvqQ%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CALDaNm1c7xPBodHw6LKp9e8hvGVJHcKH%3DDHK0iXmZuXKPnxZ3Q%40mail.gmail.com
[4] https://www.postgresql.org/message-id/CAHut%2BPv45gB4cV%2BSSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  160. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-24T01:37:49Z

    Hi. Here are my v32-0002 review comments:

======
src/backend/replication/logical/tablesync.c

1. fetch_remote_table_info

  /*
- * Get column lists for each relation.
+ * Get column lists for each relation, and check if any of the
+ * publications have the 'publish_generated_columns' parameter enabled.

I am not 100% sure about this logic anymore. Maybe it is OK, but it
requires careful testing because with Amit's "column lists take
precedence" it is now possible for the publication to say
'publish_generated_columns=false', but the publication can still
publish gencols *anyway* if they were specified in a column list.

~~~

2.
  /*
  * Fetch info about column lists for the relation (from all the
  * publications).
  */
+ StringInfo pub_names = makeStringInfo();
+
+ get_publications_str(MySubscription->publications, pub_names, true);
  resetStringInfo(&cmd);
  appendStringInfo(&cmd,
~

nit - The comment here seems misplaced.

~~~

3.
+ if (server_version >= 120000)
+ {
+ has_pub_with_pubgencols = server_version >= 180000 && has_pub_with_pubgencols;
+
+ if (!has_pub_with_pubgencols)
+ appendStringInfo(&cmd, " AND a.attgenerated = ''");
+ }

My previous review comment about this [1 #10] was:
Can the 'gencols_allowed' var be removed, and the condition just be
replaced with if (!has_pub_with_pubgencols)? It seems equivalent
unless I am mistaken.

nit - So the current v32 code is not what I was expecting. What I
meant was 'has_pub_with_pubgencols' can only be true if server_version
>= 180000, so I thought there was no reason to check it again. For
reference, I've changed it to like I meant in the nitpicks attachment.
Please see if that works the same.

======
[1] my review of v31-0002.
https://www.postgresql.org/message-id/CAHut%2BPusbhvPrL1uN1TKY%3DFd4zu3h63eDebZvsF%3Duy%2BLBKTwgA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  161. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-26T06:14:55Z

    Hi, I have written a new patch to document this feature.

The patch adds a new section to the "Logical Replication" chapter. It
applies atop the existing patches.

v33-0001 (same as v32-0001)
v33-0002 (same as v32-0002)
v33-0003 (new DOCS)

Review comments are welcome.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  162. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-27T00:03:14Z

    On Wed, Sep 25, 2024 at 11:15 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, I have written a new patch to document this feature.
>
> The patch adds a new section to the "Logical Replication" chapter. It
> applies atop the existing patches.
>
> v33-0001 (same as v32-0001)
> v33-0002 (same as v32-0002)
> v33-0003 (new DOCS)
>
> Review comments are welcome.

Thank you for updating the patch!

I think that the patch doesn't have regression tests to check if
generated column data is replicated to the subscriber as expected. I
think we should include some tests for this feature (especially with
other features such as column list).

Also, when testing this feature, I got the following warning message
even if the publication has publish_generated_columns = true:

=# create publication pub for table test (a, c) with
(publish_generated_columns = true);
WARNING:  specified generated column "c" in publication column list
for publication with publish_generated_columns as false

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  163. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-27T14:31:28Z

    On Mon, Sep 23, 2024 at 5:56 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 20 Sept 2024 at 17:15, Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, Sep 11, 2024 at 8:55 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Here are a some more review comments for patch v30-0001.
> > >
> > > ======
> > > src/sgml/ref/create_publication.sgml
> > >
> > > 1.
> > > +         <para>
> > > +          If the publisher-side column is also a generated column
> > > then this option
> > > +          has no effect; the publisher column will be filled as normal with the
> > > +          publisher-side computed or default data.
> > > +         </para>
> > >
> > > It should say "subscriber-side"; not "publisher-side". The same was
> > > already reported by Sawada-San [1].
> > >
> > > ~~~
> > >
> > > 2.
> > > +         <para>
> > > +         This parameter can only be set <literal>true</literal> if
> > > <literal>copy_data</literal> is
> > > +         set to <literal>false</literal>.
> > > +         </para>
> > >
> > > IMO this limitation should be addressed by patch 0001 like it was
> > > already done in the previous patches (e.g. v22-0002). I think
> > > Sawada-san suggested the same [1].
> > >
> > > Anyway, 'copy_data' is not a PUBLICATION option, so the fact it is
> > > mentioned like this without any reference to the SUBSCRIPTION seems
> > > like a cut/paste error from the previous implementation.
> > >
> > > ======
> > > src/backend/catalog/pg_publication.c
> > >
> > > 3. pub_collist_validate
> > > - if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > > - ereport(ERROR,
> > > - errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > > - errmsg("cannot use generated column \"%s\" in publication column list",
> > > -    colname));
> > > -
> > >
> > > Instead of just removing this ERROR entirely here, I thought it would
> > > be more user-friendly to give a WARNING if the PUBLICATION's explicit
> > > column list includes generated cols when the option
> > > "publish_generated_columns" is false. This combination doesn't seem
> > > like something a user would do intentionally, so just silently
> > > ignoring it (like the current patch does) is likely going to give
> > > someone unexpected results/grief.
> > >
> > > ======
> > > src/backend/replication/logical/proto.c
> > >
> > > 4. logicalrep_write_tuple, and logicalrep_write_attrs:
> > >
> > > - if (att->attisdropped || att->attgenerated)
> > > + if (att->attisdropped)
> > >   continue;
> > >
> > > Why aren't you also checking the new PUBLICATION option here and
> > > skipping all gencols if the "publish_generated_columns" option is
> > > false? Or is the BMS of pgoutput_column_list_init handling this case?
> > > Maybe there should be an Assert for this?
> > >
> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 5. send_relation_and_attrs
> > >
> > > - if (att->attisdropped || att->attgenerated)
> > > + if (att->attisdropped)
> > >   continue;
> > >
> > > Same question as #4.
> > >
> > > ~~~
> > >
> > > 6. prepare_all_columns_bms and pgoutput_column_list_init
> > >
> > > + if (att->attgenerated && !pub->pubgencolumns)
> > > + cols = bms_del_member(cols, i + 1);
> > >
> > > IIUC, the algorithm seems overly tricky filling the BMS with all
> > > columns, before straight away conditionally removing the generated
> > > columns. Can't it be refactored to assign all the correct columns
> > > up-front, to avoid calling bms_del_member()?
> > >
> > > ======
> > > src/bin/pg_dump/pg_dump.c
> > >
> > > 7. getPublications
> > >
> > > IIUC, there is lots of missing SQL code here (for all older versions)
> > > that should be saying "false AS pubgencolumns".
> > > e.g. compare the SQL with how "false AS pubviaroot" is used.
> > >
> > > ======
> > > src/bin/pg_dump/t/002_pg_dump.pl
> > >
> > > 8. Missing tests?
> > >
> > > I expected to see a pg_dump test for this new PUBLICATION option.
> > >
> > > ======
> > > src/test/regress/sql/publication.sql
> > >
> > > 9. Missing tests?
> > >
> > > How about adding another test case that checks this new option must be
> > > "Boolean"?
> > >
> > > ~~~
> > >
> > > 10. Missing tests?
> > >
> > > --- error: generated column "d" can't be in list
> > > +-- ok: generated columns can be in the list too
> > >  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
> > > +ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
> > >
> > > (see my earlier comment #3)
> > >
> > > IMO there should be another test case for a WARNING here if the user
> > > attempts to include generated column 'd' in an explicit PUBLICATION
> > > column list while the "publish_generated-columns" is false.
> > >
> > > ======
> > > [1]  https://www.postgresql.org/message-id/CAD21AoA-tdTz0G-vri8KM2TXeFU8RCDsOpBXUBCgwkfokF7%3DjA%40mail.gmail.com
> > >
> >
> > I have fixed all the comments. The attached patches contain the desired changes.
> > Also the merging of 0001 and 0002 can be done once there are no
> > comments on the patch to help in reviewing.
>
> The warning message appears to be incorrect. Even though
> publish_generated_columns is set to true, the warning indicates that
> it is false.
> CREATE TABLE t1 (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> postgres=# CREATE PUBLICATION pub1 FOR table t1(gen1) WITH
> (publish_generated_columns=true);
> WARNING:  specified generated column "gen1" in publication column list
> for publication with publish_generated_columns as false
>

I have fixed the warning message. The attached patches contain the
desired changes.

Thanks and Regards,
Shubham Khanna.

  164. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-27T14:39:58Z

    On Mon, Sep 23, 2024 at 6:19 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 20 Sept 2024 at 17:15, Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, Sep 11, 2024 at 8:55 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > I have fixed all the comments. The attached patches contain the desired changes.
> > Also the merging of 0001 and 0002 can be done once there are no
> > comments on the patch to help in reviewing.
>
> Few comments:
> 1) This commit  message seems wrong, currently irrespective of
> publish_generated_columns, the column specified in column list take
> preceedene:
> When 'publish_generated_columns' is false, generated columns are not
> replicated, even when present in a PUBLICATION col-list.
>
> 2) Since we have added pubgencols to pg_pubication.h we can specify
> "Bump catversion" in the commit message.
>
> 3) In create publication column list/publish_generated_columns
> documentation we should mention that if generated column is mentioned
> in column list, generated columns mentioned in column list will be
> replication irrespective of publish_generated_columns option.
>
> 4) This warning should be mentioned only if publish_generated_columns is false:
>                 if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> -                       ereport(ERROR,
> +                       ereport(WARNING,
>
> errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> -                                       errmsg("cannot use generated
> column \"%s\" in publication column list",
> +                                       errmsg("specified generated
> column \"%s\" in publication column list for publication with
> publish_generated_columns as false",
>                                                    colname));
>
> 5) These tests are not required for this feature:
> +       'ALTER PUBLICATION pub5 ADD TABLE test_table WHERE (col1 > 0);' => {
> +               create_order => 51,
> +               create_sql =>
> +                 'ALTER PUBLICATION pub5 ADD TABLE
> dump_test.test_table WHERE (col1 > 0);',
> +               regexp => qr/^
> +                       \QALTER PUBLICATION pub5 ADD TABLE ONLY
> dump_test.test_table WHERE ((col1 > 0));\E
> +                       /xm,
> +               like => { %full_runs, section_post_data => 1, },
> +               unlike => {
> +                       exclude_dump_test_schema => 1,
> +                       exclude_test_table => 1,
> +               },
> +       },
> +
> +       'ALTER PUBLICATION pub5 ADD TABLE test_second_table WHERE
> (col2 = \'test\');'
> +         => {
> +               create_order => 52,
> +               create_sql =>
> +                 'ALTER PUBLICATION pub5 ADD TABLE
> dump_test.test_second_table WHERE (col2 = \'test\');',
> +               regexp => qr/^
> +                       \QALTER PUBLICATION pub5 ADD TABLE ONLY
> dump_test.test_second_table WHERE ((col2 = 'test'::text));\E
> +                       /xm,
> +               like => { %full_runs, section_post_data => 1, },
> +               unlike => { exclude_dump_test_schema => 1, },
> +         },
>

I have addressed all the comments in the v34-0001 Patch. Please refer
to the updated v34-0001 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjJkUdYCdK_bL3yvEV%3DzKrA2dsnZYa1VMT2H5v0%2BqbaGbA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  165. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-27T14:42:42Z

    On Tue, Sep 24, 2024 at 5:16 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi. Here are my review comments for v32-0001
>
> You wrote: "I have addressed all the comments in the v32-0001 Patch.",
> however, I found multiple old review comments not addressed. Please
> give a reason if a comment is deliberately left out, otherwise, I will
> assume they are omitted by accident and so keep repeating them.
>
> There were also still some unanswered questions from previous reviews,
> so I have reminded you about those again here.
>
> ======
> Commit message
>
> 1.
> This commit enables support for the 'publish_generated_columns' option
> in logical replication, allowing the transmission of generated column
> information and data alongside regular table changes. The option
> 'publish_generated_columns' is a PUBLICATION parameter.
>
> ~
>
> That PUBLICATION info in the 2nd sentence would be easier to say in
> the 1st sentence.
> SUGGESTION:
> This commit supports the transmission of generated column information
> and data alongside regular table changes. This behaviour is controlled
> by a new PUBLICATION parameter ('publish_generated_columns').
>
> ~~~
>
> 2.
> When 'publish_generated_columns' is false, generated columns are not
> replicated, even when present in a PUBLICATION col-list.
>
> Hm. This contradicts the behaviour that Amit wanted, (e.g.
> "column-list takes precedence"). So I am not sure if this patch is
> already catering for the behaviour suggested by Amit or if that is yet
> to come in v33. For now, I am assuming that 32* has not caught up with
> the latest behaviour requirements, but that might be a wrong
> assumption; perhaps it is only this commit message that is bogus.
>
> ~~~
>
> 3. General.
>
> On the same subject, there is lots of code, like:
>
> if (att->attgenerated && !pub->pubgencols)
> continue;
>
> I suspect that might not be quite what you want for the "column-list
> takes precedence" behaviour, but I am not going to identify all those
> during this review. It needs lots of combinations of column list tests
> to verify it.
>
> ======
> doc/src/sgml/ddl.sgml
>
> 4ab.
> nit - Huh?? Not changed the linkend as told in a previous review [1-#3a]
> nit - Huh?? Not changed to call this a "parameter" instead of an
> "option" as told in a previous review [1-#3b]
>
> ======
> doc/src/sgml/protocol.sgml
>
> 5.
> -     <para>
> -      Next, the following message part appears for each column included in
> -      the publication (except generated columns):
> -     </para>
> -
>
> nit -- Huh?? I don't think you can just remove this whole paragraph.
> But, probably you can just remove the "except generated columns" part.
> I posted this same comment [4 #11] 20 patch versions back.
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 6abc.
> nit - Huh?? Not changed the parameter ID as told in a previous review [1-#6]
> nit - Huh?? Not removed paragraph "This option is only available..."
> as told in a previous review. See [1-#7]
> nit - Huh?? Not removed paragraph "This parameter can only be set" as
> told in a previous review. See [1-#8]
>
> ======
> src/backend/catalog/pg_publication.c
>
> 7.
>   if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> - ereport(ERROR,
> + ereport(WARNING,
>   errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> - errmsg("cannot use generated column \"%s\" in publication column list",
> + errmsg("specified generated column \"%s\" in publication column list
> for publication with publish_generated_columns as false",
>      colname));
>
> I did not understand how this WARNING can know
> "publish_generated_columns as false"? Should the code be checking the
> function parameter 'pubgencols'?
>
> The errmsg also seemed a bit verbose. How about:
> "specified generated column \"%s\" in publication column list when
> publish_generated_columns = false"
>
> ======
> src/backend/replication/logical/proto.c
>
> 8.
> logicalrep_write_tuple:
> logicalrep_write_attrs:
>
> Reminder. I think I have multiple questions about this code from
> previous reviews that may be still unanswered. See [2 #4]. Maybe when
> you implement Amit's "column list takes precedence" behaviour then
> this code is fine as-is (because the replication message might include
> gencols or not-gecols regardless of the 'publish_generated_columns'
> value). But I don't think that is the current implementation, so
> something did not quite seem right. I am not sure. If you say it is
> fine then I will believe it, but the question [2 #4] remains
> unanswered.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 9.
> send_relation_and_attrs:
>
> Reminder: Here is another question that was answered from [2 #5]. I
> did not really trust it for the current implementation, but for the
> "column list takes precedence" behaviour probably it will be ok.
>
> ~~~
>
> 10.
> +/*
> + * Prepare new column list bitmap. This includes all the columns of the table.
> + */
> +static Bitmapset *
> +prepare_all_columns_bms(PGOutputData *data, RelationSyncEntry *entry,
> + TupleDesc desc)
> +{
>
> This function needs a better comment with more explanation about what
> this is REALLY doing. e.g. it says "includes all columns of the
> table", but tthe implementation is skipping generated cols, so clearly
> it is not "all columns of the table".
>
> ~~~
>
> 11. pgoutput_column_list_init
>
> TBH, I struggle to read the logic of this function. Rewriting some
> parts, inverting some variables, and adding more commentary might help
> a lot.
>
> 11a.
> There are too many "negatives" (with ! operator and with the word "no"
> in the variable).
>
> e.g. code is written in a backward way like:
> if (!pub_no_list)
> cols = pub_collist_to_bitmapset(cols, cfdatum, entry->entry_cxt);
> else
> cols = prepare_all_columns_bms(data, entry, desc);
>
> instead of what could have been said:
> if (pub_rel_has_collist)
> cols = pub_collist_to_bitmapset(cols, cfdatum, entry->entry_cxt);
> else
> cols = prepare_all_columns_bms(data, entry, desc);
>
> ~
>
> 11b.
> - * If the publication is FOR ALL TABLES then it is treated the same as
> - * if there are no column lists (even if other publications have a
> - * list).
> + * If the publication is FOR ALL TABLES and include generated columns
> + * then it is treated the same as if there are no column lists (even
> + * if other publications have a list).
>   */
> - if (!pub->alltables)
> + if (!pub->alltables || !pub->pubgencols)
>
> The code does not appear to match the comment ("If the publication is
> FOR ALL TABLES and include generated columns"). If it did it should
> look like "if (pub->alltables && pub->pubgencols)".
>
> Also, should "and include generated column" be properly referring to
> the new PUBLICATION parameter name?
>
> Also, the comment is somewhat confusing. I saw in the thread Vignesh
> wrote an explanation like "To handle cases where the
> publish_generated_columns option isn't specified for all tables in a
> publication, the pubgencolumns check needs to be performed. In such
> cases, we must create a column list that excludes generated columns"
> [3]. IMO that was clearer information so something similar should be
> written in this code comment.
> ~
>
> 11c.
> + /* Build the column list bitmap in the per-entry context. */
> + if (!pub_no_list || !pub->pubgencols) /* when not null */
>
> I don't know what "when not null" means here. Aren't those both
> booleans? How can it be "null"?
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> 12. getPublications:
>
> Huh?? The code has not changed to address an old review comment I had
> posted to say there seem multiple code fragments missing that should
> say "false AS pubgencols". Refer to [2 #7].
>
> ======
> src/bin/pg_dump/t/002_pg_dump.pl
>
> 13.
> 'ALTER PUBLICATION pub5 ADD TABLE test_table WHERE (col1 > 0);' => {
> + create_order => 51,
> + create_sql =>
> +   'ALTER PUBLICATION pub5 ADD TABLE dump_test.test_table WHERE (col1 > 0);',
> + regexp => qr/^
> + \QALTER PUBLICATION pub5 ADD TABLE ONLY dump_test.test_table WHERE
> ((col1 > 0));\E
> + /xm,
> + like => { %full_runs, section_post_data => 1, },
> + unlike => {
> + exclude_dump_test_schema => 1,
> + exclude_test_table => 1,
> + },
> + },
> +
> + 'ALTER PUBLICATION pub5 ADD TABLE test_second_table WHERE (col2 = \'test\');'
> +   => {
> + create_order => 52,
> + create_sql =>
> +   'ALTER PUBLICATION pub5 ADD TABLE dump_test.test_second_table
> WHERE (col2 = \'test\');',
> + regexp => qr/^
> + \QALTER PUBLICATION pub5 ADD TABLE ONLY dump_test.test_second_table
> WHERE ((col2 = 'test'::text));\E
> + /xm,
> + like => { %full_runs, section_post_data => 1, },
> + unlike => { exclude_dump_test_schema => 1, },
> +   },
> +
>
> It wasn't clear to me how these tests are related to the patch.
> Shouldn't there instead be some ALTER tests for trying to modify the
> 'publish_generate_columns' parameter?
>
> ======
> src/test/regress/expected/publication.out
> src/test/regress/sql/publication.sql
>
> 14.
> --- error: generated column "d" can't be in list
> +-- ok: generated columns can be in the list too
>  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
> -ERROR:  cannot use generated column "d" in publication column list
> +WARNING:  specified generated column "d" in publication column list
> for publication with publish_generated_columns as false
>
> I think these tests for the WARNING scenario need to be a bit more
> deliberate. This seems to have happened as a side-effect. For example,
> I was expecting more testing like:
>
> Comments about various combinations to say what you are doing and what
> you are expecting:
> - gencols in column list with publish_generated_columns=false, expecting WARNING
> - gencols in column list with publish_generated_columns=true, NOT
> expecting WARNING
> - gencols in column list with publish_generated_columns=true, then
> ALTER PUBLICATION setting publication_generate_columns=false,
> expecting WARNING
> - NO gencols in column list with publish_generated_columns=false, then
> ALTER PUBLICATION to add gencols to column list, expecting WARNING
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 15.
> -# TEST: Generated and dropped columns are not considered for the column list.
> +# TEST: Dropped columns are not considered for the column list.
>  # So, the publication having a column list except for those columns and a
> -# publication without any column (aka all columns as part of the columns
> +# publication without any column list (aka all columns as part of the column
>  # list) are considered to have the same column list.
>  $node_publisher->safe_psql(
>   'postgres', qq(
>   CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int
> GENERATED ALWAYS AS (a + 1) STORED);
>   ALTER TABLE test_mix_4 DROP COLUMN c;
>
> - CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
> - CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
> + CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 WITH
> (publish_generated_columns = true);
> + CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4 WITH
> (publish_generated_columns = false);
>
> I felt the comment for this test ought to be saying something more
> about what you are doing with the 'publish_generated_columns'
> parameters and what behaviour it was expecting.
>
> ======
> Please refer to the attachment which addresses some of the nit
> comments mentioned above.
>
> ======
> [1] my review of v31-0001:
> https://www.postgresql.org/message-id/CAHut%2BPsv-neEP_ftvBUBahh%2BKCWw%2BqQMF9N3sGU3YHWPEzFH-Q%40mail.gmail.com
> [2] my review of v30-0001:
> https://www.postgresql.org/message-id/CAHut%2BPuaitgE4tu3nfaR%3DPCQEKjB%3DmpDtZ1aWkbwb%3DJZE8YvqQ%40mail.gmail.com
> [3] https://www.postgresql.org/message-id/CALDaNm1c7xPBodHw6LKp9e8hvGVJHcKH%3DDHK0iXmZuXKPnxZ3Q%40mail.gmail.com
> [4] https://www.postgresql.org/message-id/CAHut%2BPv45gB4cV%2BSSs6730Kb8urQyqjdZ9PBVgmpwqCycr1Ybg%40mail.gmail.com
>

I have addressed all the comments in the v34-0001 Patch. Please refer
to the updated v34-0001 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjJkUdYCdK_bL3yvEV%3DzKrA2dsnZYa1VMT2H5v0%2BqbaGbA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  166. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-09-27T14:46:54Z

    On Tue, Sep 24, 2024 at 7:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi. Here are my v32-0002 review comments:
>
> ======
> src/backend/replication/logical/tablesync.c
>
> 1. fetch_remote_table_info
>
>   /*
> - * Get column lists for each relation.
> + * Get column lists for each relation, and check if any of the
> + * publications have the 'publish_generated_columns' parameter enabled.
>
> I am not 100% sure about this logic anymore. Maybe it is OK, but it
> requires careful testing because with Amit's "column lists take
> precedence" it is now possible for the publication to say
> 'publish_generated_columns=false', but the publication can still
> publish gencols *anyway* if they were specified in a column list.
>
> ~~~
>

This comment is still open. Will fix this in the next version of patches.

> 2.
>   /*
>   * Fetch info about column lists for the relation (from all the
>   * publications).
>   */
> + StringInfo pub_names = makeStringInfo();
> +
> + get_publications_str(MySubscription->publications, pub_names, true);
>   resetStringInfo(&cmd);
>   appendStringInfo(&cmd,
> ~
>
> nit - The comment here seems misplaced.
>
> ~~~
>
> 3.
> + if (server_version >= 120000)
> + {
> + has_pub_with_pubgencols = server_version >= 180000 && has_pub_with_pubgencols;
> +
> + if (!has_pub_with_pubgencols)
> + appendStringInfo(&cmd, " AND a.attgenerated = ''");
> + }
>
> My previous review comment about this [1 #10] was:
> Can the 'gencols_allowed' var be removed, and the condition just be
> replaced with if (!has_pub_with_pubgencols)? It seems equivalent
> unless I am mistaken.
>
> nit - So the current v32 code is not what I was expecting. What I
> meant was 'has_pub_with_pubgencols' can only be true if server_version
> >= 180000, so I thought there was no reason to check it again. For
> reference, I've changed it to like I meant in the nitpicks attachment.
> Please see if that works the same.
>
> ======
> [1] my review of v31-0002.
> https://www.postgresql.org/message-id/CAHut%2BPusbhvPrL1uN1TKY%3DFd4zu3h63eDebZvsF%3Duy%2BLBKTwgA%40mail.gmail.com
>

I have addressed all the comments in the v34-0002 Patch. Please refer
to the updated v34-0002 Patch here in [1]. See [1] for the changes
added.

[1] https://www.postgresql.org/message-id/CAHv8RjJkUdYCdK_bL3yvEV%3DzKrA2dsnZYa1VMT2H5v0%2BqbaGbA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  167. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-30T06:17:05Z

    Here are my review comments for patch v34-0001

======
doc/src/sgml/ddl.sgml

1.
-      Generated columns are skipped for logical replication and cannot be
-      specified in a <command>CREATE PUBLICATION</command> column list.
+      Generated columns may be skipped during logical replication
according to the
+      <command>CREATE PUBLICATION</command> parameter
+      <link linkend="sql-createpublication-params-with-publish-generated-columns">
+      <literal>publish_generated_columns</literal></link>.

This information is not quite correct because it makes no mention of
PUBLICATION column lists. OTOH I replaced all this paragraph in the
0003 patch anyhow, so maybe it is not worth worrying about this review
comment.

======
src/backend/catalog/pg_publication.c

2.
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated && !pubgencols)
+ ereport(WARNING,
  errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
+ errmsg("specified generated column \"%s\" in publication column list
when publish_generated_columns as false",
     colname));

Back when I proposed to have this WARNING I don't think there existed
the idea that the PUBLICATION column list would override the
'publish_generated_columns' parameter. But now that this is the
implementation, I am no longer 100% sure if a warning should be given
at all because this can be a perfectly valid combination. What do
others think?

======
src/backend/replication/pgoutput/pgoutput.c

3. prepare_all_columns_bms

3a.
nit - minor rewording function comment

~

3b.
I am not sure this is a good function name, particularly since it does
not return "all" columns. Can you name it more for what it does?

~~~

4. pgoutput_column_list_init

  /*
- * If the publication is FOR ALL TABLES then it is treated the same as
- * if there are no column lists (even if other publications have a
- * list).
+ * To handle cases where the publish_generated_columns option isn't
+ * specified for all tables in a publication, the pubgencolumns check
+ * needs to be performed. In such cases, we must create a column list
+ * that excludes generated columns.
  */
- if (!pub->alltables)
+ if (!pub->alltables || !pub->pubgencols)

4a.
That comment still doesn't make much sense to me:

e.g.1. "To handle cases where the publish_generated_columns option
isn't specified for all tables in a publication". What is this trying
to say? A publication parameter is per-publication, so it is always
"for all tables in a publication".

e.g.2. " the pubgencolumns check" -- what is the pubgencols check?

Please rewrite the comment more clearly to explain the logic: What is
it doing? Why is it doing it?

~

4b.
+ if (!pub->alltables || !pub->pubgencols)

As mentioned in a previous review, there is too much negativity in
conditions like this. I think anything you can do to reverse all the
negativity will surely improve the readability of this function. See
[1 - #11a]


~

5.
- cols = pub_collist_to_bitmapset(cols, cfdatum,
- entry->entry_cxt);
+ if (!pub_rel_has_collist)
+ cols = pub_collist_to_bitmapset(cols, cfdatum, entry->entry_cxt);
+ else
+ cols = prepare_all_columns_bms(data, entry, desc);

Hm. Is that correct? The if/else is all backwards from what I would
have expected. IIUC the variable 'pub_rel_has_collist' is assigned to
the opposite value of what the name says, so then you are using it all
backwards to make it work.

nit - I have changed the code in the attachment how I think it should
be. Please check it makes sense.

~

6.
+ /* Get the number of live attributes. */
+ for (i = 0; i < desc->natts; i++)

nit - use a for-loop variable 'i'.

======
src/bin/pg_dump/pg_dump.c

7.
+ else if (fout->remoteVersion >= 130000)
+ appendPQExpBufferStr(query,
+ "SELECT p.tableoid, p.oid, p.pubname, "
+ "p.pubowner, "
+ "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
p.pubtruncate, p.pubviaroot, false AS pubviagencols "
  "FROM pg_publication p");
  else if (fout->remoteVersion >= 110000)
  appendPQExpBufferStr(query,
  "SELECT p.tableoid, p.oid, p.pubname, "
  "p.pubowner, "
- "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
p.pubtruncate, false AS pubviaroot "
+ "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
p.pubtruncate, false AS pubviaroot, false AS pubviagencols "
  "FROM pg_publication p");
  else
  appendPQExpBufferStr(query,
  "SELECT p.tableoid, p.oid, p.pubname, "
  "p.pubowner, "
- "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS
pubtruncate, false AS pubviaroot "
+ "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS
pubtruncate, false AS pubviaroot, false AS pubviagencols "
  "FROM pg_publication p");

These changes are all wrong due to a typo. There is no such column as
'pubviagencols'. I made these changes in the nit attachment. Please
check it for correctness.

s/pubviagencols/pubgencols/

======
src/test/regress/sql/publication.sql

8.
--- error: generated column "d" can't be in list
+-- ok: generated columns can be in the list too

nit - name the generated column "d", to clarify this comment

~~~

9.
+-- Test the publication 'publish_generated_columns' parameter enabled
or disabled for different combinations.

nit - add another "======" separator comment before the new test
nit - some minor adjustments to whitespace and comments

~~~

10.
+-- No gencols in column list with 'publish_generated_columns'=false.
+CREATE PUBLICATION pub3 WITH (publish_generated_columns=false);
+-- ALTER PUBLICATION to add gencols to column list.
+ALTER PUBLICATION pub3 ADD TABLE gencols(a, gen1);

nit - by adding and removing collist for the existing table, we don't
need to have a 'pub3'.

======
src/test/subscription/t/031_column_list.pl

11.
It is not clear to me -- is there, or is there not yet any test case
for the multiple publication issues that were discussed previously?
e.g. when the same table has gencols but there are multiple
publications for the same subscription and the
'publish_generated_columns' parameter or column lists conflict.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPt9ArtYc1Vtb1DqzEEFwjcoDHMyRCnUqkHQeFJuMCDkvQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  168. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-30T07:26:06Z

    Hi Shubham. Here are my review comment for patch v34-0002.

======
doc/src/sgml/ref/create_publication.sgml

1.
+         <para>
+         This parameter can only be set <literal>true</literal> if
<literal>copy_data</literal> is
+         set to <literal>false</literal>.
+         </para>

Huh? AFAIK the patch implements COPY for generated columns, so why are
you saying this limitation?

======
src/backend/replication/logical/tablesync.c

2. reminder

Previously (18/9) [1 #4] I wrote maybe that other copy_data=false
"missing" case error can be improved to share the same error message
that you have in make_copy_attnamelist. And you replied [2] it would
be addressed in the next patchset, but that was at least 2 versions
back and I don't see any change yet.

======
[1] 18/9 review
https://www.postgresql.org/message-id/CAHut%2BPusbhvPrL1uN1TKY%3DFd4zu3h63eDebZvsF%3Duy%2BLBKTwgA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHv8RjJ5_dmyCH58xQ0StXMdPt9gstemMMWytR79%2BLfOMAHdLw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  169. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-30T08:33:39Z

    Hi, here is an updated patch set v35:

v35-0001 (same as v34-0001)
v35-0002 (same as v34-0002)

v35-0003 (updated)
- fixed a linkend problem of v34-0003.
- added info in "Column Lists" about generated cols and linked that to
the new docs section.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  170. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-09-30T08:45:54Z

    Hi Vignesh,

On Mon, Sep 23, 2024 at 10:49 PM vignesh C <vignesh21@gmail.com> wrote:
>

> 3) In create publication column list/publish_generated_columns
> documentation we should mention that if generated column is mentioned
> in column list, generated columns mentioned in column list will be
> replication irrespective of publish_generated_columns option.
>

v34-0003 introduced a new Chapter 29 (Logical Replication) section for
"Generated Column Replication"
- This version also added a link from CREATE PUBLICATION
'publish_generated_column' parameter to this new section

To address your column list point, in v35-0003 I added more
information about Generate Columns in the Chapter 29 section "Column
List". The CREATE PUBLICATION column lists docs already linked to
that. See [1]

======
[1] v35-0003 - https://www.postgresql.org/message-id/CAHut%2BPvoQS9HjcGFZrTHrUQZ8vzyfAcSgeTgQEoO_-f8CrhW4A%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  171. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-03T00:09:37Z

    Hi Shubham,

The different meanings of the terms "parameter" versus "option" were
discussed in a recent thread [1], and that has made me reconsider this
generated columns feature.

Despite being in the PUBLICATION section "WITH ( publication_parameter
[= value] [, ... ] )", I think that 'publish_generated_columns' is an
"option" (not a parameter).

We should update all those places that are currently calling it a parameter:
- commit messages
- docs
- comments
- etc.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPuiRydyrYfMzR1OxOnVJf-_G8OBCLdyqu8jJ8si51d%2BEQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  172. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-03T00:53:26Z

    On Thu, Oct 3, 2024 at 10:09 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham,
>
> The different meanings of the terms "parameter" versus "option" were
> discussed in a recent thread [1], and that has made me reconsider this
> generated columns feature.
>
> Despite being in the PUBLICATION section "WITH ( publication_parameter
> [= value] [, ... ] )", I think that 'publish_generated_columns' is an
> "option" (not a parameter).
>
> We should update all those places that are currently calling it a parameter:
> - commit messages
> - docs
> - comments
> - etc.
>
> ======
> [1] https://www.postgresql.org/message-id/CAHut%2BPuiRydyrYfMzR1OxOnVJf-_G8OBCLdyqu8jJ8si51d%2BEQ%40mail.gmail.com
>

It seems there are differing opinions on that other thread about what
term to use. Probably, it is best to just leave the above suggestion
alone for now.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  173. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-03T11:00:14Z

    On Mon, Sep 30, 2024 at 11:47 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for patch v34-0001
>
> ======
> doc/src/sgml/ddl.sgml
>
> 1.
> -      Generated columns are skipped for logical replication and cannot be
> -      specified in a <command>CREATE PUBLICATION</command> column list.
> +      Generated columns may be skipped during logical replication
> according to the
> +      <command>CREATE PUBLICATION</command> parameter
> +      <link linkend="sql-createpublication-params-with-publish-generated-columns">
> +      <literal>publish_generated_columns</literal></link>.
>
> This information is not quite correct because it makes no mention of
> PUBLICATION column lists. OTOH I replaced all this paragraph in the
> 0003 patch anyhow, so maybe it is not worth worrying about this review
> comment.
>
> ======
> src/backend/catalog/pg_publication.c
>
> 2.
> - if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> - ereport(ERROR,
> + if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated && !pubgencols)
> + ereport(WARNING,
>   errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> - errmsg("cannot use generated column \"%s\" in publication column list",
> + errmsg("specified generated column \"%s\" in publication column list
> when publish_generated_columns as false",
>      colname));
>
> Back when I proposed to have this WARNING I don't think there existed
> the idea that the PUBLICATION column list would override the
> 'publish_generated_columns' parameter. But now that this is the
> implementation, I am no longer 100% sure if a warning should be given
> at all because this can be a perfectly valid combination. What do
> others think?
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 3. prepare_all_columns_bms
>
> 3a.
> nit - minor rewording function comment
>
> ~
>
> 3b.
> I am not sure this is a good function name, particularly since it does
> not return "all" columns. Can you name it more for what it does?
>
> ~~~
>
> 4. pgoutput_column_list_init
>
>   /*
> - * If the publication is FOR ALL TABLES then it is treated the same as
> - * if there are no column lists (even if other publications have a
> - * list).
> + * To handle cases where the publish_generated_columns option isn't
> + * specified for all tables in a publication, the pubgencolumns check
> + * needs to be performed. In such cases, we must create a column list
> + * that excludes generated columns.
>   */
> - if (!pub->alltables)
> + if (!pub->alltables || !pub->pubgencols)
>
> 4a.
> That comment still doesn't make much sense to me:
>
> e.g.1. "To handle cases where the publish_generated_columns option
> isn't specified for all tables in a publication". What is this trying
> to say? A publication parameter is per-publication, so it is always
> "for all tables in a publication".
>
> e.g.2. " the pubgencolumns check" -- what is the pubgencols check?
>
> Please rewrite the comment more clearly to explain the logic: What is
> it doing? Why is it doing it?
>
> ~
>
> 4b.
> + if (!pub->alltables || !pub->pubgencols)
>
> As mentioned in a previous review, there is too much negativity in
> conditions like this. I think anything you can do to reverse all the
> negativity will surely improve the readability of this function. See
> [1 - #11a]
>
>
> ~
>
> 5.
> - cols = pub_collist_to_bitmapset(cols, cfdatum,
> - entry->entry_cxt);
> + if (!pub_rel_has_collist)
> + cols = pub_collist_to_bitmapset(cols, cfdatum, entry->entry_cxt);
> + else
> + cols = prepare_all_columns_bms(data, entry, desc);
>
> Hm. Is that correct? The if/else is all backwards from what I would
> have expected. IIUC the variable 'pub_rel_has_collist' is assigned to
> the opposite value of what the name says, so then you are using it all
> backwards to make it work.
>
> nit - I have changed the code in the attachment how I think it should
> be. Please check it makes sense.
>
> ~
>
> 6.
> + /* Get the number of live attributes. */
> + for (i = 0; i < desc->natts; i++)
>
> nit - use a for-loop variable 'i'.
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> 7.
> + else if (fout->remoteVersion >= 130000)
> + appendPQExpBufferStr(query,
> + "SELECT p.tableoid, p.oid, p.pubname, "
> + "p.pubowner, "
> + "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
> p.pubtruncate, p.pubviaroot, false AS pubviagencols "
>   "FROM pg_publication p");
>   else if (fout->remoteVersion >= 110000)
>   appendPQExpBufferStr(query,
>   "SELECT p.tableoid, p.oid, p.pubname, "
>   "p.pubowner, "
> - "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
> p.pubtruncate, false AS pubviaroot "
> + "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
> p.pubtruncate, false AS pubviaroot, false AS pubviagencols "
>   "FROM pg_publication p");
>   else
>   appendPQExpBufferStr(query,
>   "SELECT p.tableoid, p.oid, p.pubname, "
>   "p.pubowner, "
> - "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS
> pubtruncate, false AS pubviaroot "
> + "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS
> pubtruncate, false AS pubviaroot, false AS pubviagencols "
>   "FROM pg_publication p");
>
> These changes are all wrong due to a typo. There is no such column as
> 'pubviagencols'. I made these changes in the nit attachment. Please
> check it for correctness.
>
> s/pubviagencols/pubgencols/
>
> ======
> src/test/regress/sql/publication.sql
>
> 8.
> --- error: generated column "d" can't be in list
> +-- ok: generated columns can be in the list too
>
> nit - name the generated column "d", to clarify this comment
>
> ~~~
>
> 9.
> +-- Test the publication 'publish_generated_columns' parameter enabled
> or disabled for different combinations.
>
> nit - add another "======" separator comment before the new test
> nit - some minor adjustments to whitespace and comments
>
> ~~~
>
> 10.
> +-- No gencols in column list with 'publish_generated_columns'=false.
> +CREATE PUBLICATION pub3 WITH (publish_generated_columns=false);
> +-- ALTER PUBLICATION to add gencols to column list.
> +ALTER PUBLICATION pub3 ADD TABLE gencols(a, gen1);
>
> nit - by adding and removing collist for the existing table, we don't
> need to have a 'pub3'.
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 11.
> It is not clear to me -- is there, or is there not yet any test case
> for the multiple publication issues that were discussed previously?
> e.g. when the same table has gencols but there are multiple
> publications for the same subscription and the
> 'publish_generated_columns' parameter or column lists conflict.
>
I have moved the new test cases to the 011_generated.pl file and
created a separate patch . I will post the TAP-TESTS once these
patches get fixed.
I have fixed all the given comments. The attached v36-0001 patch
contain the desired changes.

Thanks and Regards,
Shubham Khanna.

  174. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-03T11:04:56Z

    On Mon, Sep 30, 2024 at 12:56 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham. Here are my review comment for patch v34-0002.
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 1.
> +         <para>
> +         This parameter can only be set <literal>true</literal> if
> <literal>copy_data</literal> is
> +         set to <literal>false</literal>.
> +         </para>
>
> Huh? AFAIK the patch implements COPY for generated columns, so why are
> you saying this limitation?
>
> ======

I have fixed this in the v36-0002 patch.

> src/backend/replication/logical/tablesync.c
>
> 2. reminder
>
> Previously (18/9) [1 #4] I wrote maybe that other copy_data=false
> "missing" case error can be improved to share the same error message
> that you have in make_copy_attnamelist. And you replied [2] it would
> be addressed in the next patchset, but that was at least 2 versions
> back and I don't see any change yet.
>
This comment is still open. Will fix this and post in the next version
of patches.

Please refer to the updated v36-0002 Patch here in [1]. See [1] for
the changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2B1RDd7AnJNzOJXk--zcbTtU3nys%3DZgU3ktB4e3DWbJgg%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  175. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-04T04:05:52Z

    Hi Shubham, here are my review comments for v36-0001.

======
1. General  - merge patches

It is long past due when patches 0001 and 0002 should've been merged.
AFAIK the split was only because historically these parts had
different authors. But, keeping them separated is not helpful anymore.

======
src/backend/catalog/pg_publication.c

2.
 Bitmapset *
-pub_collist_validate(Relation targetrel, List *columns)
+pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)

Since you removed the WARNING, this parameter 'pubgencols' is unused
so it should also be removed.

======
src/backend/replication/pgoutput/pgoutput.c

3.
  /*
- * If the publication is FOR ALL TABLES then it is treated the same as
- * if there are no column lists (even if other publications have a
- * list).
+ * To handle cases where the publish_generated_columns option isn't
+ * specified for all tables in a publication, we must create a column
+ * list that excludes generated columns. So, the publisher will not
+ * replicate the generated columns.
  */
- if (!pub->alltables)
+ if (!(pub->alltables && pub->pubgencols))

I still found that comment hard to understand. Does this mean to say
something like:

------
Process potential column lists for the following cases:

a. Any publication that is not FOR ALL TABLES.

b. When the publication is FOR ALL TABLES and
'publish_generated_columns' is false.
A FOR ALL TABLES publication doesn't have user-defined column lists,
so all columns will be replicated by default. However, if
'publish_generated_columns' is set to false, column lists must still
be created to exclude any generated columns from being published
------

======
src/test/regress/sql/publication.sql

4.
+SET client_min_messages = 'WARNING';
+CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);

AFAIK you don't need to keep changing 'client_min_messages',
particularly now that you've removed the WARNING message that was
previously emitted.

~

5.
nit - minor comment changes.

======
Please refer to the attachment which implements any nits from above.

======
Kind Regards,
Peter Smith.
Fujitsu Austrlia.

  176. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-04T04:12:55Z

    Hi Shubham, I don't have any new comments for the patch v36-0002.

But, according to my records, there are multiple old comments not yet
addressed for this patch. I am giving reminders for those below so
they don't get accidentally overlooked. Please re-confirm and at the
next posted version please respond individually to each of these to
say if they are addressed or not.

======

1. General
>From review v31 [1] comment #1. Patches 0001 and 0002 should be merged.

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:

2.
>From review v31 [1] comment #4. Make the detailed useful error message
common if possible.

~~~

fetch_remote_table_info:

3.
>From review v31 [1] comment #5. I was not sure if this logic is
sophisticated enough to handle the case when the same table has
gencols but there are multiple subscribed publications and the
'publish_generated_columns' parameter differs. Is this scenario
tested?

~

4.
+ * Get column lists for each relation, and check if any of the
+ * publications have the 'publish_generated_columns' parameter enabled.

From review v32 [2] comment #1. This needs some careful testing. I was
not sure if sufficient to just check the 'publish_generated_columns'
flag. Now that "column lists take precedence" it is quite possible for
all publications to say 'publish_generated_columns=false', but the
publication can still publish gencols *anyway* if they are specified
in a column list.

======
[1] review v31 18/9 -
https://www.postgresql.org/message-id/flat/CAHv8Rj%2BKOoh58Uf5k2MN-%3DA3VdV60kCVKCh5ftqYxgkdxFSkqg%40mail.gmail.com#f2f3b48080f96ea45e1410f5b1cd9735
[2] review v32 24/9 -
https://www.postgresql.org/message-id/CAHut%2BPu7EcK_JTgWS7GzeStHk6Asb1dmEzCJU2TJf%2BW1Zy30LQ%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  177. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-08T06:07:16Z

    On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham, here are my review comments for v36-0001.
>
> ======
> 1. General  - merge patches
>
> It is long past due when patches 0001 and 0002 should've been merged.
> AFAIK the split was only because historically these parts had
> different authors. But, keeping them separated is not helpful anymore.
>
> ======
> src/backend/catalog/pg_publication.c
>
> 2.
>  Bitmapset *
> -pub_collist_validate(Relation targetrel, List *columns)
> +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
>
> Since you removed the WARNING, this parameter 'pubgencols' is unused
> so it should also be removed.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 3.
>   /*
> - * If the publication is FOR ALL TABLES then it is treated the same as
> - * if there are no column lists (even if other publications have a
> - * list).
> + * To handle cases where the publish_generated_columns option isn't
> + * specified for all tables in a publication, we must create a column
> + * list that excludes generated columns. So, the publisher will not
> + * replicate the generated columns.
>   */
> - if (!pub->alltables)
> + if (!(pub->alltables && pub->pubgencols))
>
> I still found that comment hard to understand. Does this mean to say
> something like:
>
> ------
> Process potential column lists for the following cases:
>
> a. Any publication that is not FOR ALL TABLES.
>
> b. When the publication is FOR ALL TABLES and
> 'publish_generated_columns' is false.
> A FOR ALL TABLES publication doesn't have user-defined column lists,
> so all columns will be replicated by default. However, if
> 'publish_generated_columns' is set to false, column lists must still
> be created to exclude any generated columns from being published
> ------
>
> ======
> src/test/regress/sql/publication.sql
>
> 4.
> +SET client_min_messages = 'WARNING';
> +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
>
> AFAIK you don't need to keep changing 'client_min_messages',
> particularly now that you've removed the WARNING message that was
> previously emitted.
>
> ~
>
> 5.
> nit - minor comment changes.
>
> ======
> Please refer to the attachment which implements any nits from above.
>

I have fixed all the given comments. Also, I have created a new 0003
patch for the TAP-Tests related to the '011_generated.pl' file. I am
planning to merge 0001 and 0003 patches once they will get fixed.
The attached patches contain the required changes.

Thanks and Regards,
Shubham Khanna.

  178. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-08T06:10:59Z

    On Fri, Oct 4, 2024 at 9:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Shubham, I don't have any new comments for the patch v36-0002.
>
> But, according to my records, there are multiple old comments not yet
> addressed for this patch. I am giving reminders for those below so
> they don't get accidentally overlooked. Please re-confirm and at the
> next posted version please respond individually to each of these to
> say if they are addressed or not.
>
> ======
>
> 1. General
> From review v31 [1] comment #1. Patches 0001 and 0002 should be merged.
>
> ======
> src/backend/replication/logical/tablesync.c
>
> make_copy_attnamelist:
>
> 2.
> From review v31 [1] comment #4. Make the detailed useful error message
> common if possible.
>
> ~~~

This comment is still open. Will fix this in next versions of patches.

>
> fetch_remote_table_info:
>
> 3.
> From review v31 [1] comment #5. I was not sure if this logic is
> sophisticated enough to handle the case when the same table has
> gencols but there are multiple subscribed publications and the
> 'publish_generated_columns' parameter differs. Is this scenario
> tested?
>
> ~
>
> 4.
> + * Get column lists for each relation, and check if any of the
> + * publications have the 'publish_generated_columns' parameter enabled.
>
> From review v32 [2] comment #1. This needs some careful testing. I was
> not sure if sufficient to just check the 'publish_generated_columns'
> flag. Now that "column lists take precedence" it is quite possible for
> all publications to say 'publish_generated_columns=false', but the
> publication can still publish gencols *anyway* if they are specified
> in a column list.
>
> ======
> [1] review v31 18/9 -
> https://www.postgresql.org/message-id/flat/CAHv8Rj%2BKOoh58Uf5k2MN-%3DA3VdV60kCVKCh5ftqYxgkdxFSkqg%40mail.gmail.com#f2f3b48080f96ea45e1410f5b1cd9735
> [2] review v32 24/9 -
> https://www.postgresql.org/message-id/CAHut%2BPu7EcK_JTgWS7GzeStHk6Asb1dmEzCJU2TJf%2BW1Zy30LQ%40mail.gmail.com
>

I have fixed the comments and posted the v37 patches for them. Please
refer to the updated v37 Patches here in [1]. See [1] for
the changes added.

[1] https://www.postgresql.org/message-id/CAHv8Rj%2BRnw%2B_SfSyyrvWL49AfJzx4O8YVvdU9gB%2BSQdt3%3DqF%2BA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  179. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-09T03:38:04Z

    On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
>
> On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham, here are my review comments for v36-0001.
> >
> > ======
> > 1. General  - merge patches
> >
> > It is long past due when patches 0001 and 0002 should've been merged.
> > AFAIK the split was only because historically these parts had
> > different authors. But, keeping them separated is not helpful anymore.
> >
> > ======
> > src/backend/catalog/pg_publication.c
> >
> > 2.
> >  Bitmapset *
> > -pub_collist_validate(Relation targetrel, List *columns)
> > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> >
> > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > so it should also be removed.
> >
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 3.
> >   /*
> > - * If the publication is FOR ALL TABLES then it is treated the same as
> > - * if there are no column lists (even if other publications have a
> > - * list).
> > + * To handle cases where the publish_generated_columns option isn't
> > + * specified for all tables in a publication, we must create a column
> > + * list that excludes generated columns. So, the publisher will not
> > + * replicate the generated columns.
> >   */
> > - if (!pub->alltables)
> > + if (!(pub->alltables && pub->pubgencols))
> >
> > I still found that comment hard to understand. Does this mean to say
> > something like:
> >
> > ------
> > Process potential column lists for the following cases:
> >
> > a. Any publication that is not FOR ALL TABLES.
> >
> > b. When the publication is FOR ALL TABLES and
> > 'publish_generated_columns' is false.
> > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > so all columns will be replicated by default. However, if
> > 'publish_generated_columns' is set to false, column lists must still
> > be created to exclude any generated columns from being published
> > ------
> >
> > ======
> > src/test/regress/sql/publication.sql
> >
> > 4.
> > +SET client_min_messages = 'WARNING';
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> >
> > AFAIK you don't need to keep changing 'client_min_messages',
> > particularly now that you've removed the WARNING message that was
> > previously emitted.
> >
> > ~
> >
> > 5.
> > nit - minor comment changes.
> >
> > ======
> > Please refer to the attachment which implements any nits from above.
> >
>
> I have fixed all the given comments. Also, I have created a new 0003
> patch for the TAP-Tests related to the '011_generated.pl' file. I am
> planning to merge 0001 and 0003 patches once they will get fixed.
> The attached patches contain the required changes.

Few comments:
1) Since we are no longer throwing an error for generated columns, the
function header comments also need to be updated accordingly " Checks
for and raises an ERROR for any; unknown columns, system columns,
duplicate columns or generated columns."
-               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
-                       ereport(ERROR,
-
errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
-                                       errmsg("cannot use generated
column \"%s\" in publication column list",
-                                                  colname));
-

2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
ALTER PUBLICATION ... SET (
postgres=# alter publication pub2 set (PUBLISH
PUBLISH                     PUBLISH_VIA_PARTITION_ROOT

3) I was able to compile without this include, may be this is not required:
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -118,6 +118,7 @@
 #include "utils/builtins.h"
 #include "utils/lsyscache.h"
 #include "utils/memutils.h"
+#include "utils/rel.h"

4) You can include "\dRp+ pubname" after each of the create/alter
publication to verify the columns that will be published:
+-- Test the 'publish_generated_columns' parameter enabled or disabled for
+-- different scenarios with/without generated columns in column lists.
+CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
+
+-- Generated columns in column list, when 'publish_generated_columns'=false
+CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
(publish_generated_columns=false);

+-- Generated columns in column list, when 'publish_generated_columns'=true
+CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
(publish_generated_columns=true);
+
+-- Generated columns in column list, then set
'publication_generate_columns'=false
+ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
+
+-- Remove generate columns from column list, when
'publish_generated_columns'=false
+ALTER PUBLICATION pub2 SET TABLE gencols(a);
+
+-- Add generated columns in column list, when 'publish_generated_columns'=false
+ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);

Regards,
Vignesh

  180. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-09T05:30:22Z

    On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
>
> On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham, here are my review comments for v36-0001.
> >
> > ======
> > 1. General  - merge patches
> >
> > It is long past due when patches 0001 and 0002 should've been merged.
> > AFAIK the split was only because historically these parts had
> > different authors. But, keeping them separated is not helpful anymore.
> >
> > ======
> > src/backend/catalog/pg_publication.c
> >
> > 2.
> >  Bitmapset *
> > -pub_collist_validate(Relation targetrel, List *columns)
> > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> >
> > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > so it should also be removed.
> >
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 3.
> >   /*
> > - * If the publication is FOR ALL TABLES then it is treated the same as
> > - * if there are no column lists (even if other publications have a
> > - * list).
> > + * To handle cases where the publish_generated_columns option isn't
> > + * specified for all tables in a publication, we must create a column
> > + * list that excludes generated columns. So, the publisher will not
> > + * replicate the generated columns.
> >   */
> > - if (!pub->alltables)
> > + if (!(pub->alltables && pub->pubgencols))
> >
> > I still found that comment hard to understand. Does this mean to say
> > something like:
> >
> > ------
> > Process potential column lists for the following cases:
> >
> > a. Any publication that is not FOR ALL TABLES.
> >
> > b. When the publication is FOR ALL TABLES and
> > 'publish_generated_columns' is false.
> > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > so all columns will be replicated by default. However, if
> > 'publish_generated_columns' is set to false, column lists must still
> > be created to exclude any generated columns from being published
> > ------
> >
> > ======
> > src/test/regress/sql/publication.sql
> >
> > 4.
> > +SET client_min_messages = 'WARNING';
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> >
> > AFAIK you don't need to keep changing 'client_min_messages',
> > particularly now that you've removed the WARNING message that was
> > previously emitted.
> >
> > ~
> >
> > 5.
> > nit - minor comment changes.
> >
> > ======
> > Please refer to the attachment which implements any nits from above.
> >
>
> I have fixed all the given comments. Also, I have created a new 0003
> patch for the TAP-Tests related to the '011_generated.pl' file. I am
> planning to merge 0001 and 0003 patches once they will get fixed.
> The attached patches contain the required changes.

There is inconsistency in replication when a generated column is
specified in the column list. The generated column data is not
replicated during initial sync whereas it is getting replicated during
incremental sync:
-- publisher
CREATE TABLE t1(c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED)
INSERT INTO t1 VALUES (1);
CREATE PUBLICATION pub1 for table t1(c1, c2);

--subscriber
CREATE TABLE t1(c1 int, c2 int)
CREATE SUBSCRIPTION sub1 connection 'dbname=postgres host=localhost
port=5432' PUBLICATION pub1;

-- Generate column data is not synced during initial sync
postgres=# select * from t1;
 c1 | c2
----+----
  1 |
(1 row)

-- publisher
INSERT INTO t1 VALUES (2);

-- Whereas generated column data is synced during incremental sync
postgres=# select * from t1;
 c1 | c2
----+----
  1 |
  2 |  4
(2 rows)

Regards,
Vignesh

  181. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-09T05:43:11Z

    Hi, here are my review comments for patch v37-0001.

======
Commit message

1.
Example usage of subscription option:
CREATE PUBLICATION FOR TABLE tab_gencol WITH (publish_generated_columns
= true);

~

This is wrong -- it's not a "subscription option". Better to just say
"Example usage:"

~~~

2.
When 'copy_data' is true, during the initial sync, the data is replicated from
the publisher to the subscriber using the COPY command. The normal COPY
command does not copy generated columns, so when 'publish_generated_columns'
is true...

~

By only mentioning the "when ... is true" case this description does
not cover the scenario when 'publish_generated_columns' is false when
the publication column list has a generated column.

~~~

3.
typo - /replication of generated column/replication of generated columns/
typo - /filed/filled/
typo - 'pg_publicataion' catalog

======
src/backend/replication/logical/tablesync.c

make_copy_attnamelist:
4.
nit - missing word in a comment

~~~

fetch_remote_table_info:
5.
+ appendStringInfo(&cmd,
  "  FROM pg_catalog.pg_attribute a"
  "  LEFT JOIN pg_catalog.pg_index i"
  "       ON (i.indexrelid = pg_get_replica_identity_index(%u))"
  " WHERE a.attnum > 0::pg_catalog.int2"
- "   AND NOT a.attisdropped %s"
+ "   AND NOT a.attisdropped", lrel->remoteid);
+
+ appendStringInfo(&cmd,
  "   AND a.attrelid = %u"
  " ORDER BY a.attnum",
- lrel->remoteid,
- (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
-   "AND a.attgenerated = ''" : ""),
  lrel->remoteid);

Version v37-0001 has removed a condition previously between these two
appendStringInfo's. But, that now means there is no reason to keep
these statements separated. These should be combined now to use one
appendStringInfo.

~

6.
+ if (server_version >= 120000)
+ remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
+

Are you sure the version check for 120000 is correct? IIUC, this 5
matches the 'attgenerated' column, but the SQL for that was
constructed using a different condition:
if (server_version >= 180000)
  appendStringInfo(&cmd, ", a.attgenerated != ''");

It is this 120000 versus 180000 difference that makes me suspicious of
a potential mistake.

~~~

7.
+ /*
+ * If the column is generated and neither the generated column option
+ * is specified nor it appears in the column list, we will skip it.
+ */
+ if (remotegenlist[natt] && !has_pub_with_pubgencols &&
+ !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }

7b.
I am also suspicious about how this condition interacts with the other
condition (shown below) that came earlier:
/* If the column is not in the column list, skip it. */
if (included_cols != NULL && !bms_is_member(attnum, included_cols))

Something doesn't seem right. e.g. If we can only get here by passing
the earlier condition, then it means we already know the generated
condition was *not* a member of a column list.... in which case that
should affect this new condition and the new comment too.

======
src/backend/replication/pgoutput/pgoutput.c

pgoutput_column_list_init:

8.
  /*
- * If the publication is FOR ALL TABLES then it is treated the same as
- * if there are no column lists (even if other publications have a
- * list).
+ * Process potential column lists for the following cases: a. Any
+ * publication that is not FOR ALL TABLES. b. When the publication is
+ * FOR ALL TABLES and 'publish_generated_columns' is false. FOR ALL
+ * TABLES publication doesn't have user-defined column lists, so all
+ * columns will be replicated by default. However, if
+ * 'publish_generated_columns' is set to false, column lists must
+ * still be created to exclude any generated columns from being
+ * published.
  */

nit - please reformat this comment so the bullets are readable

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  182. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-09T06:22:15Z

    On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
>
> On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham, here are my review comments for v36-0001.
> >
> > ======
> > 1. General  - merge patches
> >
> > It is long past due when patches 0001 and 0002 should've been merged.
> > AFAIK the split was only because historically these parts had
> > different authors. But, keeping them separated is not helpful anymore.
> >
> > ======
> > src/backend/catalog/pg_publication.c
> >
> > 2.
> >  Bitmapset *
> > -pub_collist_validate(Relation targetrel, List *columns)
> > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> >
> > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > so it should also be removed.
> >
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 3.
> >   /*
> > - * If the publication is FOR ALL TABLES then it is treated the same as
> > - * if there are no column lists (even if other publications have a
> > - * list).
> > + * To handle cases where the publish_generated_columns option isn't
> > + * specified for all tables in a publication, we must create a column
> > + * list that excludes generated columns. So, the publisher will not
> > + * replicate the generated columns.
> >   */
> > - if (!pub->alltables)
> > + if (!(pub->alltables && pub->pubgencols))
> >
> > I still found that comment hard to understand. Does this mean to say
> > something like:
> >
> > ------
> > Process potential column lists for the following cases:
> >
> > a. Any publication that is not FOR ALL TABLES.
> >
> > b. When the publication is FOR ALL TABLES and
> > 'publish_generated_columns' is false.
> > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > so all columns will be replicated by default. However, if
> > 'publish_generated_columns' is set to false, column lists must still
> > be created to exclude any generated columns from being published
> > ------
> >
> > ======
> > src/test/regress/sql/publication.sql
> >
> > 4.
> > +SET client_min_messages = 'WARNING';
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> >
> > AFAIK you don't need to keep changing 'client_min_messages',
> > particularly now that you've removed the WARNING message that was
> > previously emitted.
> >
> > ~
> >
> > 5.
> > nit - minor comment changes.
> >
> > ======
> > Please refer to the attachment which implements any nits from above.
> >
>
> I have fixed all the given comments. Also, I have created a new 0003
> patch for the TAP-Tests related to the '011_generated.pl' file. I am
> planning to merge 0001 and 0003 patches once they will get fixed.
> The attached patches contain the required changes.

Few comments:
1) I felt this change need not be part of this patch, if required it
can be proposed as a separate patch:
+       if (server_version >= 150000)
        {
                WalRcvExecResult *pubres;
                TupleTableSlot *tslot;
                Oid                     attrsRow[] = {INT2VECTOROID};
-               StringInfoData pub_names;
-
-               initStringInfo(&pub_names);
-               foreach(lc, MySubscription->publications)
-               {
-                       if (foreach_current_index(lc) > 0)
-                               appendStringInfoString(&pub_names, ", ");
-                       appendStringInfoString(&pub_names,
quote_literal_cstr(strVal(lfirst(lc))));
-               }
+               StringInfo      pub_names = makeStringInfo();

2) These two statements can be combined in to single appendStringInfo:
+       appendStringInfo(&cmd,
                                         "  FROM pg_catalog.pg_attribute a"
                                         "  LEFT JOIN pg_catalog.pg_index i"
                                         "       ON (i.indexrelid =
pg_get_replica_identity_index(%u))"
                                         " WHERE a.attnum > 0::pg_catalog.int2"
-                                        "   AND NOT a.attisdropped %s"
+                                        "   AND NOT a.attisdropped",
lrel->remoteid);
+
+       appendStringInfo(&cmd,
                                         "   AND a.attrelid = %u"
                                         " ORDER BY a.attnum",
-                                        lrel->remoteid,
-
(walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
-                                         "AND a.attgenerated = ''" : ""),
                                         lrel->remoteid);

3) In which scenario this will be hit:
+       /*
+        * Construct column list for COPY, excluding columns that are
subscription
+        * table generated columns.
+        */
+       for (int i = 0; i < rel->remoterel.natts; i++)
+       {
+               if (!localgenlist[i])
+                       attnamelist = lappend(attnamelist,
+
makeString(rel->remoterel.attnames[i]));
+       }

As in case of publisher having non generated columns:
CREATE TABLE t1(c1 int, c2 int)
and subscriber having generated columns:
CREATE TABLE t1(c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED)

We throw an error much earlier at
logicalrep_rel_open->logicalrep_report_missing_attrs saying:
ERROR: logical replication target relation "public.t1" is missing
replicated column: "c2"

4) To simplify the code and reduce complexity, we can refactor the
error checks to be included within the fetch_remote_table_info
function. This way, the remotegenlist will not need to be prepared and
passed to make_copy_attnamelist:
+       /*
+        * This loop checks for generated columns of the subscription table.
+        */
+       for (int i = 0; i < desc->natts; i++)
        {
-               attnamelist = lappend(attnamelist,
-
makeString(rel->remoterel.attnames[i]));
+               int                     remote_attnum;
+               Form_pg_attribute attr = TupleDescAttr(desc, i);
+
+               if (!attr->attgenerated)
+                       continue;
+
+               remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
+
                            NameStr(attr->attname));
+
+               if (remote_attnum >= 0)
+               {
+                       /*
+                        * Check if the subscription table generated
column has same name
+                        * as a non-generated column in the
corresponding publication
+                        * table.
+                        */
+                       if (!remotegenlist[remote_attnum])
+                               ereport(ERROR,
+
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+                                                errmsg("logical
replication target relation \"%s.%s\" has a generated column \"%s\" "
+                                                               "but
corresponding column on source relation is not a generated column",
+
rel->remoterel.nspname, rel->remoterel.relname,
NameStr(attr->attname))));
+
+                       /*
+                        * 'localgenlist' records that this is a
generated column in the
+                        * subscription table. Later, we use this
information to skip
+                        * adding this column to the column list for COPY.
+                        */
+                       localgenlist[remote_attnum] = true;
+               }
        }

Regards,
Vignesh

  183. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-09T09:23:57Z

    On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
>
> On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham, here are my review comments for v36-0001.
> >
> > ======
> > 1. General  - merge patches
> >
> > It is long past due when patches 0001 and 0002 should've been merged.
> > AFAIK the split was only because historically these parts had
> > different authors. But, keeping them separated is not helpful anymore.
> >
> > ======
> > src/backend/catalog/pg_publication.c
> >
> > 2.
> >  Bitmapset *
> > -pub_collist_validate(Relation targetrel, List *columns)
> > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> >
> > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > so it should also be removed.
> >
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 3.
> >   /*
> > - * If the publication is FOR ALL TABLES then it is treated the same as
> > - * if there are no column lists (even if other publications have a
> > - * list).
> > + * To handle cases where the publish_generated_columns option isn't
> > + * specified for all tables in a publication, we must create a column
> > + * list that excludes generated columns. So, the publisher will not
> > + * replicate the generated columns.
> >   */
> > - if (!pub->alltables)
> > + if (!(pub->alltables && pub->pubgencols))
> >
> > I still found that comment hard to understand. Does this mean to say
> > something like:
> >
> > ------
> > Process potential column lists for the following cases:
> >
> > a. Any publication that is not FOR ALL TABLES.
> >
> > b. When the publication is FOR ALL TABLES and
> > 'publish_generated_columns' is false.
> > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > so all columns will be replicated by default. However, if
> > 'publish_generated_columns' is set to false, column lists must still
> > be created to exclude any generated columns from being published
> > ------
> >
> > ======
> > src/test/regress/sql/publication.sql
> >
> > 4.
> > +SET client_min_messages = 'WARNING';
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> >
> > AFAIK you don't need to keep changing 'client_min_messages',
> > particularly now that you've removed the WARNING message that was
> > previously emitted.
> >
> > ~
> >
> > 5.
> > nit - minor comment changes.
> >
> > ======
> > Please refer to the attachment which implements any nits from above.
> >
>
> I have fixed all the given comments. Also, I have created a new 0003
> patch for the TAP-Tests related to the '011_generated.pl' file. I am
> planning to merge 0001 and 0003 patches once they will get fixed.
> The attached patches contain the required changes.

Few comments for v37-0002 patch:
1.a) We could include the output of each command execution like
"CREATE TABLE", "INSERT 0 3" and "CREATE PUBLICATION" as we have done
in other places like in [1]:
+test_pub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS
AS (a + 1) STORED);
+test_pub=# INSERT INTO tab_gen_to_gen VALUES (1),(2),(3);
+test_pub=# CREATE PUBLICATION pub1 FOR TABLE tab_gen_to_gen;

1.b) Similarly here too:
+test_sub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS
AS (a * 100) STORED);
+test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
PUBLICATION pub1;
+test_sub=# SELECT * from tab_gen_to_gen;

1.c) Similarly here too:
+<programlisting>
+test_pub=# CREATE TABLE t1 (a int PRIMARY KEY, b int,
+test_pub-#                  c int GENERATED ALWAYS AS (a + 1) STORED,
+test_pub-#                  d int GENERATED ALWAYS AS (b + 1) STORED);
+
+test_pub=# CREATE TABLE t2 (a int PRIMARY KEY, b int,
+test_pub-#                  c int GENERATED ALWAYS AS (a + 1) STORED,
+test_pub-#                  d int GENERATED ALWAYS AS (b + 1) STORED);
+</programlisting>
+<programlisting>
+test_sub=# CREATE TABLE t1 (a int PRIMARY KEY, b int,
+test_sub-#                  c int,
+test_sub-#                  d int GENERATED ALWAYS AS (b * 100) STORED);
+
+test_sub=# CREATE TABLE t2 (a int PRIMARY KEY, b int,
+test_sub-#                  c int,
+test_sub-#                  d int);

1.d) Similarly here too:
+<programlisting>
+test_pub=# CREATE PUBLICATION pub1 FOR TABLE t1, t2(a,c)
+test_pub-#     WITH (publish_generated_columns=false);
+</programlisting>
+<programlisting>
+test_sub=# CREATE SUBSCRIPTION sub1
+test_sub-#     CONNECTION 'dbname=test_pub'
+test_sub-#     PUBLICATION pub1;
+</programlisting>

1.e) Similarly here too:
+   Insert some data to the publisher tables:
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES (1,2);
+test_pub=# INSERT INTO t2 VALUES (1,2);

2) All of the document changes of ddl.sgml, protocol.sgml,
create_publication.sgml can also be moved from 0001 patch to 0002
patch:
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 8ab0ddb112..7b9c349343 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -514,8 +514,10 @@ CREATE TABLE people (
     </listitem>
     <listitem>
      <para>
-      Generated columns are skipped for logical replication and cannot be
-      specified in a <command>CREATE PUBLICATION</command> column list.
+      Generated columns may be skipped during logical replication
according to the
+      <command>CREATE PUBLICATION</command> parameter
+      <link linkend="sql-createpublication-params-with-publish-generated-columns">
+      <literal>publish_generated_columns</literal></link>.

3) I felt "(except generated columns)" should be removed from here too:
  <variablelist>
   <varlistentry id="protocol-logicalrep-message-formats-TupleData">
    <term>TupleData</term>
    <listitem>
     <variablelist>
      <varlistentry>
       <term>Int16</term>
       <listitem>
        <para>
         Number of columns.
        </para>
       </listitem>
      </varlistentry>
     </variablelist>

     <para>
      Next, one of the following submessages appears for each column
(except generated columns):

[1] - https://www.postgresql.org/docs/devel/logical-replication-subscription.html#LOGICAL-REPLICATION-SUBSCRIPTION-EXAMPLES

Regards,
Vignesh

  184. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-10-09T16:48:41Z

    On Mon, Oct 7, 2024 at 11:07 PM Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Shubham, here are my review comments for v36-0001.
> >
> > ======
> > 1. General  - merge patches
> >
> > It is long past due when patches 0001 and 0002 should've been merged.
> > AFAIK the split was only because historically these parts had
> > different authors. But, keeping them separated is not helpful anymore.
> >
> > ======
> > src/backend/catalog/pg_publication.c
> >
> > 2.
> >  Bitmapset *
> > -pub_collist_validate(Relation targetrel, List *columns)
> > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> >
> > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > so it should also be removed.
> >
> > ======
> > src/backend/replication/pgoutput/pgoutput.c
> >
> > 3.
> >   /*
> > - * If the publication is FOR ALL TABLES then it is treated the same as
> > - * if there are no column lists (even if other publications have a
> > - * list).
> > + * To handle cases where the publish_generated_columns option isn't
> > + * specified for all tables in a publication, we must create a column
> > + * list that excludes generated columns. So, the publisher will not
> > + * replicate the generated columns.
> >   */
> > - if (!pub->alltables)
> > + if (!(pub->alltables && pub->pubgencols))
> >
> > I still found that comment hard to understand. Does this mean to say
> > something like:
> >
> > ------
> > Process potential column lists for the following cases:
> >
> > a. Any publication that is not FOR ALL TABLES.
> >
> > b. When the publication is FOR ALL TABLES and
> > 'publish_generated_columns' is false.
> > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > so all columns will be replicated by default. However, if
> > 'publish_generated_columns' is set to false, column lists must still
> > be created to exclude any generated columns from being published
> > ------
> >
> > ======
> > src/test/regress/sql/publication.sql
> >
> > 4.
> > +SET client_min_messages = 'WARNING';
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> >
> > AFAIK you don't need to keep changing 'client_min_messages',
> > particularly now that you've removed the WARNING message that was
> > previously emitted.
> >
> > ~
> >
> > 5.
> > nit - minor comment changes.
> >
> > ======
> > Please refer to the attachment which implements any nits from above.
> >
>
> I have fixed all the given comments. Also, I have created a new 0003
> patch for the TAP-Tests related to the '011_generated.pl' file. I am
> planning to merge 0001 and 0003 patches once they will get fixed.
> The attached patches contain the required changes.
>

Regarding the 0001 patch, it seems to me that UPDATE and DELETE are
allowed on the table even if its replica identity is set to generated
columns that are not published. For example, consider the following
scenario:

create table t (a int not null, b int generated always as (a + 1)
stored not null);
create unique index t_idx on t (b);
alter table t replica identity using index t_idx;
create publication pub for table t with (publish_generated_columns = false);
insert into t values (1);
update t set a = 100 where a = 1;

The publication pub doesn't include the generated column 'b' which is
the replica identity of the table 't'. Therefore, the update message
generated by the last UPDATE would have NULL for the column 'b'. I
think we should not allow UPDATE and DELETE on such a table.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  185. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-10T00:45:00Z

    Hi - Here is patch set v38 to address documentation review comments.

Changes from v37:
v38-0001.
- The code is unchanged.
- The SGML changes previously in 0001 are now in 0002.
- In passing, the commit message was improved to address review
comments from Peter [1].
v38-0002.
- Now includes all SGML changes previously in patch 0001.
- Addresses all docs review comments from Vignesh [2].
v38-0003.
- Unchanged.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPv8Do3b7QhzHg7dRWhO317ZFZKY_mYQaFBOWVQ-P1805A%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CALDaNm3VcrDdF2A2x5QTPhajUxy150z3wsV4W4OGOwTFE4--Wg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

//////////

Details are below:

On Wed, Oct 9, 2024 at 8:24 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> >
> > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Shubham, here are my review comments for v36-0001.
> > >
> > > ======
> > > 1. General  - merge patches
> > >
> > > It is long past due when patches 0001 and 0002 should've been merged.
> > > AFAIK the split was only because historically these parts had
> > > different authors. But, keeping them separated is not helpful anymore.
> > >
> > > ======
> > > src/backend/catalog/pg_publication.c
> > >
> > > 2.
> > >  Bitmapset *
> > > -pub_collist_validate(Relation targetrel, List *columns)
> > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > >
> > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > so it should also be removed.
> > >
> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 3.
> > >   /*
> > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > - * if there are no column lists (even if other publications have a
> > > - * list).
> > > + * To handle cases where the publish_generated_columns option isn't
> > > + * specified for all tables in a publication, we must create a column
> > > + * list that excludes generated columns. So, the publisher will not
> > > + * replicate the generated columns.
> > >   */
> > > - if (!pub->alltables)
> > > + if (!(pub->alltables && pub->pubgencols))
> > >
> > > I still found that comment hard to understand. Does this mean to say
> > > something like:
> > >
> > > ------
> > > Process potential column lists for the following cases:
> > >
> > > a. Any publication that is not FOR ALL TABLES.
> > >
> > > b. When the publication is FOR ALL TABLES and
> > > 'publish_generated_columns' is false.
> > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > so all columns will be replicated by default. However, if
> > > 'publish_generated_columns' is set to false, column lists must still
> > > be created to exclude any generated columns from being published
> > > ------
> > >
> > > ======
> > > src/test/regress/sql/publication.sql
> > >
> > > 4.
> > > +SET client_min_messages = 'WARNING';
> > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > >
> > > AFAIK you don't need to keep changing 'client_min_messages',
> > > particularly now that you've removed the WARNING message that was
> > > previously emitted.
> > >
> > > ~
> > >
> > > 5.
> > > nit - minor comment changes.
> > >
> > > ======
> > > Please refer to the attachment which implements any nits from above.
> > >
> >
> > I have fixed all the given comments. Also, I have created a new 0003
> > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > planning to merge 0001 and 0003 patches once they will get fixed.
> > The attached patches contain the required changes.
>
> Few comments for v37-0002 patch:
> 1.a) We could include the output of each command execution like
> "CREATE TABLE", "INSERT 0 3" and "CREATE PUBLICATION" as we have done
> in other places like in [1]:
> +test_pub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS
> AS (a + 1) STORED);
> +test_pub=# INSERT INTO tab_gen_to_gen VALUES (1),(2),(3);
> +test_pub=# CREATE PUBLICATION pub1 FOR TABLE tab_gen_to_gen;
>
> 1.b) Similarly here too:
> +test_sub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS
> AS (a * 100) STORED);
> +test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
> PUBLICATION pub1;
> +test_sub=# SELECT * from tab_gen_to_gen;
>
> 1.c) Similarly here too:
> +<programlisting>
> +test_pub=# CREATE TABLE t1 (a int PRIMARY KEY, b int,
> +test_pub-#                  c int GENERATED ALWAYS AS (a + 1) STORED,
> +test_pub-#                  d int GENERATED ALWAYS AS (b + 1) STORED);
> +
> +test_pub=# CREATE TABLE t2 (a int PRIMARY KEY, b int,
> +test_pub-#                  c int GENERATED ALWAYS AS (a + 1) STORED,
> +test_pub-#                  d int GENERATED ALWAYS AS (b + 1) STORED);
> +</programlisting>
> +<programlisting>
> +test_sub=# CREATE TABLE t1 (a int PRIMARY KEY, b int,
> +test_sub-#                  c int,
> +test_sub-#                  d int GENERATED ALWAYS AS (b * 100) STORED);
> +
> +test_sub=# CREATE TABLE t2 (a int PRIMARY KEY, b int,
> +test_sub-#                  c int,
> +test_sub-#                  d int);
>
> 1.d) Similarly here too:
> +<programlisting>
> +test_pub=# CREATE PUBLICATION pub1 FOR TABLE t1, t2(a,c)
> +test_pub-#     WITH (publish_generated_columns=false);
> +</programlisting>
> +<programlisting>
> +test_sub=# CREATE SUBSCRIPTION sub1
> +test_sub-#     CONNECTION 'dbname=test_pub'
> +test_sub-#     PUBLICATION pub1;
> +</programlisting>
>
> 1.e) Similarly here too:
> +   Insert some data to the publisher tables:
> +<programlisting>
> +test_pub=# INSERT INTO t1 VALUES (1,2);
> +test_pub=# INSERT INTO t2 VALUES (1,2);
>

OK. All the above are fixed in v38-0002. I had thought this change was
mostly a waste of space, but I hadn't noticed there was a precedent.
IMO consistency is better, so I have made all the changes as you
suggested.

> 2) All of the document changes of ddl.sgml, protocol.sgml,
> create_publication.sgml can also be moved from 0001 patch to 0002
> patch:
> diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
> index 8ab0ddb112..7b9c349343 100644
> --- a/doc/src/sgml/ddl.sgml
> +++ b/doc/src/sgml/ddl.sgml
> @@ -514,8 +514,10 @@ CREATE TABLE people (
>      </listitem>
>      <listitem>
>       <para>
> -      Generated columns are skipped for logical replication and cannot be
> -      specified in a <command>CREATE PUBLICATION</command> column list.
> +      Generated columns may be skipped during logical replication
> according to the
> +      <command>CREATE PUBLICATION</command> parameter
> +      <link linkend="sql-createpublication-params-with-publish-generated-columns">
> +      <literal>publish_generated_columns</literal></link>.
>

OK. SGML updates are moved to patch 0002 in v38.

> 3) I felt "(except generated columns)" should be removed from here too:
>   <variablelist>
>    <varlistentry id="protocol-logicalrep-message-formats-TupleData">
>     <term>TupleData</term>
>     <listitem>
>      <variablelist>
>       <varlistentry>
>        <term>Int16</term>
>        <listitem>
>         <para>
>          Number of columns.
>         </para>
>        </listitem>
>       </varlistentry>
>      </variablelist>
>
>      <para>
>       Next, one of the following submessages appears for each column
> (except generated columns):
>

OK. This is fixed in v38. This change is in "53.9. Logical Replication
Message Formats".

  186. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-10T05:23:22Z

    Here are some comments for TAP test patch v37-0003.

I’m not in favour of the removal of such a large number of
'combination' and other 'misc' tests. In the commit message, please
delete me as a "co-author" of this patch.

======

1.
Any description or comment that still mentions "all combinations" is
no longer valid:

(e.g. in the comment message)
Add tests for all combinations of generated column replication.

(e.g. in the test file)
# The following test cases exercise logical replication for all combinations
# where there is a generated column on one or both sides of pub/sub:

and

# Furthermore, all combinations are tested using:

======
2.
+# --------------------------------------------------
+# Testcase: generated -> normal
+# Publisher table has generated column 'b'.
+# Subscriber table has normal column 'b'.
+# --------------------------------------------------
+

Now that COPY for generated columns is already implemented in patch
0001, shouldn't this test be using 'copy_data' enabled, so it can test
replication both for initial tablesync as well as normal replication?

That was the whole point of having the "# XXX copy_data=false for now.
This will be changed later." reminder comment in this file.

======

3.
Previously there were some misc tests to ensure that a generated
column which was then altered using DROP EXPRESSION would work as
expected. The test scenario was commented like:

+# =============================================================================
+# Misc test.
+#
+# A "normal -> generated" replication fails, reporting an error that the
+# subscriber side column is missing.
+#
+# In this test case we use DROP EXPRESSION to change the subscriber generated
+# column into a normal column, then verify replication works ok.
+# =============================================================================

Now in patch v37 this test no longer exists. Why?

======
4.
+# =============================================================================
+# The following test cases demonstrate behavior of generated column replication
+# when publish_generated_colums=false/true:
+#
+# Test: column list includes gencols, when publish_generated_columns=false
+# Test: column list does not include gencols, when
publish_generated_columns=false
+#
+# Test: column list includes gencols, when publish_generated_columns=true
+# Test: column list does not include gencols, when
publish_generated_columns=true
+# Test: no column list, when publish_generated_columns=true
+# =============================================================================

These tests are currently only testing the initial tablesync
replication. Since the COPY logic is different from the normal
replication logic, I think it would be better to test some normal
replication records as well, to make sure both parts work
consistently. This comment applies to all of the following test cases.

~~~

5.
+# Create table and publications.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE nogen_to_gen3 (a int, b int, gen1 int GENERATED ALWAYS
AS (a * 2) STORED, gen2 int GENERATED ALWAYS AS (a * 2) STORED);
+ CREATE TABLE nogen_to_gen4 (c int, d int, gen1 int GENERATED ALWAYS
AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (c * 2) STORED);
+ INSERT INTO nogen_to_gen3 VALUES (1, 1);
+ INSERT INTO nogen_to_gen4 VALUES (1, 1);
+ CREATE PUBLICATION pub1 FOR table nogen_to_gen3, nogen_to_gen4(gen1)
WITH (publish_generated_columns=true);
+));
+

5a.
The code should do only what the comments say it does. So, the INSERTS
should be done separately after the CREATE PUBLICATION, but before the
CREATE SUBSCRIPTION. A similar change should be made for all of these
test cases.

# Insert some initial data
INSERT INTO nogen_to_gen3 VALUES (1, 1);
INSERT INTO nogen_to_gen4 VALUES (1, 1);

~

5b.
The tables are badly named. Why are they 'nogen_to_gen', when the
publisher side has generated cols and the subscriber side does not?
This problem seems repeated in multiple subsequent test cases.

~

6.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM gen_to_nogen ORDER BY a");
+is($result, qq(1|1||2),
+ 'gen_to_nogen initial sync, when publish_generated_columns=false');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM gen_to_nogen2 ORDER BY c");
+is($result, qq(1|1||),
+ 'gen_to_nogen2 initial sync, when publish_generated_columns=false');

IMO all the "result" queries like these ones ought to have to have a
comment which explains the reason for the expected results. This
review comment applies to multiple places. Please add comments to all
of them.

~~~

7.
+# --------------------------------------------------
+# Testcase: Publisher replicates the column list data excluding generated
+# columns even though publish_generated_columns option is false.
+# --------------------------------------------------
+

7a.
This is the 2nd test case, but AFAICT it would be far easier to test
this scenario just by making another table (with an appropriate column
list) for the 1st test case.

~

7b.
BTW, I don't understand this test at all. I thought according to the
comment that it intended to use a publication column list with only
normal columns in it. But that is not what the publication looks like
here:
+ CREATE PUBLICATION pub1 FOR table nogen_to_gen, nogen_to_gen2(gen1)
WITH (publish_generated_columns=false);

Indeed, the way it is currently written I didn't see what this test is
doing that is any different from the prior test (???)

~~~

8.
+# --------------------------------------------------
+# Testcase: Although publish_generated_columns is true, publisher publishes
+# only the data of the columns specified in column list, skipping other
+# generated/non-generated columns.
+# --------------------------------------------------

versus

+# --------------------------------------------------
+# Testcase: Publisher publishes only the data of the columns specified in
+# column list skipping other generated/non-generated columns.
+# --------------------------------------------------

Again, I did not understand how these test cases differ from each
other. Surely, those can be combined easily enough just by adding
another table with a different kind of column list.

~~~

9.
+# --------------------------------------------------
+# Testcase: Publisher replicates all columns if publish_generated_columns is
+# enabled and there is no column list
+# --------------------------------------------------
+

Here is yet another test case that AFAICT can just be combined with
other test cases that were using publish_generated_columns=true. It
seems all you need is one extra table with no column list. You don't
need all the extra create/drop pub/sub overheads to test this.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  187. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-16T17:54:56Z

    On Wed, Oct 9, 2024 at 9:08 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> >
> > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Shubham, here are my review comments for v36-0001.
> > >
> > > ======
> > > 1. General  - merge patches
> > >
> > > It is long past due when patches 0001 and 0002 should've been merged.
> > > AFAIK the split was only because historically these parts had
> > > different authors. But, keeping them separated is not helpful anymore.
> > >
> > > ======
> > > src/backend/catalog/pg_publication.c
> > >
> > > 2.
> > >  Bitmapset *
> > > -pub_collist_validate(Relation targetrel, List *columns)
> > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > >
> > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > so it should also be removed.
> > >
> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 3.
> > >   /*
> > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > - * if there are no column lists (even if other publications have a
> > > - * list).
> > > + * To handle cases where the publish_generated_columns option isn't
> > > + * specified for all tables in a publication, we must create a column
> > > + * list that excludes generated columns. So, the publisher will not
> > > + * replicate the generated columns.
> > >   */
> > > - if (!pub->alltables)
> > > + if (!(pub->alltables && pub->pubgencols))
> > >
> > > I still found that comment hard to understand. Does this mean to say
> > > something like:
> > >
> > > ------
> > > Process potential column lists for the following cases:
> > >
> > > a. Any publication that is not FOR ALL TABLES.
> > >
> > > b. When the publication is FOR ALL TABLES and
> > > 'publish_generated_columns' is false.
> > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > so all columns will be replicated by default. However, if
> > > 'publish_generated_columns' is set to false, column lists must still
> > > be created to exclude any generated columns from being published
> > > ------
> > >
> > > ======
> > > src/test/regress/sql/publication.sql
> > >
> > > 4.
> > > +SET client_min_messages = 'WARNING';
> > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > >
> > > AFAIK you don't need to keep changing 'client_min_messages',
> > > particularly now that you've removed the WARNING message that was
> > > previously emitted.
> > >
> > > ~
> > >
> > > 5.
> > > nit - minor comment changes.
> > >
> > > ======
> > > Please refer to the attachment which implements any nits from above.
> > >
> >
> > I have fixed all the given comments. Also, I have created a new 0003
> > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > planning to merge 0001 and 0003 patches once they will get fixed.
> > The attached patches contain the required changes.
>
> Few comments:
> 1) Since we are no longer throwing an error for generated columns, the
> function header comments also need to be updated accordingly " Checks
> for and raises an ERROR for any; unknown columns, system columns,
> duplicate columns or generated columns."
> -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> -                       ereport(ERROR,
> -
> errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> -                                       errmsg("cannot use generated
> column \"%s\" in publication column list",
> -                                                  colname));
> -
>
> 2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
> ALTER PUBLICATION ... SET (
> postgres=# alter publication pub2 set (PUBLISH
> PUBLISH                     PUBLISH_VIA_PARTITION_ROOT
>
> 3) I was able to compile without this include, may be this is not required:
> --- a/src/backend/replication/logical/tablesync.c
> +++ b/src/backend/replication/logical/tablesync.c
> @@ -118,6 +118,7 @@
>  #include "utils/builtins.h"
>  #include "utils/lsyscache.h"
>  #include "utils/memutils.h"
> +#include "utils/rel.h"
>
> 4) You can include "\dRp+ pubname" after each of the create/alter
> publication to verify the columns that will be published:
> +-- Test the 'publish_generated_columns' parameter enabled or disabled for
> +-- different scenarios with/without generated columns in column lists.
> +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> +
> +-- Generated columns in column list, when 'publish_generated_columns'=false
> +CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
> (publish_generated_columns=false);
>
> +-- Generated columns in column list, when 'publish_generated_columns'=true
> +CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
> (publish_generated_columns=true);
> +
> +-- Generated columns in column list, then set
> 'publication_generate_columns'=false
> +ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
> +
> +-- Remove generate columns from column list, when
> 'publish_generated_columns'=false
> +ALTER PUBLICATION pub2 SET TABLE gencols(a);
> +
> +-- Add generated columns in column list, when 'publish_generated_columns'=false
> +ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);
>

I have fixed all the given comments. The attached patches contain the
required changes.

Thanks and Regards,
Shubham Khanna.

  188. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-16T17:59:25Z

    On Wed, Oct 9, 2024 at 11:00 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> >
> > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Shubham, here are my review comments for v36-0001.
> > >
> > > ======
> > > 1. General  - merge patches
> > >
> > > It is long past due when patches 0001 and 0002 should've been merged.
> > > AFAIK the split was only because historically these parts had
> > > different authors. But, keeping them separated is not helpful anymore.
> > >
> > > ======
> > > src/backend/catalog/pg_publication.c
> > >
> > > 2.
> > >  Bitmapset *
> > > -pub_collist_validate(Relation targetrel, List *columns)
> > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > >
> > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > so it should also be removed.
> > >
> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 3.
> > >   /*
> > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > - * if there are no column lists (even if other publications have a
> > > - * list).
> > > + * To handle cases where the publish_generated_columns option isn't
> > > + * specified for all tables in a publication, we must create a column
> > > + * list that excludes generated columns. So, the publisher will not
> > > + * replicate the generated columns.
> > >   */
> > > - if (!pub->alltables)
> > > + if (!(pub->alltables && pub->pubgencols))
> > >
> > > I still found that comment hard to understand. Does this mean to say
> > > something like:
> > >
> > > ------
> > > Process potential column lists for the following cases:
> > >
> > > a. Any publication that is not FOR ALL TABLES.
> > >
> > > b. When the publication is FOR ALL TABLES and
> > > 'publish_generated_columns' is false.
> > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > so all columns will be replicated by default. However, if
> > > 'publish_generated_columns' is set to false, column lists must still
> > > be created to exclude any generated columns from being published
> > > ------
> > >
> > > ======
> > > src/test/regress/sql/publication.sql
> > >
> > > 4.
> > > +SET client_min_messages = 'WARNING';
> > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > >
> > > AFAIK you don't need to keep changing 'client_min_messages',
> > > particularly now that you've removed the WARNING message that was
> > > previously emitted.
> > >
> > > ~
> > >
> > > 5.
> > > nit - minor comment changes.
> > >
> > > ======
> > > Please refer to the attachment which implements any nits from above.
> > >
> >
> > I have fixed all the given comments. Also, I have created a new 0003
> > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > planning to merge 0001 and 0003 patches once they will get fixed.
> > The attached patches contain the required changes.
>
> There is inconsistency in replication when a generated column is
> specified in the column list. The generated column data is not
> replicated during initial sync whereas it is getting replicated during
> incremental sync:
> -- publisher
> CREATE TABLE t1(c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED)
> INSERT INTO t1 VALUES (1);
> CREATE PUBLICATION pub1 for table t1(c1, c2);
>
> --subscriber
> CREATE TABLE t1(c1 int, c2 int)
> CREATE SUBSCRIPTION sub1 connection 'dbname=postgres host=localhost
> port=5432' PUBLICATION pub1;
>
> -- Generate column data is not synced during initial sync
> postgres=# select * from t1;
>  c1 | c2
> ----+----
>   1 |
> (1 row)
>
> -- publisher
> INSERT INTO t1 VALUES (2);
>
> -- Whereas generated column data is synced during incremental sync
> postgres=# select * from t1;
>  c1 | c2
> ----+----
>   1 |
>   2 |  4
> (2 rows)
>

There was an issue for this scenario:
CREATE TABLE t1(c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED)
create publication pub1 for table t1(c1, c2)

In this case included_cols was getting set to NULL.
Changed it to get included_cols as it is instead of replacing with
NULL and changed the condition to:
    if (server_version >= 180000)
    {
      remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
      /*
       * If the column is generated and neither the generated column
       * option is specified nor it appears in the column list, we will
       * skip it.
       */
      if (remotegenlist[natt] && !has_pub_with_pubgencols && !included_cols)
      {
        ExecClearTuple(slot);
        continue;
      }
    }

I will further think if there is a better solution for this.
Please refer to the updated v39 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjLjb%2B98i5ZQUphivxdOZ3hSGLfq2SiWQetUvk8zGyAQwQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  189. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-16T18:01:57Z

    On Wed, Oct 9, 2024 at 11:13 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for patch v37-0001.
>
> ======
> Commit message
>
> 1.
> Example usage of subscription option:
> CREATE PUBLICATION FOR TABLE tab_gencol WITH (publish_generated_columns
> = true);
>
> ~
>
> This is wrong -- it's not a "subscription option". Better to just say
> "Example usage:"
>
> ~~~
>
> 2.
> When 'copy_data' is true, during the initial sync, the data is replicated from
> the publisher to the subscriber using the COPY command. The normal COPY
> command does not copy generated columns, so when 'publish_generated_columns'
> is true...
>
> ~
>
> By only mentioning the "when ... is true" case this description does
> not cover the scenario when 'publish_generated_columns' is false when
> the publication column list has a generated column.
>
> ~~~
>
> 3.
> typo - /replication of generated column/replication of generated columns/
> typo - /filed/filled/
> typo - 'pg_publicataion' catalog
>
> ======
> src/backend/replication/logical/tablesync.c
>
> make_copy_attnamelist:
> 4.
> nit - missing word in a comment
>
> ~~~
>
> fetch_remote_table_info:
> 5.
> + appendStringInfo(&cmd,
>   "  FROM pg_catalog.pg_attribute a"
>   "  LEFT JOIN pg_catalog.pg_index i"
>   "       ON (i.indexrelid = pg_get_replica_identity_index(%u))"
>   " WHERE a.attnum > 0::pg_catalog.int2"
> - "   AND NOT a.attisdropped %s"
> + "   AND NOT a.attisdropped", lrel->remoteid);
> +
> + appendStringInfo(&cmd,
>   "   AND a.attrelid = %u"
>   " ORDER BY a.attnum",
> - lrel->remoteid,
> - (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
> -   "AND a.attgenerated = ''" : ""),
>   lrel->remoteid);
>
> Version v37-0001 has removed a condition previously between these two
> appendStringInfo's. But, that now means there is no reason to keep
> these statements separated. These should be combined now to use one
> appendStringInfo.
>
> ~
>
> 6.
> + if (server_version >= 120000)
> + remotegenlist[natt] = DatumGetBool(slot_getattr(slot, 5, &isnull));
> +
>
> Are you sure the version check for 120000 is correct? IIUC, this 5
> matches the 'attgenerated' column, but the SQL for that was
> constructed using a different condition:
> if (server_version >= 180000)
>   appendStringInfo(&cmd, ", a.attgenerated != ''");
>
> It is this 120000 versus 180000 difference that makes me suspicious of
> a potential mistake.
>
> ~~~
>
> 7.
> + /*
> + * If the column is generated and neither the generated column option
> + * is specified nor it appears in the column list, we will skip it.
> + */
> + if (remotegenlist[natt] && !has_pub_with_pubgencols &&
> + !bms_is_member(attnum, included_cols))
> + {
> + ExecClearTuple(slot);
> + continue;
> + }
>
> 7b.
> I am also suspicious about how this condition interacts with the other
> condition (shown below) that came earlier:
> /* If the column is not in the column list, skip it. */
> if (included_cols != NULL && !bms_is_member(attnum, included_cols))
>
> Something doesn't seem right. e.g. If we can only get here by passing
> the earlier condition, then it means we already know the generated
> condition was *not* a member of a column list.... in which case that
> should affect this new condition and the new comment too.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> pgoutput_column_list_init:
>
> 8.
>   /*
> - * If the publication is FOR ALL TABLES then it is treated the same as
> - * if there are no column lists (even if other publications have a
> - * list).
> + * Process potential column lists for the following cases: a. Any
> + * publication that is not FOR ALL TABLES. b. When the publication is
> + * FOR ALL TABLES and 'publish_generated_columns' is false. FOR ALL
> + * TABLES publication doesn't have user-defined column lists, so all
> + * columns will be replicated by default. However, if
> + * 'publish_generated_columns' is set to false, column lists must
> + * still be created to exclude any generated columns from being
> + * published.
>   */
>
> nit - please reformat this comment so the bullets are readable
>

I have fixed all the comments and posted the v39 patches for them.
Please refer to the updated v39 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjLjb%2B98i5ZQUphivxdOZ3hSGLfq2SiWQetUvk8zGyAQwQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  190. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-16T18:02:51Z

    On Wed, Oct 9, 2024 at 11:52 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> >
> > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Shubham, here are my review comments for v36-0001.
> > >
> > > ======
> > > 1. General  - merge patches
> > >
> > > It is long past due when patches 0001 and 0002 should've been merged.
> > > AFAIK the split was only because historically these parts had
> > > different authors. But, keeping them separated is not helpful anymore.
> > >
> > > ======
> > > src/backend/catalog/pg_publication.c
> > >
> > > 2.
> > >  Bitmapset *
> > > -pub_collist_validate(Relation targetrel, List *columns)
> > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > >
> > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > so it should also be removed.
> > >
> > > ======
> > > src/backend/replication/pgoutput/pgoutput.c
> > >
> > > 3.
> > >   /*
> > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > - * if there are no column lists (even if other publications have a
> > > - * list).
> > > + * To handle cases where the publish_generated_columns option isn't
> > > + * specified for all tables in a publication, we must create a column
> > > + * list that excludes generated columns. So, the publisher will not
> > > + * replicate the generated columns.
> > >   */
> > > - if (!pub->alltables)
> > > + if (!(pub->alltables && pub->pubgencols))
> > >
> > > I still found that comment hard to understand. Does this mean to say
> > > something like:
> > >
> > > ------
> > > Process potential column lists for the following cases:
> > >
> > > a. Any publication that is not FOR ALL TABLES.
> > >
> > > b. When the publication is FOR ALL TABLES and
> > > 'publish_generated_columns' is false.
> > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > so all columns will be replicated by default. However, if
> > > 'publish_generated_columns' is set to false, column lists must still
> > > be created to exclude any generated columns from being published
> > > ------
> > >
> > > ======
> > > src/test/regress/sql/publication.sql
> > >
> > > 4.
> > > +SET client_min_messages = 'WARNING';
> > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > >
> > > AFAIK you don't need to keep changing 'client_min_messages',
> > > particularly now that you've removed the WARNING message that was
> > > previously emitted.
> > >
> > > ~
> > >
> > > 5.
> > > nit - minor comment changes.
> > >
> > > ======
> > > Please refer to the attachment which implements any nits from above.
> > >
> >
> > I have fixed all the given comments. Also, I have created a new 0003
> > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > planning to merge 0001 and 0003 patches once they will get fixed.
> > The attached patches contain the required changes.
>
> Few comments:
> 1) I felt this change need not be part of this patch, if required it
> can be proposed as a separate patch:
> +       if (server_version >= 150000)
>         {
>                 WalRcvExecResult *pubres;
>                 TupleTableSlot *tslot;
>                 Oid                     attrsRow[] = {INT2VECTOROID};
> -               StringInfoData pub_names;
> -
> -               initStringInfo(&pub_names);
> -               foreach(lc, MySubscription->publications)
> -               {
> -                       if (foreach_current_index(lc) > 0)
> -                               appendStringInfoString(&pub_names, ", ");
> -                       appendStringInfoString(&pub_names,
> quote_literal_cstr(strVal(lfirst(lc))));
> -               }
> +               StringInfo      pub_names = makeStringInfo();
>
> 2) These two statements can be combined in to single appendStringInfo:
> +       appendStringInfo(&cmd,
>                                          "  FROM pg_catalog.pg_attribute a"
>                                          "  LEFT JOIN pg_catalog.pg_index i"
>                                          "       ON (i.indexrelid =
> pg_get_replica_identity_index(%u))"
>                                          " WHERE a.attnum > 0::pg_catalog.int2"
> -                                        "   AND NOT a.attisdropped %s"
> +                                        "   AND NOT a.attisdropped",
> lrel->remoteid);
> +
> +       appendStringInfo(&cmd,
>                                          "   AND a.attrelid = %u"
>                                          " ORDER BY a.attnum",
> -                                        lrel->remoteid,
> -
> (walrcv_server_version(LogRepWorkerWalRcvConn) >= 120000 ?
> -                                         "AND a.attgenerated = ''" : ""),
>                                          lrel->remoteid);
>
> 3) In which scenario this will be hit:
> +       /*
> +        * Construct column list for COPY, excluding columns that are
> subscription
> +        * table generated columns.
> +        */
> +       for (int i = 0; i < rel->remoterel.natts; i++)
> +       {
> +               if (!localgenlist[i])
> +                       attnamelist = lappend(attnamelist,
> +
> makeString(rel->remoterel.attnames[i]));
> +       }
>
> As in case of publisher having non generated columns:
> CREATE TABLE t1(c1 int, c2 int)
> and subscriber having generated columns:
> CREATE TABLE t1(c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED)
>
> We throw an error much earlier at
> logicalrep_rel_open->logicalrep_report_missing_attrs saying:
> ERROR: logical replication target relation "public.t1" is missing
> replicated column: "c2"
>
> 4) To simplify the code and reduce complexity, we can refactor the
> error checks to be included within the fetch_remote_table_info
> function. This way, the remotegenlist will not need to be prepared and
> passed to make_copy_attnamelist:
> +       /*
> +        * This loop checks for generated columns of the subscription table.
> +        */
> +       for (int i = 0; i < desc->natts; i++)
>         {
> -               attnamelist = lappend(attnamelist,
> -
> makeString(rel->remoterel.attnames[i]));
> +               int                     remote_attnum;
> +               Form_pg_attribute attr = TupleDescAttr(desc, i);
> +
> +               if (!attr->attgenerated)
> +                       continue;
> +
> +               remote_attnum = logicalrep_rel_att_by_name(&rel->remoterel,
> +
>                             NameStr(attr->attname));
> +
> +               if (remote_attnum >= 0)
> +               {
> +                       /*
> +                        * Check if the subscription table generated
> column has same name
> +                        * as a non-generated column in the
> corresponding publication
> +                        * table.
> +                        */
> +                       if (!remotegenlist[remote_attnum])
> +                               ereport(ERROR,
> +
> (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> +                                                errmsg("logical
> replication target relation \"%s.%s\" has a generated column \"%s\" "
> +                                                               "but
> corresponding column on source relation is not a generated column",
> +
> rel->remoterel.nspname, rel->remoterel.relname,
> NameStr(attr->attname))));
> +
> +                       /*
> +                        * 'localgenlist' records that this is a
> generated column in the
> +                        * subscription table. Later, we use this
> information to skip
> +                        * adding this column to the column list for COPY.
> +                        */
> +                       localgenlist[remote_attnum] = true;
> +               }
>         }
>

I have fixed all the comments and posted the v39 patches for them.
Please refer to the updated v39 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjLjb%2B98i5ZQUphivxdOZ3hSGLfq2SiWQetUvk8zGyAQwQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  191. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-16T18:03:35Z

    On Thu, Oct 10, 2024 at 10:53 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some comments for TAP test patch v37-0003.
>
> I’m not in favour of the removal of such a large number of
> 'combination' and other 'misc' tests. In the commit message, please
> delete me as a "co-author" of this patch.
>
> ======
>
> 1.
> Any description or comment that still mentions "all combinations" is
> no longer valid:
>
> (e.g. in the comment message)
> Add tests for all combinations of generated column replication.
>
> (e.g. in the test file)
> # The following test cases exercise logical replication for all combinations
> # where there is a generated column on one or both sides of pub/sub:
>
> and
>
> # Furthermore, all combinations are tested using:
>
> ======
> 2.
> +# --------------------------------------------------
> +# Testcase: generated -> normal
> +# Publisher table has generated column 'b'.
> +# Subscriber table has normal column 'b'.
> +# --------------------------------------------------
> +
>
> Now that COPY for generated columns is already implemented in patch
> 0001, shouldn't this test be using 'copy_data' enabled, so it can test
> replication both for initial tablesync as well as normal replication?
>
> That was the whole point of having the "# XXX copy_data=false for now.
> This will be changed later." reminder comment in this file.
>
> ======
>
> 3.
> Previously there were some misc tests to ensure that a generated
> column which was then altered using DROP EXPRESSION would work as
> expected. The test scenario was commented like:
>
> +# =============================================================================
> +# Misc test.
> +#
> +# A "normal -> generated" replication fails, reporting an error that the
> +# subscriber side column is missing.
> +#
> +# In this test case we use DROP EXPRESSION to change the subscriber generated
> +# column into a normal column, then verify replication works ok.
> +# =============================================================================
>
> Now in patch v37 this test no longer exists. Why?
>
> ======
> 4.
> +# =============================================================================
> +# The following test cases demonstrate behavior of generated column replication
> +# when publish_generated_colums=false/true:
> +#
> +# Test: column list includes gencols, when publish_generated_columns=false
> +# Test: column list does not include gencols, when
> publish_generated_columns=false
> +#
> +# Test: column list includes gencols, when publish_generated_columns=true
> +# Test: column list does not include gencols, when
> publish_generated_columns=true
> +# Test: no column list, when publish_generated_columns=true
> +# =============================================================================
>
> These tests are currently only testing the initial tablesync
> replication. Since the COPY logic is different from the normal
> replication logic, I think it would be better to test some normal
> replication records as well, to make sure both parts work
> consistently. This comment applies to all of the following test cases.
>
> ~~~
>
> 5.
> +# Create table and publications.
> +$node_publisher->safe_psql(
> + 'postgres', qq(
> + CREATE TABLE nogen_to_gen3 (a int, b int, gen1 int GENERATED ALWAYS
> AS (a * 2) STORED, gen2 int GENERATED ALWAYS AS (a * 2) STORED);
> + CREATE TABLE nogen_to_gen4 (c int, d int, gen1 int GENERATED ALWAYS
> AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (c * 2) STORED);
> + INSERT INTO nogen_to_gen3 VALUES (1, 1);
> + INSERT INTO nogen_to_gen4 VALUES (1, 1);
> + CREATE PUBLICATION pub1 FOR table nogen_to_gen3, nogen_to_gen4(gen1)
> WITH (publish_generated_columns=true);
> +));
> +
>
> 5a.
> The code should do only what the comments say it does. So, the INSERTS
> should be done separately after the CREATE PUBLICATION, but before the
> CREATE SUBSCRIPTION. A similar change should be made for all of these
> test cases.
>
> # Insert some initial data
> INSERT INTO nogen_to_gen3 VALUES (1, 1);
> INSERT INTO nogen_to_gen4 VALUES (1, 1);
>
> ~
>
> 5b.
> The tables are badly named. Why are they 'nogen_to_gen', when the
> publisher side has generated cols and the subscriber side does not?
> This problem seems repeated in multiple subsequent test cases.
>
> ~
>
> 6.
> +$result = $node_subscriber->safe_psql('postgres',
> + "SELECT * FROM gen_to_nogen ORDER BY a");
> +is($result, qq(1|1||2),
> + 'gen_to_nogen initial sync, when publish_generated_columns=false');
> +
> +$result = $node_subscriber->safe_psql('postgres',
> + "SELECT * FROM gen_to_nogen2 ORDER BY c");
> +is($result, qq(1|1||),
> + 'gen_to_nogen2 initial sync, when publish_generated_columns=false');
>
> IMO all the "result" queries like these ones ought to have to have a
> comment which explains the reason for the expected results. This
> review comment applies to multiple places. Please add comments to all
> of them.
>
> ~~~
>
> 7.
> +# --------------------------------------------------
> +# Testcase: Publisher replicates the column list data excluding generated
> +# columns even though publish_generated_columns option is false.
> +# --------------------------------------------------
> +
>
> 7a.
> This is the 2nd test case, but AFAICT it would be far easier to test
> this scenario just by making another table (with an appropriate column
> list) for the 1st test case.
>
> ~
>
> 7b.
> BTW, I don't understand this test at all. I thought according to the
> comment that it intended to use a publication column list with only
> normal columns in it. But that is not what the publication looks like
> here:
> + CREATE PUBLICATION pub1 FOR table nogen_to_gen, nogen_to_gen2(gen1)
> WITH (publish_generated_columns=false);
>
> Indeed, the way it is currently written I didn't see what this test is
> doing that is any different from the prior test (???)
>
> ~~~
>
> 8.
> +# --------------------------------------------------
> +# Testcase: Although publish_generated_columns is true, publisher publishes
> +# only the data of the columns specified in column list, skipping other
> +# generated/non-generated columns.
> +# --------------------------------------------------
>
> versus
>
> +# --------------------------------------------------
> +# Testcase: Publisher publishes only the data of the columns specified in
> +# column list skipping other generated/non-generated columns.
> +# --------------------------------------------------
>
> Again, I did not understand how these test cases differ from each
> other. Surely, those can be combined easily enough just by adding
> another table with a different kind of column list.
>
> ~~~
>
> 9.
> +# --------------------------------------------------
> +# Testcase: Publisher replicates all columns if publish_generated_columns is
> +# enabled and there is no column list
> +# --------------------------------------------------
> +
>
> Here is yet another test case that AFAICT can just be combined with
> other test cases that were using publish_generated_columns=true. It
> seems all you need is one extra table with no column list. You don't
> need all the extra create/drop pub/sub overheads to test this.
>
> ======

I have fixed all the comments and posted the v39 patches for them.
Please refer to the updated v39 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjLjb%2B98i5ZQUphivxdOZ3hSGLfq2SiWQetUvk8zGyAQwQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  192. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-17T07:27:14Z

    On Wed, 16 Oct 2024 at 23:25, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Wed, Oct 9, 2024 at 9:08 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> > >
> > > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Shubham, here are my review comments for v36-0001.
> > > >
> > > > ======
> > > > 1. General  - merge patches
> > > >
> > > > It is long past due when patches 0001 and 0002 should've been merged.
> > > > AFAIK the split was only because historically these parts had
> > > > different authors. But, keeping them separated is not helpful anymore.
> > > >
> > > > ======
> > > > src/backend/catalog/pg_publication.c
> > > >
> > > > 2.
> > > >  Bitmapset *
> > > > -pub_collist_validate(Relation targetrel, List *columns)
> > > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > > >
> > > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > > so it should also be removed.
> > > >
> > > > ======
> > > > src/backend/replication/pgoutput/pgoutput.c
> > > >
> > > > 3.
> > > >   /*
> > > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > > - * if there are no column lists (even if other publications have a
> > > > - * list).
> > > > + * To handle cases where the publish_generated_columns option isn't
> > > > + * specified for all tables in a publication, we must create a column
> > > > + * list that excludes generated columns. So, the publisher will not
> > > > + * replicate the generated columns.
> > > >   */
> > > > - if (!pub->alltables)
> > > > + if (!(pub->alltables && pub->pubgencols))
> > > >
> > > > I still found that comment hard to understand. Does this mean to say
> > > > something like:
> > > >
> > > > ------
> > > > Process potential column lists for the following cases:
> > > >
> > > > a. Any publication that is not FOR ALL TABLES.
> > > >
> > > > b. When the publication is FOR ALL TABLES and
> > > > 'publish_generated_columns' is false.
> > > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > > so all columns will be replicated by default. However, if
> > > > 'publish_generated_columns' is set to false, column lists must still
> > > > be created to exclude any generated columns from being published
> > > > ------
> > > >
> > > > ======
> > > > src/test/regress/sql/publication.sql
> > > >
> > > > 4.
> > > > +SET client_min_messages = 'WARNING';
> > > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > >
> > > > AFAIK you don't need to keep changing 'client_min_messages',
> > > > particularly now that you've removed the WARNING message that was
> > > > previously emitted.
> > > >
> > > > ~
> > > >
> > > > 5.
> > > > nit - minor comment changes.
> > > >
> > > > ======
> > > > Please refer to the attachment which implements any nits from above.
> > > >
> > >
> > > I have fixed all the given comments. Also, I have created a new 0003
> > > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > > planning to merge 0001 and 0003 patches once they will get fixed.
> > > The attached patches contain the required changes.
> >
> > Few comments:
> > 1) Since we are no longer throwing an error for generated columns, the
> > function header comments also need to be updated accordingly " Checks
> > for and raises an ERROR for any; unknown columns, system columns,
> > duplicate columns or generated columns."
> > -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > -                       ereport(ERROR,
> > -
> > errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > -                                       errmsg("cannot use generated
> > column \"%s\" in publication column list",
> > -                                                  colname));
> > -
> >
> > 2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
> > ALTER PUBLICATION ... SET (
> > postgres=# alter publication pub2 set (PUBLISH
> > PUBLISH                     PUBLISH_VIA_PARTITION_ROOT
> >
> > 3) I was able to compile without this include, may be this is not required:
> > --- a/src/backend/replication/logical/tablesync.c
> > +++ b/src/backend/replication/logical/tablesync.c
> > @@ -118,6 +118,7 @@
> >  #include "utils/builtins.h"
> >  #include "utils/lsyscache.h"
> >  #include "utils/memutils.h"
> > +#include "utils/rel.h"
> >
> > 4) You can include "\dRp+ pubname" after each of the create/alter
> > publication to verify the columns that will be published:
> > +-- Test the 'publish_generated_columns' parameter enabled or disabled for
> > +-- different scenarios with/without generated columns in column lists.
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > +
> > +-- Generated columns in column list, when 'publish_generated_columns'=false
> > +CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
> > (publish_generated_columns=false);
> >
> > +-- Generated columns in column list, when 'publish_generated_columns'=true
> > +CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
> > (publish_generated_columns=true);
> > +
> > +-- Generated columns in column list, then set
> > 'publication_generate_columns'=false
> > +ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
> > +
> > +-- Remove generate columns from column list, when
> > 'publish_generated_columns'=false
> > +ALTER PUBLICATION pub2 SET TABLE gencols(a);
> > +
> > +-- Add generated columns in column list, when 'publish_generated_columns'=false
> > +ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);
> >
>
> I have fixed all the given comments. The attached patches contain the
> required changes.

Few comments:
1) This change is not required:
diff --git a/src/backend/catalog/pg_subscription.c
b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..fcfbf86c0b 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -551,3 +551,34 @@ GetSubscriptionRelations(Oid subid, bool not_ready)

        return res;
 }
+
+/*
+ * Add publication names from the list to a string.
+ */
+void
+get_publications_str(List *publications, StringInfo dest, bool quote_literal)
+{
+       ListCell   *lc;
+       bool            first = true;
+
+       Assert(publications != NIL);
+
+       foreach(lc, publications)
+       {
+               char       *pubname = strVal(lfirst(lc));
+
+               if (first)
+                       first = false;
+               else
+                       appendStringInfoString(dest, ", ");
+
+               if (quote_literal)
+                       appendStringInfoString(dest,
quote_literal_cstr(pubname));
+               else
+               {
+                       appendStringInfoChar(dest, '"');
+                       appendStringInfoString(dest, pubname);
+                       appendStringInfoChar(dest, '"');
+               }
+       }
+}

It can be moved to subscriptioncmds.c file as earlier.

2) This line change is not required:
  *             Process and validate the 'columns' list and ensure the
columns are all
- *             valid to use for a publication.  Checks for and raises
an ERROR for
- *             any; unknown columns, system columns, duplicate
columns or generated
- *             columns.
+ *             valid to use for a publication. Checks for and raises
an ERROR for

3) Can we store this information in LogicalRepRelation instead of
having a local variable as column information is being stored, that
way remotegenlist and remotegenlist_res can be removed and code will
be more simpler:
+               if (server_version >= 180000)
+               {
+                       remotegenlist[natt] =
DatumGetBool(slot_getattr(slot, 5, &isnull));
+
+                       /*
+                        * If the column is generated and neither the
generated column
+                        * option is specified nor it appears in the
column list, we will
+                        * skip it.
+                        */
+                       if (remotegenlist[natt] &&
!has_pub_with_pubgencols && !included_cols)
+                       {
+                               ExecClearTuple(slot);
+                               continue;
+                       }
+               }
+
                rel_colname = TextDatumGetCString(slot_getattr(slot,
2, &isnull));
                Assert(!isnull);

@@ -1015,7 +1112,7 @@ fetch_remote_table_info(char *nspname, char *relname,
        ExecDropSingleTupleTableSlot(slot);

        lrel->natts = natt;
-
+       *remotegenlist_res = remotegenlist;

Regards,
Vignesh

  193. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-17T10:29:01Z

    On Wed, 16 Oct 2024 at 23:25, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Wed, Oct 9, 2024 at 9:08 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> > >
> > > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Shubham, here are my review comments for v36-0001.
> > > >
> > > > ======
> > > > 1. General  - merge patches
> > > >
> > > > It is long past due when patches 0001 and 0002 should've been merged.
> > > > AFAIK the split was only because historically these parts had
> > > > different authors. But, keeping them separated is not helpful anymore.
> > > >
> > > > ======
> > > > src/backend/catalog/pg_publication.c
> > > >
> > > > 2.
> > > >  Bitmapset *
> > > > -pub_collist_validate(Relation targetrel, List *columns)
> > > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > > >
> > > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > > so it should also be removed.
> > > >
> > > > ======
> > > > src/backend/replication/pgoutput/pgoutput.c
> > > >
> > > > 3.
> > > >   /*
> > > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > > - * if there are no column lists (even if other publications have a
> > > > - * list).
> > > > + * To handle cases where the publish_generated_columns option isn't
> > > > + * specified for all tables in a publication, we must create a column
> > > > + * list that excludes generated columns. So, the publisher will not
> > > > + * replicate the generated columns.
> > > >   */
> > > > - if (!pub->alltables)
> > > > + if (!(pub->alltables && pub->pubgencols))
> > > >
> > > > I still found that comment hard to understand. Does this mean to say
> > > > something like:
> > > >
> > > > ------
> > > > Process potential column lists for the following cases:
> > > >
> > > > a. Any publication that is not FOR ALL TABLES.
> > > >
> > > > b. When the publication is FOR ALL TABLES and
> > > > 'publish_generated_columns' is false.
> > > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > > so all columns will be replicated by default. However, if
> > > > 'publish_generated_columns' is set to false, column lists must still
> > > > be created to exclude any generated columns from being published
> > > > ------
> > > >
> > > > ======
> > > > src/test/regress/sql/publication.sql
> > > >
> > > > 4.
> > > > +SET client_min_messages = 'WARNING';
> > > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > >
> > > > AFAIK you don't need to keep changing 'client_min_messages',
> > > > particularly now that you've removed the WARNING message that was
> > > > previously emitted.
> > > >
> > > > ~
> > > >
> > > > 5.
> > > > nit - minor comment changes.
> > > >
> > > > ======
> > > > Please refer to the attachment which implements any nits from above.
> > > >
> > >
> > > I have fixed all the given comments. Also, I have created a new 0003
> > > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > > planning to merge 0001 and 0003 patches once they will get fixed.
> > > The attached patches contain the required changes.
> >
> > Few comments:
> > 1) Since we are no longer throwing an error for generated columns, the
> > function header comments also need to be updated accordingly " Checks
> > for and raises an ERROR for any; unknown columns, system columns,
> > duplicate columns or generated columns."
> > -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > -                       ereport(ERROR,
> > -
> > errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > -                                       errmsg("cannot use generated
> > column \"%s\" in publication column list",
> > -                                                  colname));
> > -
> >
> > 2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
> > ALTER PUBLICATION ... SET (
> > postgres=# alter publication pub2 set (PUBLISH
> > PUBLISH                     PUBLISH_VIA_PARTITION_ROOT
> >
> > 3) I was able to compile without this include, may be this is not required:
> > --- a/src/backend/replication/logical/tablesync.c
> > +++ b/src/backend/replication/logical/tablesync.c
> > @@ -118,6 +118,7 @@
> >  #include "utils/builtins.h"
> >  #include "utils/lsyscache.h"
> >  #include "utils/memutils.h"
> > +#include "utils/rel.h"
> >
> > 4) You can include "\dRp+ pubname" after each of the create/alter
> > publication to verify the columns that will be published:
> > +-- Test the 'publish_generated_columns' parameter enabled or disabled for
> > +-- different scenarios with/without generated columns in column lists.
> > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > +
> > +-- Generated columns in column list, when 'publish_generated_columns'=false
> > +CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
> > (publish_generated_columns=false);
> >
> > +-- Generated columns in column list, when 'publish_generated_columns'=true
> > +CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
> > (publish_generated_columns=true);
> > +
> > +-- Generated columns in column list, then set
> > 'publication_generate_columns'=false
> > +ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
> > +
> > +-- Remove generate columns from column list, when
> > 'publish_generated_columns'=false
> > +ALTER PUBLICATION pub2 SET TABLE gencols(a);
> > +
> > +-- Add generated columns in column list, when 'publish_generated_columns'=false
> > +ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);
> >
>
> I have fixed all the given comments. The attached patches contain the
> required changes.

Few comments:
1) File mode change is not required:
 src/test/subscription/t/011_generated.pl | 354 +++++++++++++++++++++++
 1 file changed, 354 insertions(+)
 mode change 100644 => 100755 src/test/subscription/t/011_generated.pl

diff --git a/src/test/subscription/t/011_generated.pl
b/src/test/subscription/t/011_generated.pl
old mode 100644
new mode 100755
index 8b2e5f4708..d1f2718078
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl

2) Here copy_data=true looks obvious no need to mention again and
again in comments:
+# Create table and subscription with copy_data=true.
+$node_subscriber->safe_psql(
+       'postgres', qq(
+       CREATE TABLE tab_gen_to_nogen (a int, b int);
+       CREATE SUBSCRIPTION regress_sub1_gen_to_nogen CONNECTION
'$publisher_connstr' PUBLICATION regress_pub1_gen_to_nogen WITH
(copy_data = true);
+));
+
+# Create table and subscription with copy_data=true.
+$node_subscriber->safe_psql(
+       'test_pgc_true', qq(
+       CREATE TABLE tab_gen_to_nogen (a int, b int);
+       CREATE SUBSCRIPTION regress_sub2_gen_to_nogen CONNECTION
'$publisher_connstr' PUBLICATION regress_pub2_gen_to_nogen WITH
(copy_data = true);
+));
+
+# Wait for initial sync.
+$node_subscriber->wait_for_subscription_sync($node_publisher,
+       'regress_sub1_gen_to_nogen', 'postgres');
+$node_subscriber->wait_for_subscription_sync($node_publisher,
+       'regress_sub2_gen_to_nogen', 'test_pgc_true');
+
+# Initial sync test when publish_generated_columns=false and copy_data=true.
+# Verify that column 'b' is not replicated.
+$result = $node_subscriber->safe_psql('postgres',
+       "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|
+2|
+3|), 'tab_gen_to_nogen initial sync, when publish_generated_columns=false');
+
+# Initial sync test when publish_generated_columns=true and copy_data=true.
+$result = $node_subscriber->safe_psql('test_pgc_true',
+       "SELECT a, b FROM tab_gen_to_nogen");
+is( $result, qq(1|2
+2|4
+3|6),
+       'tab_gen_to_nogen initial sync, when publish_generated_columns=true');

3) The database test_pgc_true and also can be cleaned as it is not
required after this:
+# cleanup
+$node_subscriber->safe_psql('postgres',
+       "DROP SUBSCRIPTION regress_sub1_gen_to_nogen");
+$node_subscriber->safe_psql('test_pgc_true',
+       "DROP SUBSCRIPTION regress_sub2_gen_to_nogen");
+$node_publisher->safe_psql(
+       'postgres', qq(
+       DROP PUBLICATION regress_pub1_gen_to_nogen;
+       DROP PUBLICATION regress_pub2_gen_to_nogen;
+));

4) There is no error message verification in this test, let's add the
error verification:
+# =============================================================================
+# Misc test.
+#
+# A "normal -> generated" replication fails, reporting an error that the
+# subscriber side column is missing.
+#
+# In this test case we use DROP EXPRESSION to change the subscriber generated
+# column into a normal column, then verify replication works ok.
+# =============================================================================

5)
5.a) If possible have one regular column and one generated column in the tables
+# --------------------------------------------------
+# Testcase: Publisher replicates the column list data including generated
+# columns even though publish_generated_columns option is false.
+# --------------------------------------------------
+
+# Create table and publications.
+$node_publisher->safe_psql(
+       'postgres', qq(
+       CREATE TABLE gen_to_nogen (a int, b int, gen1 int GENERATED
ALWAYS AS (a * 2) STORED, gen2 int GENERATED ALWAYS AS (a * 2)
STORED);
+       CREATE TABLE gen_to_nogen2 (c int, d int, gen1 int GENERATED
ALWAYS AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (c * 2)
STORED);
+       CREATE TABLE nogen_to_gen2 (c int, d int, gen1 int GENERATED
ALWAYS AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (c * 2)
STORED);
+       CREATE PUBLICATION pub1 FOR table gen_to_nogen(a, b, gen2),
gen_to_nogen2, nogen_to_gen2(gen1) WITH
(publish_generated_columns=false);
+));

5.b) Try to have same columns in all the tables

6) These are inserting two records:
+# Insert data to verify incremental replication
+$node_publisher->safe_psql(
+       'postgres', qq(
+       INSERT INTO gen_to_nogen VALUES (2), (3);
+       INSERT INTO gen_to_nogen2 VALUES (2), (3);
+       INSERT INTO nogen_to_gen2 VALUES (2), (3);
+));

I felt you wanted this to be:
+# Insert data to verify incremental replication
+$node_publisher->safe_psql(
+       'postgres', qq(
+       INSERT INTO gen_to_nogen VALUES (2, 3);
+       INSERT INTO gen_to_nogen2 VALUES (2, 3);
+       INSERT INTO nogen_to_gen2 VALUES (2, 3);
+));

Regards,
Vignesh

  194. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-18T12:12:08Z

    On Thu, Oct 17, 2024 at 12:58 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 16 Oct 2024 at 23:25, Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, Oct 9, 2024 at 9:08 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> > > >
> > > > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > > >
> > > > > Hi Shubham, here are my review comments for v36-0001.
> > > > >
> > > > > ======
> > > > > 1. General  - merge patches
> > > > >
> > > > > It is long past due when patches 0001 and 0002 should've been merged.
> > > > > AFAIK the split was only because historically these parts had
> > > > > different authors. But, keeping them separated is not helpful anymore.
> > > > >
> > > > > ======
> > > > > src/backend/catalog/pg_publication.c
> > > > >
> > > > > 2.
> > > > >  Bitmapset *
> > > > > -pub_collist_validate(Relation targetrel, List *columns)
> > > > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > > > >
> > > > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > > > so it should also be removed.
> > > > >
> > > > > ======
> > > > > src/backend/replication/pgoutput/pgoutput.c
> > > > >
> > > > > 3.
> > > > >   /*
> > > > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > > > - * if there are no column lists (even if other publications have a
> > > > > - * list).
> > > > > + * To handle cases where the publish_generated_columns option isn't
> > > > > + * specified for all tables in a publication, we must create a column
> > > > > + * list that excludes generated columns. So, the publisher will not
> > > > > + * replicate the generated columns.
> > > > >   */
> > > > > - if (!pub->alltables)
> > > > > + if (!(pub->alltables && pub->pubgencols))
> > > > >
> > > > > I still found that comment hard to understand. Does this mean to say
> > > > > something like:
> > > > >
> > > > > ------
> > > > > Process potential column lists for the following cases:
> > > > >
> > > > > a. Any publication that is not FOR ALL TABLES.
> > > > >
> > > > > b. When the publication is FOR ALL TABLES and
> > > > > 'publish_generated_columns' is false.
> > > > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > > > so all columns will be replicated by default. However, if
> > > > > 'publish_generated_columns' is set to false, column lists must still
> > > > > be created to exclude any generated columns from being published
> > > > > ------
> > > > >
> > > > > ======
> > > > > src/test/regress/sql/publication.sql
> > > > >
> > > > > 4.
> > > > > +SET client_min_messages = 'WARNING';
> > > > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > > >
> > > > > AFAIK you don't need to keep changing 'client_min_messages',
> > > > > particularly now that you've removed the WARNING message that was
> > > > > previously emitted.
> > > > >
> > > > > ~
> > > > >
> > > > > 5.
> > > > > nit - minor comment changes.
> > > > >
> > > > > ======
> > > > > Please refer to the attachment which implements any nits from above.
> > > > >
> > > >
> > > > I have fixed all the given comments. Also, I have created a new 0003
> > > > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > > > planning to merge 0001 and 0003 patches once they will get fixed.
> > > > The attached patches contain the required changes.
> > >
> > > Few comments:
> > > 1) Since we are no longer throwing an error for generated columns, the
> > > function header comments also need to be updated accordingly " Checks
> > > for and raises an ERROR for any; unknown columns, system columns,
> > > duplicate columns or generated columns."
> > > -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > > -                       ereport(ERROR,
> > > -
> > > errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > > -                                       errmsg("cannot use generated
> > > column \"%s\" in publication column list",
> > > -                                                  colname));
> > > -
> > >
> > > 2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
> > > ALTER PUBLICATION ... SET (
> > > postgres=# alter publication pub2 set (PUBLISH
> > > PUBLISH                     PUBLISH_VIA_PARTITION_ROOT
> > >
> > > 3) I was able to compile without this include, may be this is not required:
> > > --- a/src/backend/replication/logical/tablesync.c
> > > +++ b/src/backend/replication/logical/tablesync.c
> > > @@ -118,6 +118,7 @@
> > >  #include "utils/builtins.h"
> > >  #include "utils/lsyscache.h"
> > >  #include "utils/memutils.h"
> > > +#include "utils/rel.h"
> > >
> > > 4) You can include "\dRp+ pubname" after each of the create/alter
> > > publication to verify the columns that will be published:
> > > +-- Test the 'publish_generated_columns' parameter enabled or disabled for
> > > +-- different scenarios with/without generated columns in column lists.
> > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > +
> > > +-- Generated columns in column list, when 'publish_generated_columns'=false
> > > +CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
> > > (publish_generated_columns=false);
> > >
> > > +-- Generated columns in column list, when 'publish_generated_columns'=true
> > > +CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
> > > (publish_generated_columns=true);
> > > +
> > > +-- Generated columns in column list, then set
> > > 'publication_generate_columns'=false
> > > +ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
> > > +
> > > +-- Remove generate columns from column list, when
> > > 'publish_generated_columns'=false
> > > +ALTER PUBLICATION pub2 SET TABLE gencols(a);
> > > +
> > > +-- Add generated columns in column list, when 'publish_generated_columns'=false
> > > +ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);
> > >
> >
> > I have fixed all the given comments. The attached patches contain the
> > required changes.
>
> Few comments:
> 1) This change is not required:
> diff --git a/src/backend/catalog/pg_subscription.c
> b/src/backend/catalog/pg_subscription.c
> index 9efc9159f2..fcfbf86c0b 100644
> --- a/src/backend/catalog/pg_subscription.c
> +++ b/src/backend/catalog/pg_subscription.c
> @@ -551,3 +551,34 @@ GetSubscriptionRelations(Oid subid, bool not_ready)
>
>         return res;
>  }
> +
> +/*
> + * Add publication names from the list to a string.
> + */
> +void
> +get_publications_str(List *publications, StringInfo dest, bool quote_literal)
> +{
> +       ListCell   *lc;
> +       bool            first = true;
> +
> +       Assert(publications != NIL);
> +
> +       foreach(lc, publications)
> +       {
> +               char       *pubname = strVal(lfirst(lc));
> +
> +               if (first)
> +                       first = false;
> +               else
> +                       appendStringInfoString(dest, ", ");
> +
> +               if (quote_literal)
> +                       appendStringInfoString(dest,
> quote_literal_cstr(pubname));
> +               else
> +               {
> +                       appendStringInfoChar(dest, '"');
> +                       appendStringInfoString(dest, pubname);
> +                       appendStringInfoChar(dest, '"');
> +               }
> +       }
> +}
>
> It can be moved to subscriptioncmds.c file as earlier.
>
> 2) This line change is not required:
>   *             Process and validate the 'columns' list and ensure the
> columns are all
> - *             valid to use for a publication.  Checks for and raises
> an ERROR for
> - *             any; unknown columns, system columns, duplicate
> columns or generated
> - *             columns.
> + *             valid to use for a publication. Checks for and raises
> an ERROR for
>
> 3) Can we store this information in LogicalRepRelation instead of
> having a local variable as column information is being stored, that
> way remotegenlist and remotegenlist_res can be removed and code will
> be more simpler:
> +               if (server_version >= 180000)
> +               {
> +                       remotegenlist[natt] =
> DatumGetBool(slot_getattr(slot, 5, &isnull));
> +
> +                       /*
> +                        * If the column is generated and neither the
> generated column
> +                        * option is specified nor it appears in the
> column list, we will
> +                        * skip it.
> +                        */
> +                       if (remotegenlist[natt] &&
> !has_pub_with_pubgencols && !included_cols)
> +                       {
> +                               ExecClearTuple(slot);
> +                               continue;
> +                       }
> +               }
> +
>                 rel_colname = TextDatumGetCString(slot_getattr(slot,
> 2, &isnull));
>                 Assert(!isnull);
>
> @@ -1015,7 +1112,7 @@ fetch_remote_table_info(char *nspname, char *relname,
>         ExecDropSingleTupleTableSlot(slot);
>
>         lrel->natts = natt;
> -
> +       *remotegenlist_res = remotegenlist;

I have fixed all the given comments. The attached v40-0001 patch
contains the required changes.

Thanks and Regards,
Shubham Khanna.

  195. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-18T12:15:22Z

    On Thu, Oct 17, 2024 at 3:59 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 16 Oct 2024 at 23:25, Shubham Khanna
> <khannashubham1197@gmail.com> wrote:
> >
> > On Wed, Oct 9, 2024 at 9:08 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> > > >
> > > > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > > >
> > > > > Hi Shubham, here are my review comments for v36-0001.
> > > > >
> > > > > ======
> > > > > 1. General  - merge patches
> > > > >
> > > > > It is long past due when patches 0001 and 0002 should've been merged.
> > > > > AFAIK the split was only because historically these parts had
> > > > > different authors. But, keeping them separated is not helpful anymore.
> > > > >
> > > > > ======
> > > > > src/backend/catalog/pg_publication.c
> > > > >
> > > > > 2.
> > > > >  Bitmapset *
> > > > > -pub_collist_validate(Relation targetrel, List *columns)
> > > > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > > > >
> > > > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > > > so it should also be removed.
> > > > >
> > > > > ======
> > > > > src/backend/replication/pgoutput/pgoutput.c
> > > > >
> > > > > 3.
> > > > >   /*
> > > > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > > > - * if there are no column lists (even if other publications have a
> > > > > - * list).
> > > > > + * To handle cases where the publish_generated_columns option isn't
> > > > > + * specified for all tables in a publication, we must create a column
> > > > > + * list that excludes generated columns. So, the publisher will not
> > > > > + * replicate the generated columns.
> > > > >   */
> > > > > - if (!pub->alltables)
> > > > > + if (!(pub->alltables && pub->pubgencols))
> > > > >
> > > > > I still found that comment hard to understand. Does this mean to say
> > > > > something like:
> > > > >
> > > > > ------
> > > > > Process potential column lists for the following cases:
> > > > >
> > > > > a. Any publication that is not FOR ALL TABLES.
> > > > >
> > > > > b. When the publication is FOR ALL TABLES and
> > > > > 'publish_generated_columns' is false.
> > > > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > > > so all columns will be replicated by default. However, if
> > > > > 'publish_generated_columns' is set to false, column lists must still
> > > > > be created to exclude any generated columns from being published
> > > > > ------
> > > > >
> > > > > ======
> > > > > src/test/regress/sql/publication.sql
> > > > >
> > > > > 4.
> > > > > +SET client_min_messages = 'WARNING';
> > > > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > > >
> > > > > AFAIK you don't need to keep changing 'client_min_messages',
> > > > > particularly now that you've removed the WARNING message that was
> > > > > previously emitted.
> > > > >
> > > > > ~
> > > > >
> > > > > 5.
> > > > > nit - minor comment changes.
> > > > >
> > > > > ======
> > > > > Please refer to the attachment which implements any nits from above.
> > > > >
> > > >
> > > > I have fixed all the given comments. Also, I have created a new 0003
> > > > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > > > planning to merge 0001 and 0003 patches once they will get fixed.
> > > > The attached patches contain the required changes.
> > >
> > > Few comments:
> > > 1) Since we are no longer throwing an error for generated columns, the
> > > function header comments also need to be updated accordingly " Checks
> > > for and raises an ERROR for any; unknown columns, system columns,
> > > duplicate columns or generated columns."
> > > -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > > -                       ereport(ERROR,
> > > -
> > > errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > > -                                       errmsg("cannot use generated
> > > column \"%s\" in publication column list",
> > > -                                                  colname));
> > > -
> > >
> > > 2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
> > > ALTER PUBLICATION ... SET (
> > > postgres=# alter publication pub2 set (PUBLISH
> > > PUBLISH                     PUBLISH_VIA_PARTITION_ROOT
> > >
> > > 3) I was able to compile without this include, may be this is not required:
> > > --- a/src/backend/replication/logical/tablesync.c
> > > +++ b/src/backend/replication/logical/tablesync.c
> > > @@ -118,6 +118,7 @@
> > >  #include "utils/builtins.h"
> > >  #include "utils/lsyscache.h"
> > >  #include "utils/memutils.h"
> > > +#include "utils/rel.h"
> > >
> > > 4) You can include "\dRp+ pubname" after each of the create/alter
> > > publication to verify the columns that will be published:
> > > +-- Test the 'publish_generated_columns' parameter enabled or disabled for
> > > +-- different scenarios with/without generated columns in column lists.
> > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > +
> > > +-- Generated columns in column list, when 'publish_generated_columns'=false
> > > +CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
> > > (publish_generated_columns=false);
> > >
> > > +-- Generated columns in column list, when 'publish_generated_columns'=true
> > > +CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
> > > (publish_generated_columns=true);
> > > +
> > > +-- Generated columns in column list, then set
> > > 'publication_generate_columns'=false
> > > +ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
> > > +
> > > +-- Remove generate columns from column list, when
> > > 'publish_generated_columns'=false
> > > +ALTER PUBLICATION pub2 SET TABLE gencols(a);
> > > +
> > > +-- Add generated columns in column list, when 'publish_generated_columns'=false
> > > +ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);
> > >
> >
> > I have fixed all the given comments. The attached patches contain the
> > required changes.
>
> Few comments:
> 1) File mode change is not required:
>  src/test/subscription/t/011_generated.pl | 354 +++++++++++++++++++++++
>  1 file changed, 354 insertions(+)
>  mode change 100644 => 100755 src/test/subscription/t/011_generated.pl
>
> diff --git a/src/test/subscription/t/011_generated.pl
> b/src/test/subscription/t/011_generated.pl
> old mode 100644
> new mode 100755
> index 8b2e5f4708..d1f2718078
> --- a/src/test/subscription/t/011_generated.pl
> +++ b/src/test/subscription/t/011_generated.pl
>
> 2) Here copy_data=true looks obvious no need to mention again and
> again in comments:
> +# Create table and subscription with copy_data=true.
> +$node_subscriber->safe_psql(
> +       'postgres', qq(
> +       CREATE TABLE tab_gen_to_nogen (a int, b int);
> +       CREATE SUBSCRIPTION regress_sub1_gen_to_nogen CONNECTION
> '$publisher_connstr' PUBLICATION regress_pub1_gen_to_nogen WITH
> (copy_data = true);
> +));
> +
> +# Create table and subscription with copy_data=true.
> +$node_subscriber->safe_psql(
> +       'test_pgc_true', qq(
> +       CREATE TABLE tab_gen_to_nogen (a int, b int);
> +       CREATE SUBSCRIPTION regress_sub2_gen_to_nogen CONNECTION
> '$publisher_connstr' PUBLICATION regress_pub2_gen_to_nogen WITH
> (copy_data = true);
> +));
> +
> +# Wait for initial sync.
> +$node_subscriber->wait_for_subscription_sync($node_publisher,
> +       'regress_sub1_gen_to_nogen', 'postgres');
> +$node_subscriber->wait_for_subscription_sync($node_publisher,
> +       'regress_sub2_gen_to_nogen', 'test_pgc_true');
> +
> +# Initial sync test when publish_generated_columns=false and copy_data=true.
> +# Verify that column 'b' is not replicated.
> +$result = $node_subscriber->safe_psql('postgres',
> +       "SELECT a, b FROM tab_gen_to_nogen");
> +is( $result, qq(1|
> +2|
> +3|), 'tab_gen_to_nogen initial sync, when publish_generated_columns=false');
> +
> +# Initial sync test when publish_generated_columns=true and copy_data=true.
> +$result = $node_subscriber->safe_psql('test_pgc_true',
> +       "SELECT a, b FROM tab_gen_to_nogen");
> +is( $result, qq(1|2
> +2|4
> +3|6),
> +       'tab_gen_to_nogen initial sync, when publish_generated_columns=true');
>
> 3) The database test_pgc_true and also can be cleaned as it is not
> required after this:
> +# cleanup
> +$node_subscriber->safe_psql('postgres',
> +       "DROP SUBSCRIPTION regress_sub1_gen_to_nogen");
> +$node_subscriber->safe_psql('test_pgc_true',
> +       "DROP SUBSCRIPTION regress_sub2_gen_to_nogen");
> +$node_publisher->safe_psql(
> +       'postgres', qq(
> +       DROP PUBLICATION regress_pub1_gen_to_nogen;
> +       DROP PUBLICATION regress_pub2_gen_to_nogen;
> +));
>
> 4) There is no error message verification in this test, let's add the
> error verification:
> +# =============================================================================
> +# Misc test.
> +#
> +# A "normal -> generated" replication fails, reporting an error that the
> +# subscriber side column is missing.
> +#
> +# In this test case we use DROP EXPRESSION to change the subscriber generated
> +# column into a normal column, then verify replication works ok.
> +# =============================================================================
>
> 5)
> 5.a) If possible have one regular column and one generated column in the tables
> +# --------------------------------------------------
> +# Testcase: Publisher replicates the column list data including generated
> +# columns even though publish_generated_columns option is false.
> +# --------------------------------------------------
> +
> +# Create table and publications.
> +$node_publisher->safe_psql(
> +       'postgres', qq(
> +       CREATE TABLE gen_to_nogen (a int, b int, gen1 int GENERATED
> ALWAYS AS (a * 2) STORED, gen2 int GENERATED ALWAYS AS (a * 2)
> STORED);
> +       CREATE TABLE gen_to_nogen2 (c int, d int, gen1 int GENERATED
> ALWAYS AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (c * 2)
> STORED);
> +       CREATE TABLE nogen_to_gen2 (c int, d int, gen1 int GENERATED
> ALWAYS AS (c * 2) STORED, gen2 int GENERATED ALWAYS AS (c * 2)
> STORED);
> +       CREATE PUBLICATION pub1 FOR table gen_to_nogen(a, b, gen2),
> gen_to_nogen2, nogen_to_gen2(gen1) WITH
> (publish_generated_columns=false);
> +));
>
> 5.b) Try to have same columns in all the tables
>
> 6) These are inserting two records:
> +# Insert data to verify incremental replication
> +$node_publisher->safe_psql(
> +       'postgres', qq(
> +       INSERT INTO gen_to_nogen VALUES (2), (3);
> +       INSERT INTO gen_to_nogen2 VALUES (2), (3);
> +       INSERT INTO nogen_to_gen2 VALUES (2), (3);
> +));
>
> I felt you wanted this to be:
> +# Insert data to verify incremental replication
> +$node_publisher->safe_psql(
> +       'postgres', qq(
> +       INSERT INTO gen_to_nogen VALUES (2, 3);
> +       INSERT INTO gen_to_nogen2 VALUES (2, 3);
> +       INSERT INTO nogen_to_gen2 VALUES (2, 3);
> +));

I have fixed all the comments and posted the v40 patches for them.
Please refer to the updated v40 Patches here in [1]. See [1] for the
changes added.

[1] https://www.postgresql.org/message-id/CAHv8RjLviXAWtB3Kcn1A1jPpqORpkNay1y2U%2B55K64sqwCdrGw%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  196. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-21T03:37:21Z

    On Fri, 18 Oct 2024 at 17:42, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
>
> I have fixed all the given comments. The attached v40-0001 patch
> contains the required changes.

1) The recent patch removed the function header comment where
generated column is specified, that change is required:
@@ -511,7 +511,6 @@ pub_collist_validate(Relation targetrel, List *columns)
 {
        Bitmapset  *set = NULL;
        ListCell   *lc;
-       TupleDesc       tupdesc = RelationGetDescr(targetrel);

        foreach(lc, columns)
        {
@@ -530,12 +529,6 @@ pub_collist_validate(Relation targetrel, List *columns)
                                        errmsg("cannot use system
column \"%s\" in publication column list",
                                                   colname));

-               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
-                       ereport(ERROR,
-
errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
-                                       errmsg("cannot use generated
column \"%s\" in publication column list",
-                                                  colname));
-

2) This change is no more required as get_publications_str changes are
removed now:
diff --git a/src/include/catalog/pg_subscription.h
b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..6657186317 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -20,6 +20,7 @@
 #include "access/xlogdefs.h"
 #include "catalog/genbki.h"
 #include "catalog/pg_subscription_d.h"
+#include "lib/stringinfo.h"

Regards,
Vignesh

  197. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-21T05:21:13Z

    On Fri, 18 Oct 2024 at 17:42, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> On Thu, Oct 17, 2024 at 12:58 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 16 Oct 2024 at 23:25, Shubham Khanna
> > <khannashubham1197@gmail.com> wrote:
> > >
> > > On Wed, Oct 9, 2024 at 9:08 AM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Tue, 8 Oct 2024 at 11:37, Shubham Khanna <khannashubham1197@gmail.com> wrote:
> > > > >
> > > > > On Fri, Oct 4, 2024 at 9:36 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > > > >
> > > > > > Hi Shubham, here are my review comments for v36-0001.
> > > > > >
> > > > > > ======
> > > > > > 1. General  - merge patches
> > > > > >
> > > > > > It is long past due when patches 0001 and 0002 should've been merged.
> > > > > > AFAIK the split was only because historically these parts had
> > > > > > different authors. But, keeping them separated is not helpful anymore.
> > > > > >
> > > > > > ======
> > > > > > src/backend/catalog/pg_publication.c
> > > > > >
> > > > > > 2.
> > > > > >  Bitmapset *
> > > > > > -pub_collist_validate(Relation targetrel, List *columns)
> > > > > > +pub_collist_validate(Relation targetrel, List *columns, bool pubgencols)
> > > > > >
> > > > > > Since you removed the WARNING, this parameter 'pubgencols' is unused
> > > > > > so it should also be removed.
> > > > > >
> > > > > > ======
> > > > > > src/backend/replication/pgoutput/pgoutput.c
> > > > > >
> > > > > > 3.
> > > > > >   /*
> > > > > > - * If the publication is FOR ALL TABLES then it is treated the same as
> > > > > > - * if there are no column lists (even if other publications have a
> > > > > > - * list).
> > > > > > + * To handle cases where the publish_generated_columns option isn't
> > > > > > + * specified for all tables in a publication, we must create a column
> > > > > > + * list that excludes generated columns. So, the publisher will not
> > > > > > + * replicate the generated columns.
> > > > > >   */
> > > > > > - if (!pub->alltables)
> > > > > > + if (!(pub->alltables && pub->pubgencols))
> > > > > >
> > > > > > I still found that comment hard to understand. Does this mean to say
> > > > > > something like:
> > > > > >
> > > > > > ------
> > > > > > Process potential column lists for the following cases:
> > > > > >
> > > > > > a. Any publication that is not FOR ALL TABLES.
> > > > > >
> > > > > > b. When the publication is FOR ALL TABLES and
> > > > > > 'publish_generated_columns' is false.
> > > > > > A FOR ALL TABLES publication doesn't have user-defined column lists,
> > > > > > so all columns will be replicated by default. However, if
> > > > > > 'publish_generated_columns' is set to false, column lists must still
> > > > > > be created to exclude any generated columns from being published
> > > > > > ------
> > > > > >
> > > > > > ======
> > > > > > src/test/regress/sql/publication.sql
> > > > > >
> > > > > > 4.
> > > > > > +SET client_min_messages = 'WARNING';
> > > > > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > > > >
> > > > > > AFAIK you don't need to keep changing 'client_min_messages',
> > > > > > particularly now that you've removed the WARNING message that was
> > > > > > previously emitted.
> > > > > >
> > > > > > ~
> > > > > >
> > > > > > 5.
> > > > > > nit - minor comment changes.
> > > > > >
> > > > > > ======
> > > > > > Please refer to the attachment which implements any nits from above.
> > > > > >
> > > > >
> > > > > I have fixed all the given comments. Also, I have created a new 0003
> > > > > patch for the TAP-Tests related to the '011_generated.pl' file. I am
> > > > > planning to merge 0001 and 0003 patches once they will get fixed.
> > > > > The attached patches contain the required changes.
> > > >
> > > > Few comments:
> > > > 1) Since we are no longer throwing an error for generated columns, the
> > > > function header comments also need to be updated accordingly " Checks
> > > > for and raises an ERROR for any; unknown columns, system columns,
> > > > duplicate columns or generated columns."
> > > > -               if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
> > > > -                       ereport(ERROR,
> > > > -
> > > > errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > > > -                                       errmsg("cannot use generated
> > > > column \"%s\" in publication column list",
> > > > -                                                  colname));
> > > > -
> > > >
> > > > 2) Tab completion missing for "PUBLISH_GENERATED_COLUMNS" option in
> > > > ALTER PUBLICATION ... SET (
> > > > postgres=# alter publication pub2 set (PUBLISH
> > > > PUBLISH                     PUBLISH_VIA_PARTITION_ROOT
> > > >
> > > > 3) I was able to compile without this include, may be this is not required:
> > > > --- a/src/backend/replication/logical/tablesync.c
> > > > +++ b/src/backend/replication/logical/tablesync.c
> > > > @@ -118,6 +118,7 @@
> > > >  #include "utils/builtins.h"
> > > >  #include "utils/lsyscache.h"
> > > >  #include "utils/memutils.h"
> > > > +#include "utils/rel.h"
> > > >
> > > > 4) You can include "\dRp+ pubname" after each of the create/alter
> > > > publication to verify the columns that will be published:
> > > > +-- Test the 'publish_generated_columns' parameter enabled or disabled for
> > > > +-- different scenarios with/without generated columns in column lists.
> > > > +CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
> > > > +
> > > > +-- Generated columns in column list, when 'publish_generated_columns'=false
> > > > +CREATE PUBLICATION pub1 FOR table gencols(a, gen1) WITH
> > > > (publish_generated_columns=false);
> > > >
> > > > +-- Generated columns in column list, when 'publish_generated_columns'=true
> > > > +CREATE PUBLICATION pub2 FOR table gencols(a, gen1) WITH
> > > > (publish_generated_columns=true);
> > > > +
> > > > +-- Generated columns in column list, then set
> > > > 'publication_generate_columns'=false
> > > > +ALTER PUBLICATION pub2 SET (publish_generated_columns = false);
> > > > +
> > > > +-- Remove generate columns from column list, when
> > > > 'publish_generated_columns'=false
> > > > +ALTER PUBLICATION pub2 SET TABLE gencols(a);
> > > > +
> > > > +-- Add generated columns in column list, when 'publish_generated_columns'=false
> > > > +ALTER PUBLICATION pub2 SET TABLE gencols(a, gen1);
> > > >
> > >
> > > I have fixed all the given comments. The attached patches contain the
> > > required changes.
> >
> > Few comments:
> > 1) This change is not required:
> > diff --git a/src/backend/catalog/pg_subscription.c
> > b/src/backend/catalog/pg_subscription.c
> > index 9efc9159f2..fcfbf86c0b 100644
> > --- a/src/backend/catalog/pg_subscription.c
> > +++ b/src/backend/catalog/pg_subscription.c
> > @@ -551,3 +551,34 @@ GetSubscriptionRelations(Oid subid, bool not_ready)
> >
> >         return res;
> >  }
> > +
> > +/*
> > + * Add publication names from the list to a string.
> > + */
> > +void
> > +get_publications_str(List *publications, StringInfo dest, bool quote_literal)
> > +{
> > +       ListCell   *lc;
> > +       bool            first = true;
> > +
> > +       Assert(publications != NIL);
> > +
> > +       foreach(lc, publications)
> > +       {
> > +               char       *pubname = strVal(lfirst(lc));
> > +
> > +               if (first)
> > +                       first = false;
> > +               else
> > +                       appendStringInfoString(dest, ", ");
> > +
> > +               if (quote_literal)
> > +                       appendStringInfoString(dest,
> > quote_literal_cstr(pubname));
> > +               else
> > +               {
> > +                       appendStringInfoChar(dest, '"');
> > +                       appendStringInfoString(dest, pubname);
> > +                       appendStringInfoChar(dest, '"');
> > +               }
> > +       }
> > +}
> >
> > It can be moved to subscriptioncmds.c file as earlier.
> >
> > 2) This line change is not required:
> >   *             Process and validate the 'columns' list and ensure the
> > columns are all
> > - *             valid to use for a publication.  Checks for and raises
> > an ERROR for
> > - *             any; unknown columns, system columns, duplicate
> > columns or generated
> > - *             columns.
> > + *             valid to use for a publication. Checks for and raises
> > an ERROR for
> >
> > 3) Can we store this information in LogicalRepRelation instead of
> > having a local variable as column information is being stored, that
> > way remotegenlist and remotegenlist_res can be removed and code will
> > be more simpler:
> > +               if (server_version >= 180000)
> > +               {
> > +                       remotegenlist[natt] =
> > DatumGetBool(slot_getattr(slot, 5, &isnull));
> > +
> > +                       /*
> > +                        * If the column is generated and neither the
> > generated column
> > +                        * option is specified nor it appears in the
> > column list, we will
> > +                        * skip it.
> > +                        */
> > +                       if (remotegenlist[natt] &&
> > !has_pub_with_pubgencols && !included_cols)
> > +                       {
> > +                               ExecClearTuple(slot);
> > +                               continue;
> > +                       }
> > +               }
> > +
> >                 rel_colname = TextDatumGetCString(slot_getattr(slot,
> > 2, &isnull));
> >                 Assert(!isnull);
> >
> > @@ -1015,7 +1112,7 @@ fetch_remote_table_info(char *nspname, char *relname,
> >         ExecDropSingleTupleTableSlot(slot);
> >
> >         lrel->natts = natt;
> > -
> > +       *remotegenlist_res = remotegenlist;
>
> I have fixed all the given comments. The attached v40-0001 patch
> contains the required changes.

Few comments:
1) Add a test case to ensure that an error is properly raised for the
issue reported by Swada-san in [1]:
create table t (a int not null, b int generated always as (a + 1)
stored not null);
create unique index t_idx on t (b);
alter table t replica identity using index t_idx;
insert into t values (1);
update t set a = 100 where a = 1;

2) The existing comments only reference the column list, so we should
revise them to include an important point about generated columns. If
generated columns are set to false, it may result in some columns not
being replicated:
+               if (!isnull)
+               {
+                       /* With REPLICA IDENTITY FULL, no column list
is allowed. */
+                       if (relation->rd_rel->relreplident ==
REPLICA_IDENTITY_FULL)
+                               result = true;
+
+                       /* Transform the column list datum to a bitmapset. */
+                       columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+               }
+               else
+               {
+                       TupleDesc       desc = RelationGetDescr(relation);
+                       int                     nliveatts = 0;
+
+                       for (int i = 0; i < desc->natts; i++)
+                       {
+                               Form_pg_attribute att = TupleDescAttr(desc, i);
+
+                               /* Skip if the attribute is dropped or
generated */
+                               if (att->attisdropped)
+                                       continue;
+
+                               nliveatts++;
+
+                               if (att->attgenerated)
+                                       continue;
+
+                               columns = bms_add_member(columns, i + 1);
+                       }

3) Now that we are sending generated columns from the publisher, the
comment in the tuples_equal function is no longer accurate. We need to
update the comment to reflect the new behavior of the function.
Specifically, it should clarify how generated columns are considered
in the equality check:
/*
 * Compare the tuples in the slots by checking if they have equal values.
 */
static bool
tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
TypeCacheEntry **eq)
{
....

/*
* Ignore dropped and generated columns as the publisher doesn't send
* those
*/
if (att->attisdropped || att->attgenerated)
continue;

4) This change is not required:
@@ -1015,7 +1110,6 @@ fetch_remote_table_info(char *nspname, char *relname,
        ExecDropSingleTupleTableSlot(slot);

        lrel->natts = natt;
-
        walrcv_clear_result(res);

[1] - https://www.postgresql.org/message-id/CAD21AoB%3DDBVDNCGBja%2BsDa2-w9tsM7_E%3DZgyw2qYMR1R0FwDsg%40mail.gmail.com

Regards,
Vignesh

  198. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-21T11:19:39Z

    On Fri, Oct 18, 2024 at 5:42 PM Shubham Khanna
<khannashubham1197@gmail.com> wrote:
> > >
> > > I have fixed all the given comments. The attached patches contain the
> > > required changes.

Review comments:
===============
1.
>
B. when generated columns are not published

* Publisher not-generated column => subscriber not-generated column:
  This is just normal logical replication (not changed by this patch).

* Publisher not-generated column => subscriber generated column:
  This will give ERROR.
>

Is the second behavior introduced by the patch? If so, why?

2.
@@ -1213,7 +1207,10 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
{
...
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated && !pub->pubgencols)
  continue;

It is better to combine the above conditions and write a comment on it.

3.
@@ -368,18 +379,50 @@ pub_collist_contains_invalid_column(Oid pubid,
Relation relation, List *ancestor
  Anum_pg_publication_rel_prattrs,
  &isnull);

- if (!isnull)
+ if (!isnull || !pubgencols)
  {
  int x;
  Bitmapset  *idattrs;
  Bitmapset  *columns = NULL;

- /* With REPLICA IDENTITY FULL, no column list is allowed. */
- if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- result = true;
+ if (!isnull)
+ {
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ /* Transform the column list datum to a bitmapset. */
+ columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+ }
+ else
+ {
+ TupleDesc desc = RelationGetDescr(relation);
+ int nliveatts = 0;
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped or generated */
+ if (att->attisdropped)
+ continue;
+
+ nliveatts++;
+
+ if (att->attgenerated)
+ continue;
+
+ columns = bms_add_member(columns, i + 1);
+ }

- /* Transform the column list datum to a bitmapset. */
- columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+ /* Return if all columns of the table will be replicated */
+ if (bms_num_members(columns) == nliveatts)
+ {
+ bms_free(columns);
+ ReleaseSysCache(tuple);
+ return false;
+ }

Won't this lead to traversing the entire column list for default cases
where publish_generated_columns would be false which could hurt the
update/delete's performance? Irrespective of that, it is better to
write some comments to explain this logic.

4. Some minimum parts of 0002 like the changes in
/doc/src/sgml/ref/create_publication.sgml should be part of 0001
patch. We can always add examples or more details in the docs as a
later patch.

-- 
With Regards,
Amit Kapila.

  199. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-22T01:02:58Z

    Hi SHubham, Here are my review comments for v40-0001 (code)

Please don't post a blanket response of "I have fixed all the
comments" response to this. Sometimes things get missed. Instead,
please reply done/not-done/whatever individually, so I can track the
changes properly.

======
src/backend/catalog/pg_publication.c

1.
- if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("cannot use generated column \"%s\" in publication column list",
-    colname));
-

This function no longer rejects generated columns from a publication
column list. So, now the function comment is wrong because it still
says "Checks for and raises an ERROR for any; unknown columns, system
columns, duplicate columns or generated columns."

NOTE: This was fixed already in v39-0001, but then broken again in
v40-0001 (???)

nit - also remove that semicolon (;) from the function comment.

======
src/backend/commands/publicationcmds.c

2.
- if (!isnull)
+ if (!isnull || !pubgencols)
  {
  int x;
  Bitmapset  *idattrs;
  Bitmapset  *columns = NULL;

- /* With REPLICA IDENTITY FULL, no column list is allowed. */
- if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- result = true;
+ if (!isnull)
+ {
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ /* Transform the column list datum to a bitmapset. */
+ columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+ }
+ else
+ {
+ TupleDesc desc = RelationGetDescr(relation);
+ int nliveatts = 0;
+
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ /* Skip if the attribute is dropped or generated */
+ if (att->attisdropped)
+ continue;
+
+ nliveatts++;
+
+ if (att->attgenerated)
+ continue;
+
+ columns = bms_add_member(columns, i + 1);
+ }

- /* Transform the column list datum to a bitmapset. */
- columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+ /* Return if all columns of the table will be replicated */
+ if (bms_num_members(columns) == nliveatts)
+ {
+ bms_free(columns);
+ ReleaseSysCache(tuple);
+ return false;
+ }
+ }

2a.
AFAIK this code was written to deal with Sawada-San's comment about
UPDATE/DELETE [1], but the logic is not very clear. It needs more
comments on the "else" part, the generated columns and how the new
logic solves Sawada-San's problem.

~

2b.
This function is now dealing both with 'publish_via_root' as well as
'publish_generated_columns'. I am suspicious that you may be handling
each of these parameters independently (i.e.. there is some early
return "Return if all columns of the table will be replicated") but
there might be some scenarios where a *combination* of pubviaroot and
gencols is not working as expected.

For example, there is a whole comment later about this: "If pubviaroot
is true, we are validating the column list of the parent table, but
the bitmap contains the replica identity information of the child
table." which I am not sure is being dealt with correctly. IMO there
need to be more test cases added for these tricky combination
scenarios to prove the code is good.

~

2c.
I expected to see some more REPLICA IDENTITY tests to reproduce the
problem scenario that Sawada-San reported. Where are those?

======
src/backend/replication/logical/tablesync.c

3. make_copy_attnamelist

Patch v38-0001 used to have a COPY error:

+ /*
+ * Check if the subscription table generated column has same name
+ * as a non-generated column in the corresponding publication
+ * table.
+ */
+ if (!remotegenlist[remote_attnum])
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical replication target relation \"%s.%s\" has a
generated column \"%s\" "
+ "but corresponding column on source relation is not a generated column",
+ rel->remoterel.nspname, rel->remoterel.relname, NameStr(attr->attname))));
+

My most recent comment about this code was something along the lines
of "Make this detailed useful error message common if possible".

But, in v39 all this ERROR was removed (and it is still missing in
v40). Why? I did not see any other review comments asking for this to
be removed. AFAIK this was a correct error message for
not_generated->generated. Won't the current code now just silently
ignore/skip this, which is contrary behaviour to what normal
replication would do?

The recently deleted combo TAP tests could have detected this.

~~~

fetch_remote_table_info

4.
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
There was a new 'server_version' variable added, so why not use it
here? In previous versions, we used to do so, but now since v39, the
code has reverted yet again. (???).

~~~

5.
  appendStringInfo(&cmd,
  "SELECT DISTINCT"
- "  (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
- "   THEN NULL ELSE gpt.attrs END)"
+ "  (gpt.attrs)"
  "  FROM pg_publication p,"
  "  LATERAL pg_get_publication_tables(p.pubname) gpt,"
  "  pg_class c"

Is this change OK? This fragment is guarded only by server_version >=
150000 (not 180000) so I was unsure. Can you explain why you made this
change, and why it is correct even for versions older than 180000?

~~~

6.
  lrel->natts = natt;
-
  walrcv_clear_result(res);
nit - this whitespace change has nothing to do with this patch. Remove it.

~~~

copy_table


7.
+ * We also need to use this same COPY (SELECT ...) syntax when
+ * 'publish_generated_columns' is specified as true and the remote
+ * table has generated columns, because copy of generated columns is
+ * not supported by the normal COPY.

That mention about "when 'publish_generated_columns' is specified" is
not strictly true anymore, because the publication column list
overrides this parameter. It may be better to word this like below:

SUGGESTION
We also need to use this same COPY (SELECT ...) syntax when generated
columns are published, because copying generated columns is not
supported by the normal COPY.

======
src/backend/utils/cache/relcache.c

8.
  /*
  * Check if all columns are part of the REPLICA IDENTITY index or not.
  *
- * If the publication is FOR ALL TABLES then it means the table has no
- * column list and we can skip the validation.
+ * If the publication is FOR ALL TABLES and publication includes
+ * generated columns then it means that all the table will replicate
+ * all columns and we can skip the validation.
  */

/table/tables/

Also, I think this can be re-worded to name the
'publish_generated_columns' parameter which might be more clear.

SUGGESTION:
If the publication is FOR ALL TABLES then column lists are not
possible. In this case, if 'publish_generated_columns' is true then
all table columns will be replicated, so the validation can be
skipped.

======
src/include/replication/logicalproto.h

9.
  Bitmapset  *attkeys; /* Bitmap of key columns */
+ bool    *remotegenlist; /* Array to store whether each column is
+ * generated */
 } LogicalRepRelation;

Since this is just a palloc'ed array same as the other fields like
'attnames' and 'atttyps', maybe it should be named and commented more
similarly to those? E.g more like:

bool *attremotegen; /* remote column is generated? */

======
src/test/regress/sql/publication.sql

10.
+-- Remove generate columns from column list, when
'publish_generated_columns'=false
+ALTER PUBLICATION pub2 SET TABLE gencols(a);
+\dRp+ pub2

typo /generate/generated/

======
src/test/subscription/t/100_bugs.pl

11.
- CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5
* a) STORED, c int);
+ CREATE TABLE generated_cols (a int, b_gen int, c int);

Hmm. But, should we be modifying tests in the "bugs" test file for a
new feature? If it is really necessary, then at least update the
comment to explain why.

~~~

12.
- CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5
* a) STORED, c int);
+ CREATE TABLE generated_cols (a int, b_gen int, c int);

Now you've ended up with a table called 'generated_cols' which has no
generated cols in it at all. Isn't this contrary to why this test case
existed in the first place? In other words, it feels a bit like a
hack. Perhaps you could have left all the tables as-is but just say
'publish_generated_columns=false'. But then that should be the default
parameter value anyhow, so TBH it is not clear to me why 100_bugs.pl
needed any changes at all. Can you give the reason?

======

PSA the nitpicks attachment which implements some of the cosmetic
suggestions from above.

======

[1] https://www.postgresql.org/message-id/CAD21AoB%3DDBVDNCGBja%2BsDa2-w9tsM7_E%3DZgyw2qYMR1R0FwDsg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia.

  200. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-22T07:28:18Z

    Hi Shubham, here are some comments for v40-0003 (TAP) patch.

======
Combo tests

1.
+# =============================================================================
+# The following test cases exercise logical replication for the combinations
+# where there is a generated column on one or both sides of pub/sub:
+# - generated -> normal

Saying "where there is a generated column on one or both sides of the
pub/sub" was a valid comment back when all possible combinations were
tested. Now, if you are only going to test one case ("generated ->
normal") then that comment is plain wrong.

2.
Why have you removed "copy_data=true" from some comments, but not consistently?

======
DROP EXPRESSION test

3.
(from v39-0003)

+# A "normal -> generated" replication fails, reporting an error that the
+# subscriber side column is missing.

In v40-003 this comment was broken. In v39-0003 the above comment
mentioned about failing, but in v40-0003 that comment was removed. The
v39 comment was better, because that is the whole point of this test
-- that normal->generated would fail *unless* the DROP EXPRESSION
worked correctly and changed it to a normal->normal test.

Previously we had already "proved" normal->generated failed as
expected, but since you've removed all those combo tests, now all we
have is this comment to explain it.

~~~

4.
+# Insert data to verify replication.
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_alter VALUES (1,1), (2,2), (3,3)");

Remove unnecessary whitespace.

======
First "Testcase"

5.
+# --------------------------------------------------
+# Testcase: Publisher replicates the column list data including generated
+# columns even though publish_generated_columns option is false.
+# --------------------------------------------------
+

That comment needs updating to reflect what this test case is *really*
doing. E.g., now you are testing tables without column lists as well
as tables with column lists.

~~~

6.
+# Create table and publications.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab_gen_to_gen (a int, gen1 int GENERATED ALWAYS AS (a
* 2) STORED);
+ CREATE TABLE tab_gen_to_gen2 (a int, gen1 int GENERATED ALWAYS AS (a
* 2) STORED);
+ CREATE PUBLICATION pub1 FOR table tab_gen_to_gen,
tab_gen_to_gen2(gen1) WITH (publish_generated_columns=false);
+));
+

Calling these tables 'gen_to_gen' makes no sense. Replication to a
subscriber-side generated column does not work, and anyway, your
subscriber-side tables do not have generated columns in them. Please
be very careful with the table names -- misleading names cause a lot
of unnecessary confusion.

~~~

7.
+# Insert values into tables.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ INSERT INTO tab_gen_to_gen (a) VALUES (1), (1);
+ INSERT INTO tab_gen_to_gen2 (a) VALUES (1), (1);
+));
+

It seems unusual to insert initial values 1,1, and then later insert
values 2,3. Wouldn't values 1,2, and then 3,4 be better?

~~~

8.
+# Create table and subscription with copy_data=true.

Sometimes you say "copy_data-true" in the comments and other times you
do not. I could not make sense of why the differences -- I guess
perhaps this was supposed to be a global replacement but some got
missed by mistake (???)

~~~

9.
Remove multiple unnecessary whitespace, in various places in this test.

~~~

10.
+# Initial sync test when publish_generated_columns=false.

It would be better if this test comments  (and others like this one)
would also say more about what is the result expected and why.

~~~

11.
+# Incremental replication test when publish_generated_columns=false.
+# Verify that column 'b' is not replicated.
+$node_publisher->wait_for_catchup('sub1');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab_gen_to_gen ORDER BY a");
+is( $result, qq(1|
+1|
+2|
+3|),
+ 'tab_gen_to_gen incremental replication, when publish_generated_columns=false'
+);

There is not a column 'b' (I think now you called it "gen1") so the
comment is bogus. Please take care that comments are kept up-to-date
whenever you change the code.

~~~

12.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab_gen_to_gen2 ORDER BY a");
+is( $result, qq(|2
+|2
+|4
+|6),
+ 'tab_gen_to_gen2 incremental replication, when
publish_generated_columns=false'
+);
+

Here you should have a comment to explain that the expected result was
generated column 'gen1' should be replicated because it was specified
in a column list, so that overrides the
publish_generated_columns=false.

======
Second "Testcase"

13.
All the above comments also apply to this second test case:

e.g. the "Testcase" comment needed updating.
e.g. table names like 'gen_to_gen' make no sense.
e.g. the initial data values 1,1 seem strange to me.
e.g. some comments have spurious "copy_data = true" and some do not.
e.g. unnecessary blank lines.
e.g. not enough comments describing what are the expected results and why.
e.g. multiple bogus mentions of a column 'b', when there is no such column name

======
FYI - The attached nitpicks just show some of the blank lines I
removed; nothing else.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  201. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-22T10:50:00Z

    On Wed, Oct 9, 2024 at 10:19 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Regarding the 0001 patch, it seems to me that UPDATE and DELETE are
> allowed on the table even if its replica identity is set to generated
> columns that are not published. For example, consider the following
> scenario:
>
> create table t (a int not null, b int generated always as (a + 1)
> stored not null);
> create unique index t_idx on t (b);
> alter table t replica identity using index t_idx;
> create publication pub for table t with (publish_generated_columns = false);
> insert into t values (1);
> update t set a = 100 where a = 1;
>
> The publication pub doesn't include the generated column 'b' which is
> the replica identity of the table 't'. Therefore, the update message
> generated by the last UPDATE would have NULL for the column 'b'. I
> think we should not allow UPDATE and DELETE on such a table.
>

I see the same behavior even without a patch on the HEAD. See the
following example executed on HEAD:

postgres=# create table t (a int not null, b int generated always as (a + 1)
postgres(# stored not null);
CREATE TABLE
postgres=# create unique index t_idx on t (b);
CREATE INDEX
postgres=# alter table t replica identity using index t_idx;
ALTER TABLE
postgres=# create publication pub for table t;
CREATE PUBLICATION
postgres=# insert into t values (1);
INSERT 0 1
postgres=# update t set a = 100 where a = 1;
UPDATE 1

So, the update is allowed even when we don't publish generated
columns, if so, why do we need to handle it in this patch when the
user gave publish_generated_columns=false?

Also, on the subscriber side, I see the ERROR: "publisher did not send
replica identity column expected by the logical replication target
relation "public.t"".

Considering this, I feel if find this behavior buggy then we should
fix this separately rather than part of this patch. What do you think?

-- 
With Regards,
Amit Kapila.

  202. Re: Pgoutput not capturing the generated columns

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-10-22T16:11:32Z

    On Tue, Oct 22, 2024 at 3:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 9, 2024 at 10:19 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Regarding the 0001 patch, it seems to me that UPDATE and DELETE are
> > allowed on the table even if its replica identity is set to generated
> > columns that are not published. For example, consider the following
> > scenario:
> >
> > create table t (a int not null, b int generated always as (a + 1)
> > stored not null);
> > create unique index t_idx on t (b);
> > alter table t replica identity using index t_idx;
> > create publication pub for table t with (publish_generated_columns = false);
> > insert into t values (1);
> > update t set a = 100 where a = 1;
> >
> > The publication pub doesn't include the generated column 'b' which is
> > the replica identity of the table 't'. Therefore, the update message
> > generated by the last UPDATE would have NULL for the column 'b'. I
> > think we should not allow UPDATE and DELETE on such a table.
> >
>
> I see the same behavior even without a patch on the HEAD. See the
> following example executed on HEAD:
>
> postgres=# create table t (a int not null, b int generated always as (a + 1)
> postgres(# stored not null);
> CREATE TABLE
> postgres=# create unique index t_idx on t (b);
> CREATE INDEX
> postgres=# alter table t replica identity using index t_idx;
> ALTER TABLE
> postgres=# create publication pub for table t;
> CREATE PUBLICATION
> postgres=# insert into t values (1);
> INSERT 0 1
> postgres=# update t set a = 100 where a = 1;
> UPDATE 1
>
> So, the update is allowed even when we don't publish generated
> columns, if so, why do we need to handle it in this patch when the
> user gave publish_generated_columns=false?
>
> Also, on the subscriber side, I see the ERROR: "publisher did not send
> replica identity column expected by the logical replication target
> relation "public.t"".

Good point.

> Considering this, I feel if find this behavior buggy then we should
> fix this separately rather than part of this patch. What do you think?

Agreed. It's better to fix it separately.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  203. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-23T05:00:05Z

    Recently (~ version v39/v40) some changes to 'get_publications_str'
calls got removed from this patchset because it was decided it was
really a separate problem, unrelated to this generated columns
feature.

FYI - I've started a new thread "Refactor to use common function
'get_publications_str'" [1] to address it separately.

======
[1] https://www.postgresql.org/message-id/CAHut+PtJMk4bKXqtpvqVy9ckknCgK9P6=FeG8zHF=6+Em_Snpw@mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  204. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-23T06:21:17Z

    On Tue, Oct 22, 2024 at 9:42 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Oct 22, 2024 at 3:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> > Considering this, I feel if find this behavior buggy then we should
> > fix this separately rather than part of this patch. What do you think?
>
> Agreed. It's better to fix it separately.
>

Thanks. One more thing that I didn't like about the patch is that it
used column_list to address the "publish_generated_columns = false"
case such that we build column_list without generated columns for the
same. The first problem is that it will add overhead to always probe
column_list during proto.c calls (for example during
logicalrep_write_attrs()), then it makes the column_list code complex
especially the handling in pgoutput_column_list_init(), and finally
this appears to be a misuse of column_list.

So, I suggest remembering this information in RelationSyncEntry and
then using it at the required places. We discussed above that
contradictory values of "publish_generated_columns" across
publications for the same relations are not accepted, so we can detect
that during get_rel_sync_entry() and give an ERROR for the same.

Additional comment on the 0003 patch
+# =============================================================================
+# Misc test.
+#
+# A "normal -> generated" replication.
+#
+# In this test case we use DROP EXPRESSION to change the subscriber generated
+# column into a normal column, then verify replication works ok.
+# =============================================================================

In patch 0003, why do we have the above test? This doesn't seem to be
directly related to this patch.

-- 
With Regards,
Amit Kapila.

  205. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-23T06:55:55Z

    On Wed, Oct 23, 2024 at 5:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:

> Additional comment on the 0003 patch
> +# =============================================================================
> +# Misc test.
> +#
> +# A "normal -> generated" replication.
> +#
> +# In this test case we use DROP EXPRESSION to change the subscriber generated
> +# column into a normal column, then verify replication works ok.
> +# =============================================================================
>
> In patch 0003, why do we have the above test? This doesn't seem to be
> directly related to this patch.
>
> --

Perhaps the test should be turned around, to test this feature more directly...

e.g. Replication of table tab(a int, b int) ==>  tab(a int, b int, c int)

test_pub=# create table tab(a int, b int);

then, dynamically add a generated column "c" to the publisher table
test_pub=# alter table tab add column c int GENERATED ALWAYS AS (a + b) STORED;
test_pub=# insert into tab values (1,2);

then, verify that replication works for the newly added generated
column "c" to the existing normal column "c" at the subscriber.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  206. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-23T08:37:53Z

    On Wed, Oct 23, 2024 at 12:26 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Wed, Oct 23, 2024 at 5:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> > Additional comment on the 0003 patch
> > +# =============================================================================
> > +# Misc test.
> > +#
> > +# A "normal -> generated" replication.
> > +#
> > +# In this test case we use DROP EXPRESSION to change the subscriber generated
> > +# column into a normal column, then verify replication works ok.
> > +# =============================================================================
> >
> > In patch 0003, why do we have the above test? This doesn't seem to be
> > directly related to this patch.
> >
> > --
>
> Perhaps the test should be turned around, to test this feature more directly...
>
> e.g. Replication of table tab(a int, b int) ==>  tab(a int, b int, c int)
>
> test_pub=# create table tab(a int, b int);
>
> then, dynamically add a generated column "c" to the publisher table
> test_pub=# alter table tab add column c int GENERATED ALWAYS AS (a + b) STORED;
> test_pub=# insert into tab values (1,2);
>
> then, verify that replication works for the newly added generated
> column "c" to the existing normal column "c" at the subscriber.
>

This is testing whether the invalidation mechanism works for this case
which I see no reason to not work as this patch hasn't changed
anything in this regard. We should verify this but not sure it will
add much value in keeping this in regression tests.

-- 
With Regards,
Amit Kapila.

  207. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-24T06:45:46Z

    On Wed, 23 Oct 2024 at 11:51, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Oct 22, 2024 at 9:42 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Oct 22, 2024 at 3:50 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> >
> > > Considering this, I feel if find this behavior buggy then we should
> > > fix this separately rather than part of this patch. What do you think?
> >
> > Agreed. It's better to fix it separately.
> >
>
> Thanks. One more thing that I didn't like about the patch is that it
> used column_list to address the "publish_generated_columns = false"
> case such that we build column_list without generated columns for the
> same. The first problem is that it will add overhead to always probe
> column_list during proto.c calls (for example during
> logicalrep_write_attrs()), then it makes the column_list code complex
> especially the handling in pgoutput_column_list_init(), and finally
> this appears to be a misuse of column_list.

I simplified the process into three steps: a) Iterate through the
column list of publications and raise an error if there are any
discrepancies in the column list. b) Examine the non-column list of
publications to identify any conflicting options for
publish_generated_columns, and set this option accordingly. c)
Finally, verify that the columns in the column list align with the
table's columns based on the publish_generated_columns option.

> So, I suggest remembering this information in RelationSyncEntry and
> then using it at the required places. We discussed above that
> contradictory values of "publish_generated_columns" across
> publications for the same relations are not accepted, so we can detect
> that during get_rel_sync_entry() and give an ERROR for the same.

Resolved the issue by adding a new variable, pubgencols, to determine
whether the generated columns need to be published. This variable will
later be utilized in logicalrep_write_tuple and logicalrep_write_attrs
to replicate the generated columns to the subscriber if required.

> Additional comment on the 0003 patch
> +# =============================================================================
> +# Misc test.
> +#
> +# A "normal -> generated" replication.
> +#
> +# In this test case we use DROP EXPRESSION to change the subscriber generated
> +# column into a normal column, then verify replication works ok.
> +# =============================================================================
>
> In patch 0003, why do we have the above test? This doesn't seem to be
> directly related to this patch.

Removed this.

The attached v41 version patch has the changes for the same.

Regards,
Vignesh

  208. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-24T06:47:46Z

    On Wed, Oct 23, 2024 at 11:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Thanks. One more thing that I didn't like about the patch is that it
> used column_list to address the "publish_generated_columns = false"
> case such that we build column_list without generated columns for the
> same. The first problem is that it will add overhead to always probe
> column_list during proto.c calls (for example during
> logicalrep_write_attrs()), then it makes the column_list code complex
> especially the handling in pgoutput_column_list_init(), and finally
> this appears to be a misuse of column_list.
>
> So, I suggest remembering this information in RelationSyncEntry and
> then using it at the required places. We discussed above that
> contradictory values of "publish_generated_columns" across
> publications for the same relations are not accepted, so we can detect
> that during get_rel_sync_entry() and give an ERROR for the same.
>

The changes in tablesync look complicated and I am not sure whether it
handles the conflicting publish_generated_columns option correctly. I
have few thoughts for the same.
* The handling of contradictory options in multiple publications needs
to be the same as for column lists. I think it is handled (a) during
subscription creation, (b) during copy in fetch_remote_table_info(),
and (c) during replication. See Peter's email
(https://www.postgresql.org/message-id/CAHut%2BPs985rc95cB2x5yMF56p6Lf192AmCJOpAtK_%2BC5YGUF2A%40mail.gmail.com)
to understand why this new option has to be handled in the same way as
the column list.

* While fetching column list via pg_get_publication_tables(), we
should detect contradictory publish_generated_columns options similar
to column lists, and then after we get publish_generated_columns as
return value, we can even use that while fetching attribute
information.

A few additional comments:
1.
- /* Regular table with no row filter */
- if (lrel.relkind == RELKIND_RELATION && qual == NIL)
+ /*
+ * Check if the remote table has any generated columns that should be
+ * copied.
+ */
+ for (int i = 0; i < relmapentry->remoterel.natts; i++)
+ {
+ if (lrel.attremotegen[i])
+ {
+ gencol_copy_needed = true;
+ break;
+ }
+ }

Can't we get this information from fetch_remote_table_info() instead
of traversing the entire column list again?

2.
@@ -1015,7 +1110,6 @@ fetch_remote_table_info(char *nspname, char *relname,
  ExecDropSingleTupleTableSlot(slot);

  lrel->natts = natt;
-
  walrcv_clear_result(res);

Spurious line removal.

3. Why do we have to specifically exclude generated columns of a
subscriber-side table in make_copy_attnamelist()? Can't we rely on
fetch_remote_table_info() and logicalrep_rel_open() that the final
remote attrlist will contain the generated column only if the
subscriber doesn't have a generated column otherwise it would have
given an error in logicalrep_rel_open()?

-- 
With Regards,
Amit Kapila.

  209. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-24T11:14:14Z

    On Thu, Oct 24, 2024 at 12:15 PM vignesh C <vignesh21@gmail.com> wrote:
>
> The attached v41 version patch has the changes for the same.
>

Please find comments for the new version as follows:
1.
+      Generated columns may be skipped during logical replication
according to the
+      <command>CREATE PUBLICATION</command> option
+      <link linkend="sql-createpublication-params-with-publish-generated-columns">
+      <literal>include_generated_columns</literal></link>.

The above statement doesn't sound to be clear. Can we change it to:
"Generated columns are allowed to be replicated during logical
replication according to the <command>CREATE PUBLICATION</command>
option .."?

2.
 static void publication_invalidation_cb(Datum arg, int cacheid,
  uint32 hashvalue);
-static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx,
- Bitmapset *columns);
 static void send_repl_origin(LogicalDecodingContext *ctx,
...
...
 static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data,
  Relation relation);
+static void send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx,
+ RelationSyncEntry *relentry);

Why the declaration of this function is changed?

3.
+ /*
+ * Skip publishing generated columns if the option is not specified or
+ * if they are not included in the column list.
+ */
+ if (att->attgenerated && !relentry->pubgencols && !columns)

In the comment above, shouldn't "specified or" be "specified and"?

4.
+pgoutput_pubgencol_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)

{
...
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+
+ /* No need to check column list publications */
+ if (is_column_list_publication(pub, entry->publish_as_relid))

Are we ignoring column_list publications because for such publications
the value of column_list prevails and we ignore
'publish_generated_columns' value? If so, it is not clear from the
comments.

5.
  /* Initialize the column list */
  pgoutput_column_list_init(data, rel_publications, entry);
+
+ /* Initialize publish generated columns value */
+ pgoutput_pubgencol_init(data, rel_publications, entry);
+
+ /*
+ * Check if there is conflict with the columns selected for the
+ * publication.
+ */
+ check_conflicting_columns(rel_publications, entry);
  }

It looks odd to check conflicting column lists among publications
twice once in pgoutput_column_list_init() and then in
check_conflicting_columns(). Can we merge those?

-- 
With Regards,
Amit Kapila.

  210. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-24T15:17:37Z

    On Thu, 24 Oct 2024 at 12:17, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 23, 2024 at 11:51 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Thanks. One more thing that I didn't like about the patch is that it
> > used column_list to address the "publish_generated_columns = false"
> > case such that we build column_list without generated columns for the
> > same. The first problem is that it will add overhead to always probe
> > column_list during proto.c calls (for example during
> > logicalrep_write_attrs()), then it makes the column_list code complex
> > especially the handling in pgoutput_column_list_init(), and finally
> > this appears to be a misuse of column_list.
> >
> > So, I suggest remembering this information in RelationSyncEntry and
> > then using it at the required places. We discussed above that
> > contradictory values of "publish_generated_columns" across
> > publications for the same relations are not accepted, so we can detect
> > that during get_rel_sync_entry() and give an ERROR for the same.
> >
>
> The changes in tablesync look complicated and I am not sure whether it
> handles the conflicting publish_generated_columns option correctly. I
> have few thoughts for the same.
> * The handling of contradictory options in multiple publications needs
> to be the same as for column lists. I think it is handled (a) during
> subscription creation, (b) during copy in fetch_remote_table_info(),
> and (c) during replication. See Peter's email
> (https://www.postgresql.org/message-id/CAHut%2BPs985rc95cB2x5yMF56p6Lf192AmCJOpAtK_%2BC5YGUF2A%40mail.gmail.com)
> to understand why this new option has to be handled in the same way as
> the column list.
>
> * While fetching column list via pg_get_publication_tables(), we
> should detect contradictory publish_generated_columns options similar
> to column lists, and then after we get publish_generated_columns as
> return value, we can even use that while fetching attribute
> information.

Modified it to detect conflicting column lists for publications,
allowing the detection of publish_generated_columns conflicts using
the same code currently used for column list detection.

> A few additional comments:
> 1.
> - /* Regular table with no row filter */
> - if (lrel.relkind == RELKIND_RELATION && qual == NIL)
> + /*
> + * Check if the remote table has any generated columns that should be
> + * copied.
> + */
> + for (int i = 0; i < relmapentry->remoterel.natts; i++)
> + {
> + if (lrel.attremotegen[i])
> + {
> + gencol_copy_needed = true;
> + break;
> + }
> + }
>
> Can't we get this information from fetch_remote_table_info() instead
> of traversing the entire column list again?

Yes, modified

> 2.
> @@ -1015,7 +1110,6 @@ fetch_remote_table_info(char *nspname, char *relname,
>   ExecDropSingleTupleTableSlot(slot);
>
>   lrel->natts = natt;
> -
>   walrcv_clear_result(res);
>
> Spurious line removal.

Fixed this

> 3. Why do we have to specifically exclude generated columns of a
> subscriber-side table in make_copy_attnamelist()? Can't we rely on
> fetch_remote_table_info() and logicalrep_rel_open() that the final
> remote attrlist will contain the generated column only if the
> subscriber doesn't have a generated column otherwise it would have
> given an error in logicalrep_rel_open()?

Yes, it can be used. The updated v42 version of patch has the changes
for the same.

Regards,
Vignesh

  211. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-24T15:20:01Z

    On Thu, 24 Oct 2024 at 16:44, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 24, 2024 at 12:15 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v41 version patch has the changes for the same.
> >
>
> Please find comments for the new version as follows:
> 1.
> +      Generated columns may be skipped during logical replication
> according to the
> +      <command>CREATE PUBLICATION</command> option
> +      <link linkend="sql-createpublication-params-with-publish-generated-columns">
> +      <literal>include_generated_columns</literal></link>.
>
> The above statement doesn't sound to be clear. Can we change it to:
> "Generated columns are allowed to be replicated during logical
> replication according to the <command>CREATE PUBLICATION</command>
> option .."?

Modified

> 2.
>  static void publication_invalidation_cb(Datum arg, int cacheid,
>   uint32 hashvalue);
> -static void send_relation_and_attrs(Relation relation, TransactionId xid,
> - LogicalDecodingContext *ctx,
> - Bitmapset *columns);
>  static void send_repl_origin(LogicalDecodingContext *ctx,
> ...
> ...
>  static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data,
>   Relation relation);
> +static void send_relation_and_attrs(Relation relation, TransactionId xid,
> + LogicalDecodingContext *ctx,
> + RelationSyncEntry *relentry);
>
> Why the declaration of this function is changed?

Two changes were made: a) The function declaration need to be moved
down as the RelationSyncEntry structure is defined below. b) Bitmapset
was replaced with RelationSyncEntry to give send_relation_and_attrs
access to RelationSyncEntry.pubgencols and RelationSyncEntry.columns.
Instead of adding a new parameter to the function, RelationSyncEntry
was utilized, as it contains both pubgencols and columns members.

> 3.
> + /*
> + * Skip publishing generated columns if the option is not specified or
> + * if they are not included in the column list.
> + */
> + if (att->attgenerated && !relentry->pubgencols && !columns)
>
> In the comment above, shouldn't "specified or" be "specified and"?

Modified

> 4.
> +pgoutput_pubgencol_init(PGOutputData *data, List *publications,
> + RelationSyncEntry *entry)
>
> {
> ...
> + foreach(lc, publications)
> + {
> + Publication *pub = lfirst(lc);
> +
> + /* No need to check column list publications */
> + if (is_column_list_publication(pub, entry->publish_as_relid))
>
> Are we ignoring column_list publications because for such publications
> the value of column_list prevails and we ignore
> 'publish_generated_columns' value? If so, it is not clear from the
> comments.

Yes column takes precedence over publish_generated_columns value, so
column list publications are skipped. Modified the comments
accordingly.

> 5.
>   /* Initialize the column list */
>   pgoutput_column_list_init(data, rel_publications, entry);
> +
> + /* Initialize publish generated columns value */
> + pgoutput_pubgencol_init(data, rel_publications, entry);
> +
> + /*
> + * Check if there is conflict with the columns selected for the
> + * publication.
> + */
> + check_conflicting_columns(rel_publications, entry);
>   }
>
> It looks odd to check conflicting column lists among publications
> twice once in pgoutput_column_list_init() and then in
> check_conflicting_columns(). Can we merge those?

Modified it to check from pgoutput_column_list_init

The v42 version patch attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm2wFZRzSJLcNi_uMZcSUWuZ8%2Bkktc0n3Nfw9Fdti9WbVA%40mail.gmail.com

Regards,
Vignesh

  212. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-25T06:37:31Z

    On Thu, Oct 24, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
>
> The v42 version patch attached at [1] has the changes for the same.
>

Some more comments:
1.
@@ -1017,7 +1089,31 @@ pgoutput_column_list_init(PGOutputData *data,
List *publications,
 {
  ListCell   *lc;
  bool first = true;
+ Bitmapset  *relcols = NULL;
  Relation relation = RelationIdGetRelation(entry->publish_as_relid);
+ TupleDesc desc = RelationGetDescr(relation);
+ MemoryContext oldcxt = NULL;
+ bool collistpubexist = false;
+
+ pgoutput_ensure_entry_cxt(data, entry);
+
+ oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
+
+ /*
+ * Prepare the columns that will be published for FOR ALL TABLES and
+ * FOR TABLES IN SCHEMA publication.
+ */
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || (att->attgenerated && !entry->pubgencols))
+ continue;
+
+ relcols = bms_add_member(relcols, att->attnum);
+ }
+
+ MemoryContextSwitchTo(oldcxt);

This code is unnecessary for cases when the table's publication has a
column list. So, I suggest to form this list only when required. Also,
have an assertion that pubgencols value for entry and publication
matches.

2.
@@ -1115,10 +1186,17 @@ pgoutput_column_list_init(PGOutputData *data,
List *publications,
  ereport(ERROR,
  errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
  errmsg("cannot use different column lists for table \"%s.%s\" in
different publications",
-    get_namespace_name(RelationGetNamespace(relation)),
-    RelationGetRelationName(relation)));
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));

Is there a reason to make the above change? It appears to be a spurious change.

3.
+ /* Check if there is any generated column present */
+ for (int i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+ if (att->attgenerated)

Add one empty line between the above two lines.

4.
+ else if (entry->pubgencols != pub->pubgencols)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot use different values of publish_generated_columns for
table \"%s.%s\" in different publications",
+ get_namespace_name(RelationGetNamespace(relation)),
+ RelationGetRelationName(relation)));

The last two lines are not aligned.

-- 
With Regards,
Amit Kapila.

  213. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-25T10:24:05Z

    On Fri, Oct 25, 2024 at 12:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 24, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The v42 version patch attached at [1] has the changes for the same.
> >
>
> Some more comments:
>

1.
+pgoutput_pubgencol_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)

Can we name it as check_and_init_gencol? I don't know if it is a good
idea to append a prefix pgoutput for local functions. It is primarily
used for exposed functions from pgoutput.c. I see that in a few cases
we do that for local functions as well but that is not a norm.

A related point:
+ /* Initialize publish generated columns value */
+ pgoutput_pubgencol_init(data, rel_publications, entry);

Accordingly change this comment to something like: "Check whether to
publish to generated columns.".

2.
+/*
+ * Returns true if the relation has column list associated with the
+ * publication, false if the relation has no column list associated with the
+ * publication.
+ */
+bool
+is_column_list_publication(Publication *pub, Oid relid)
...
...

How about naming the above function as has_column_list_defined()?
Also, you can write the above comment as: "Returns true if the
relation has column list associated with the publication, false
otherwise."

3.
+ /*
+ * The column list takes precedence over pubgencols, so skip checking
+ * column list publications.
+ */
+ if (is_column_list_publication(pub, entry->publish_as_relid))

Let's change this comment to: "The column list takes precedence over
publish_generated_columns option. Those will be checked later, see
pgoutput_column_list_init."

-- 
With Regards,
Amit Kapila.

  214. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-25T14:04:07Z

    On Fri, Oct 25, 2024 at 12:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 24, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The v42 version patch attached at [1] has the changes for the same.
> >
>
> Some more comments:
> 1.
> @@ -1017,7 +1089,31 @@ pgoutput_column_list_init(PGOutputData *data,
> List *publications,
>  {
>   ListCell   *lc;
>   bool first = true;
> + Bitmapset  *relcols = NULL;
>   Relation relation = RelationIdGetRelation(entry->publish_as_relid);
> + TupleDesc desc = RelationGetDescr(relation);
> + MemoryContext oldcxt = NULL;
> + bool collistpubexist = false;
> +
> + pgoutput_ensure_entry_cxt(data, entry);
> +
> + oldcxt = MemoryContextSwitchTo(entry->entry_cxt);
> +
> + /*
> + * Prepare the columns that will be published for FOR ALL TABLES and
> + * FOR TABLES IN SCHEMA publication.
> + */
> + for (int i = 0; i < desc->natts; i++)
> + {
> + Form_pg_attribute att = TupleDescAttr(desc, i);
> +
> + if (att->attisdropped || (att->attgenerated && !entry->pubgencols))
> + continue;
> +
> + relcols = bms_add_member(relcols, att->attnum);
> + }
> +
> + MemoryContextSwitchTo(oldcxt);
>
> This code is unnecessary for cases when the table's publication has a
> column list. So, I suggest to form this list only when required. Also,
> have an assertion that pubgencols value for entry and publication
> matches.
>

Modified and also included Assertion.

> 2.
> @@ -1115,10 +1186,17 @@ pgoutput_column_list_init(PGOutputData *data,
> List *publications,
>   ereport(ERROR,
>   errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>   errmsg("cannot use different column lists for table \"%s.%s\" in
> different publications",
> -    get_namespace_name(RelationGetNamespace(relation)),
> -    RelationGetRelationName(relation)));
> + get_namespace_name(RelationGetNamespace(relation)),
> + RelationGetRelationName(relation)));
>
> Is there a reason to make the above change? It appears to be a spurious change.
>

Change is not required, removed now.

> 3.
> + /* Check if there is any generated column present */
> + for (int i = 0; i < desc->natts; i++)
> + {
> + Form_pg_attribute att = TupleDescAttr(desc, i);
> + if (att->attgenerated)
>
> Add one empty line between the above two lines.
>

Added.

> 4.
> + else if (entry->pubgencols != pub->pubgencols)
> + ereport(ERROR,
> + errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> + errmsg("cannot use different values of publish_generated_columns for
> table \"%s.%s\" in different publications",
> + get_namespace_name(RelationGetNamespace(relation)),
> + RelationGetRelationName(relation)));
>
> The last two lines are not aligned.
>

Modified.

The updated v43 version of patches contain the changes for the same.

Thanks and Regards,
Shubham Khanna.

  215. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-25T14:10:56Z

    On Fri, Oct 25, 2024 at 3:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Oct 25, 2024 at 12:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Oct 24, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > The v42 version patch attached at [1] has the changes for the same.
> > >
> >
> > Some more comments:
> >
>
> 1.
> +pgoutput_pubgencol_init(PGOutputData *data, List *publications,
> + RelationSyncEntry *entry)
>
> Can we name it as check_and_init_gencol? I don't know if it is a good
> idea to append a prefix pgoutput for local functions. It is primarily
> used for exposed functions from pgoutput.c. I see that in a few cases
> we do that for local functions as well but that is not a norm.
>
> A related point:
> + /* Initialize publish generated columns value */
> + pgoutput_pubgencol_init(data, rel_publications, entry);
>
> Accordingly change this comment to something like: "Check whether to
> publish to generated columns.".
>

Fixed.

> 2.
> +/*
> + * Returns true if the relation has column list associated with the
> + * publication, false if the relation has no column list associated with the
> + * publication.
> + */
> +bool
> +is_column_list_publication(Publication *pub, Oid relid)
> ...
> ...
>
> How about naming the above function as has_column_list_defined()?
> Also, you can write the above comment as: "Returns true if the
> relation has column list associated with the publication, false
> otherwise."
>

Fixed.

> 3.
> + /*
> + * The column list takes precedence over pubgencols, so skip checking
> + * column list publications.
> + */
> + if (is_column_list_publication(pub, entry->publish_as_relid))
>
> Let's change this comment to: "The column list takes precedence over
> publish_generated_columns option. Those will be checked later, see
> pgoutput_column_list_init."
>

Fixed.

The v43 version patch attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CAHv8RjJJJRzy83tG0nB90ivYcp7sFKTU%3D_BcQ-nUZ7VbHFwceA%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  216. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-28T02:13:10Z

    Hi, here are my review comments for patch v43-0001.

======

1. Missing docs update?

The CREATE PUBLICATION docs currently says:
When a column list is specified, only the named columns are
replicated. If no column list is specified, all columns of the table
are replicated through this publication, including any columns added
later.

~

For this patch, should that be updated to say "... all columns (except
generated columns) of the table are replicated..."

======
src/backend/replication/logical/proto.c

2.
+static bool
+should_publish_column(Form_pg_attribute att, Bitmapset *columns)
+{
+ if (att->attisdropped)
+ return false;
+
+ /*
+ * Skip publishing generated columns if they are not included in the
+ * column list.
+ */
+ if (att->attgenerated && !columns)
+ return false;
+
+ if (!column_in_column_list(att->attnum, columns))
+ return false;
+
+ return true;
+}

Here, I wanted to suggest that the whole "Skip publishing generated
columns" if-part is unnecessary because the next check
(!column_in_column_list) is going to return false for the same
scenario anyhow.

But, unfortunately, the "column_in_column_list" function has some
special NULL handling logic in it; this means none of this code is
quite what it seems to be (e.g. the function name
column_in_column_list is somewhat misleading)

IMO it would be better to change the column_in_column_list signature
-- add another boolean param to say if a NULL column list is allowed
or not. That will remove any subtle behaviour and then you can remove
the "if (att->attgenerated && !columns)" part.

======
src/backend/replication/pgoutput/pgoutput.c

3. send_relation_and_attrs

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

  if (att->atttypid < FirstGenbkiObjectId)
  continue;

+ /*
+ * Skip publishing generated columns if they are not included in the
+ * column list.
+ */
+ if (att->attgenerated && !columns)
+ continue;
+
  /* Skip this attribute if it's not present in the column list */
  if (columns != NULL && !bms_is_member(att->attnum, columns))
  continue;
~

Most of that code above looks to be doing the very same thing as the
new 'should_publish_column' in proto.c. Won't it be better to expose
the other function and share the common logic?

~~~

4. pgoutput_column_list_init

- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
  continue;

+ if (att->attgenerated)
+ {
+ if (bms_is_member(att->attnum, cols))
+ gencolpresent = true;
+
+ continue;
+ }
+
  nliveatts++;
  }

  /*
- * If column list includes all the columns of the table,
- * set it to NULL.
+ * If column list includes all the columns of the table
+ * and there are no generated columns, set it to NULL.
  */
- if (bms_num_members(cols) == nliveatts)
+ if (bms_num_members(cols) == nliveatts && !gencolpresent)
  {

Something seems not quite right (or maybe redundant) with this logic.
For example, because you unconditionally 'continue' for generated
columns, then AFAICT it is just not possible for bms_num_members(cols)
== nliveatts and at the same time 'gencolpresent' to be true. So you
could've just Asserted (!gencolpresent) instead of checking it in the
condition and mentioning it in the comment.

======
src/test/regress/expected/publication.out

5.
--- error: generated column "d" can't be in list
+-- ok: generated column "d" can be in the list too
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
-ERROR:  cannot use generated column "d" in publication column list

By allowing the above to work without giving ERROR, I think you've
broken many subsequent test expected results. e.g. I don't trust these
"expected" results anymore because I didn't think these next test
errors should have been affected, right?

 -- error: system attributes "ctid" not allowed in column list
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-ERROR:  cannot use system column "ctid" in publication column list
+ERROR:  relation "testpub_tbl5" is already member of publication
"testpub_fortable"

Hmm - looks like a wrong expected result to me.

~

 -- error: duplicates not allowed in column list
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, a);
-ERROR:  duplicate column "a" in publication column list
+ERROR:  relation "testpub_tbl5" is already member of publication
"testpub_fortable"

Hmm - looks like a wrong expected result to me.

probably more like this...

======
src/test/subscription/t/031_column_list.pl

6.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE test_gen (a int PRIMARY KEY, b int);
+));
+
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE SUBSCRIPTION sub_gen CONNECTION '$publisher_connstr'
PUBLICATION pub_gen;
+));

Should combine these.

~~~

7.
+$node_publisher->wait_for_catchup('sub_gen');
+
+is( $node_subscriber->safe_psql(
+ 'postgres', "SELECT * FROM test_gen ORDER BY a"),
+ qq(1|2),
+ 'replication with generated columns in column list');
+

But, this is only testing normal replication. You should also include
some initial table data so you can test that the initial table
synchronization works too. Otherwise, I think current this patch has
no proof that the initial 'copy_data' even works at all.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  217. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-10-28T03:17:23Z

    Dear Shubham,

Thanks for updating the patch! I resumed reviewing the patch set.
Here are only cosmetic comments as my rehabilitation.

01. getPublications()

I feel we could follow the notation like getSubscriptions(), because number of
parameters became larger. How do you feel like attached?


02. fetch_remote_table_info()

```
                          "SELECT DISTINCT"
-                         "  (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
-                         "   THEN NULL ELSE gpt.attrs END)"
+                         "  (gpt.attrs)"
```

I think no need to separate lines and add bracket. How about like below?

```
						 "SELECT DISTINCT gpt.attrs"
```

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  218. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-28T05:33:55Z

    On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for patch v43-0001.
>
> ======
> src/backend/replication/logical/proto.c
>
> 2.
> +static bool
> +should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> +{
> + if (att->attisdropped)
> + return false;
> +
> + /*
> + * Skip publishing generated columns if they are not included in the
> + * column list.
> + */
> + if (att->attgenerated && !columns)
> + return false;
> +
> + if (!column_in_column_list(att->attnum, columns))
> + return false;
> +
> + return true;
> +}
>
> Here, I wanted to suggest that the whole "Skip publishing generated
> columns" if-part is unnecessary because the next check
> (!column_in_column_list) is going to return false for the same
> scenario anyhow.
>
> But, unfortunately, the "column_in_column_list" function has some
> special NULL handling logic in it; this means none of this code is
> quite what it seems to be (e.g. the function name
> column_in_column_list is somewhat misleading)
>
> IMO it would be better to change the column_in_column_list signature
> -- add another boolean param to say if a NULL column list is allowed
> or not. That will remove any subtle behaviour and then you can remove
> the "if (att->attgenerated && !columns)" part.
>

The NULL column list still means all columns, so changing the behavior
as you are proposing doesn't make sense and would make the code
difficult to understand.

>
> 4. pgoutput_column_list_init
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
> + if (att->attgenerated)
> + {
> + if (bms_is_member(att->attnum, cols))
> + gencolpresent = true;
> +
> + continue;
> + }
> +
>   nliveatts++;
>   }
>
>   /*
> - * If column list includes all the columns of the table,
> - * set it to NULL.
> + * If column list includes all the columns of the table
> + * and there are no generated columns, set it to NULL.
>   */
> - if (bms_num_members(cols) == nliveatts)
> + if (bms_num_members(cols) == nliveatts && !gencolpresent)
>   {
>
> Something seems not quite right (or maybe redundant) with this logic.
> For example, because you unconditionally 'continue' for generated
> columns, then AFAICT it is just not possible for bms_num_members(cols)
> == nliveatts and at the same time 'gencolpresent' to be true. So you
> could've just Asserted (!gencolpresent) instead of checking it in the
> condition and mentioning it in the comment.
>

It seems part of the logic is redundant. I propose to change something
along the lines of the attached. I haven't tested the attached change
as it shows how we can improve this part of code.

-- 
With Regards,
Amit Kapila.

  219. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-28T06:45:03Z

    On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> 7.
> +$node_publisher->wait_for_catchup('sub_gen');
> +
> +is( $node_subscriber->safe_psql(
> + 'postgres', "SELECT * FROM test_gen ORDER BY a"),
> + qq(1|2),
> + 'replication with generated columns in column list');
> +
>
> But, this is only testing normal replication. You should also include
> some initial table data so you can test that the initial table
> synchronization works too. Otherwise, I think current this patch has
> no proof that the initial 'copy_data' even works at all.
>

Per my tests, the initial copy doesn't work with 0001 alone. It needs
changes in table sync.c from the 0002 patch. Now, we can commit 0001
after fixing comments and mentioning in the commit message that this
patch supports only the replication of generated columns when
specified in the column list. The initial sync and replication of
generated columns when not specified in the column list will be
supported in future commits. OTOH, if the change to make table sync
work is simple, we can even combine that change.

-- 
With Regards,
Amit Kapila.

  220. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-10-28T06:54:09Z

    Dear Shubham,

More comments for v43-0001.

01. publication.out and publication.sql

I think your fix is not sufficient, even if it pass tests. 

```
-- error: system attributes "ctid" not allowed in column list
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-ERROR:  cannot use system column "ctid" in publication column list
+ERROR:  relation "testpub_tbl5" is already member of publication "testpub_fortable"
 ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl1 (id, ctid);
 ERROR:  cannot use system column "ctid" in publication column list
 -- error: duplicates not allowed in column list
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, a);
-ERROR:  duplicate column "a" in publication column list
+ERROR:  relation "testpub_tbl5" is already member of publication "testpub_fortable
```

The error message is not match with the comment. The comment said that the table
has already been added in the publication. I think the first line [1] succeeded by your change
and testpub_tbl5 became a member at that time then upcoming ALTER statements failed
by the duplicate registration.

```
-- ok
 ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ERROR:  relation "testpub_tbl5" is already member of publication "testpub_fortable"
```

You said OK but same error happened.

```
ALTER TABLE testpub_tbl5 DROP COLUMN c;     -- no dice
-ERROR:  cannot drop column c of table testpub_tbl5 because other objects depend on it
-DETAIL:  publication of table testpub_tbl5 in publication testpub_fortable depends on column c of table testpub_tbl5
-HINT:  Use DROP ... CASCADE to drop the dependent objects too.
```

This statement should be failed because c was included in the column.
However, it succeeded because previous ALTER PUBLICATION was failed.
Upcoming SQLs wrongly thawed ERRORs because of this.

Please look at all of differences before doing copy-and-paste.

02. 031_column_list.pl

```
-# TEST: Generated and dropped columns are not considered for the column list.
+# TEST: Dropped columns are not considered for the column list.
 # So, the publication having a column list except for those columns and a
 # publication without any column (aka all columns as part of the columns
 # list) are considered to have the same column list.
```

Based on the comment, this case does not test the behavior of generated columns
anymore. So, I felt column 'd' could be removed from the case.

03. 031_column_list.pl

Can we test that generated columns won't be replaced if it does not included in
the column list?

[1]:
```
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
```

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  221. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-28T06:56:43Z

    On Mon, Oct 28, 2024 at 4:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi, here are my review comments for patch v43-0001.
> >
> > ======
> > src/backend/replication/logical/proto.c
> >
> > 2.
> > +static bool
> > +should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> > +{
> > + if (att->attisdropped)
> > + return false;
> > +
> > + /*
> > + * Skip publishing generated columns if they are not included in the
> > + * column list.
> > + */
> > + if (att->attgenerated && !columns)
> > + return false;
> > +
> > + if (!column_in_column_list(att->attnum, columns))
> > + return false;
> > +
> > + return true;
> > +}
> >
> > Here, I wanted to suggest that the whole "Skip publishing generated
> > columns" if-part is unnecessary because the next check
> > (!column_in_column_list) is going to return false for the same
> > scenario anyhow.
> >
> > But, unfortunately, the "column_in_column_list" function has some
> > special NULL handling logic in it; this means none of this code is
> > quite what it seems to be (e.g. the function name
> > column_in_column_list is somewhat misleading)
> >
> > IMO it would be better to change the column_in_column_list signature
> > -- add another boolean param to say if a NULL column list is allowed
> > or not. That will remove any subtle behaviour and then you can remove
> > the "if (att->attgenerated && !columns)" part.
> >
>
> The NULL column list still means all columns, so changing the behavior
> as you are proposing doesn't make sense and would make the code
> difficult to understand.
>

My point was that the function 'column_in_column_list' would return
true even when there is no publication column list at all, so that
function name is misleading.

And, because in patch 0001 the generated columns only work when
specified via a column list it means now there is a difference
between:
- NULL (all columns specified in the column list) and
- NULL (no column list at all).

which seems strange and likely to cause confusion.

On closer inspection, this function 'column_in_column_list; is only
called from one place -- the new 'should_publish_column()'. I think
the function column_in_column_list should be thrown away and just
absorbed into the calling function 'should_publish_column'. Then the
misleading function name is eliminated, and the special NULL handling
can be commented on properly.

======
Regards,
Peter Smith.
Fujitsu Australia

  222. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-28T07:14:21Z

    On Mon, Oct 28, 2024 at 5:45 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > 7.
> > +$node_publisher->wait_for_catchup('sub_gen');
> > +
> > +is( $node_subscriber->safe_psql(
> > + 'postgres', "SELECT * FROM test_gen ORDER BY a"),
> > + qq(1|2),
> > + 'replication with generated columns in column list');
> > +
> >
> > But, this is only testing normal replication. You should also include
> > some initial table data so you can test that the initial table
> > synchronization works too. Otherwise, I think current this patch has
> > no proof that the initial 'copy_data' even works at all.
> >
>
> Per my tests, the initial copy doesn't work with 0001 alone. It needs
> changes in table sync.c from the 0002 patch. Now, we can commit 0001
> after fixing comments and mentioning in the commit message that this
> patch supports only the replication of generated columns when
> specified in the column list. The initial sync and replication of
> generated columns when not specified in the column list will be
> supported in future commits. OTOH, if the change to make table sync
> work is simple, we can even combine that change.
>

If this comes to a vote, then my vote is to refactor the necessary
tablesync COPY code back into patch 0001 so that patch 0001 can
replicate initial data properly stand alone.

Otherwise, (if we accept patch 0001 only partly works, like now) users
would have to jump through hoops to get any benefit from this patch by
itself. This is particularly true because the CREATE SUBSCRIPTION
'copy_data' parameter default is true, so patch 0001 is going to be
broken by default if there is any pre-existing table data when
publishing generated columns to default subscriptions.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  223. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-28T08:50:03Z

    On Mon, Oct 28, 2024 at 12:27 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Oct 28, 2024 at 4:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi, here are my review comments for patch v43-0001.
> > >
> > > ======
> > > src/backend/replication/logical/proto.c
> > >
> > > 2.
> > > +static bool
> > > +should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> > > +{
> > > + if (att->attisdropped)
> > > + return false;
> > > +
> > > + /*
> > > + * Skip publishing generated columns if they are not included in the
> > > + * column list.
> > > + */
> > > + if (att->attgenerated && !columns)
> > > + return false;
> > > +
> > > + if (!column_in_column_list(att->attnum, columns))
> > > + return false;
> > > +
> > > + return true;
> > > +}
> > >
> > > Here, I wanted to suggest that the whole "Skip publishing generated
> > > columns" if-part is unnecessary because the next check
> > > (!column_in_column_list) is going to return false for the same
> > > scenario anyhow.
> > >
> > > But, unfortunately, the "column_in_column_list" function has some
> > > special NULL handling logic in it; this means none of this code is
> > > quite what it seems to be (e.g. the function name
> > > column_in_column_list is somewhat misleading)
> > >
> > > IMO it would be better to change the column_in_column_list signature
> > > -- add another boolean param to say if a NULL column list is allowed
> > > or not. That will remove any subtle behaviour and then you can remove
> > > the "if (att->attgenerated && !columns)" part.
> > >
> >
> > The NULL column list still means all columns, so changing the behavior
> > as you are proposing doesn't make sense and would make the code
> > difficult to understand.
> >
>
> My point was that the function 'column_in_column_list' would return
> true even when there is no publication column list at all, so that
> function name is misleading.
>
> And, because in patch 0001 the generated columns only work when
> specified via a column list it means now there is a difference
> between:
> - NULL (all columns specified in the column list) and
> - NULL (no column list at all).
>
> which seems strange and likely to cause confusion.
>

This is no more strange than it was before the 0001 patch. Also, the
comment atop the function clarifies the special condition of the
function. OTOH, I am fine with pulling the check outside function as
you are proposing especially because now it is called from just one
place.

-- 
With Regards,
Amit Kapila.

  224. RE: Pgoutput not capturing the generated columns

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2024-10-28T11:43:02Z

    On Monday, October 28, 2024 1:34 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> 
> On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com>
> wrote:
> 
> >
> > 4. pgoutput_column_list_init
> >
> > - if (att->attisdropped || att->attgenerated)
> > + if (att->attisdropped)
> >   continue;
> >
> > + if (att->attgenerated)
> > + {
> > + if (bms_is_member(att->attnum, cols)) gencolpresent = true;
> > +
> > + continue;
> > + }
> > +
> >   nliveatts++;
> >   }
> >
> >   /*
> > - * If column list includes all the columns of the table,
> > - * set it to NULL.
> > + * If column list includes all the columns of the table
> > + * and there are no generated columns, set it to NULL.
> >   */
> > - if (bms_num_members(cols) == nliveatts)
> > + if (bms_num_members(cols) == nliveatts && !gencolpresent)
> >   {
> >
> > Something seems not quite right (or maybe redundant) with this logic.
> > For example, because you unconditionally 'continue' for generated
> > columns, then AFAICT it is just not possible for bms_num_members(cols)
> > == nliveatts and at the same time 'gencolpresent' to be true. So you
> > could've just Asserted (!gencolpresent) instead of checking it in the
> > condition and mentioning it in the comment.

I think it's possible for the condition you mentioned to happen.

For example:
 
CREATE TABLE test_mix_4 (a int primary key, b int, d int GENERATED ALWAYS AS (a + 1) STORED);
CREATE PUBLICATION pub FOR TABLE test_mix_4(a, d);

> >
> 
> It seems part of the logic is redundant. I propose to change something along the
> lines of the attached. I haven't tested the attached change as it shows how we
> can improve this part of code.

Thanks for the changes. I tried and faced an unexpected behavior
that the walsender will report Error "cannot use different column lists fo.."
in the following case:

Pub:
    CREATE TABLE test_mix_4 (a int PRIMARY KEY, b int, c int, d int GENERATED ALWAYS AS (a + 1) STORED);
    ALTER TABLE test_mix_4 DROP COLUMN c;
    CREATE PUBLICATION pub_mix_7 FOR TABLE test_mix_4 (a, b);
    CREATE PUBLICATION pub_mix_8 FOR TABLE test_mix_4;
Sub:
    CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub_mix_7, pub_mix_8;

The pub_mix_7 publishes column a,b which should be converted to NULL
in pgoutput, but was not due to the check of att_gen_present.

Based on above, I feel we can keep the original code as it is.

Best Regards,
Hou zj

  225. RE: Pgoutput not capturing the generated columns

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2024-10-28T11:45:41Z

    On Monday, October 28, 2024 2:54 PM Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> wrote:
> 
> 
> 02. 031_column_list.pl
> 
> ```
> -# TEST: Generated and dropped columns are not considered for the column
> list.
> +# TEST: Dropped columns are not considered for the column list.
>  # So, the publication having a column list except for those columns and a  #
> publication without any column (aka all columns as part of the columns  #
> list) are considered to have the same column list.
> ```
> 
> Based on the comment, this case does not test the behavior of generated
> columns anymore. So, I felt column 'd' could be removed from the case.

I think keeping the generated column can test the cases you mentioned
in comment #03, so we can modify the comments here to make that clear.

> 
> 03. 031_column_list.pl
> 
> Can we test that generated columns won't be replaced if it does not included in
> the column list?

As stated above, it can be covered in existing tests.

Best Regards,
Hou zj

  226. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-28T13:07:26Z

    On Mon, Oct 28, 2024 at 7:43 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are my review comments for patch v43-0001.
>
> ======
>
> 1. Missing docs update?
>
> The CREATE PUBLICATION docs currently says:
> When a column list is specified, only the named columns are
> replicated. If no column list is specified, all columns of the table
> are replicated through this publication, including any columns added
> later.
>
> ~
>
> For this patch, should that be updated to say "... all columns (except
> generated columns) of the table are replicated..."
>
> ======
> src/backend/replication/logical/proto.c
>
> 2.
> +static bool
> +should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> +{
> + if (att->attisdropped)
> + return false;
> +
> + /*
> + * Skip publishing generated columns if they are not included in the
> + * column list.
> + */
> + if (att->attgenerated && !columns)
> + return false;
> +
> + if (!column_in_column_list(att->attnum, columns))
> + return false;
> +
> + return true;
> +}
>
> Here, I wanted to suggest that the whole "Skip publishing generated
> columns" if-part is unnecessary because the next check
> (!column_in_column_list) is going to return false for the same
> scenario anyhow.
>
> But, unfortunately, the "column_in_column_list" function has some
> special NULL handling logic in it; this means none of this code is
> quite what it seems to be (e.g. the function name
> column_in_column_list is somewhat misleading)
>
> IMO it would be better to change the column_in_column_list signature
> -- add another boolean param to say if a NULL column list is allowed
> or not. That will remove any subtle behaviour and then you can remove
> the "if (att->attgenerated && !columns)" part.
>
> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 3. send_relation_and_attrs
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
>   if (att->atttypid < FirstGenbkiObjectId)
>   continue;
>
> + /*
> + * Skip publishing generated columns if they are not included in the
> + * column list.
> + */
> + if (att->attgenerated && !columns)
> + continue;
> +
>   /* Skip this attribute if it's not present in the column list */
>   if (columns != NULL && !bms_is_member(att->attnum, columns))
>   continue;
> ~
>
> Most of that code above looks to be doing the very same thing as the
> new 'should_publish_column' in proto.c. Won't it be better to expose
> the other function and share the common logic?
>
> ~~~
>
> 4. pgoutput_column_list_init
>
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
>   continue;
>
> + if (att->attgenerated)
> + {
> + if (bms_is_member(att->attnum, cols))
> + gencolpresent = true;
> +
> + continue;
> + }
> +
>   nliveatts++;
>   }
>
>   /*
> - * If column list includes all the columns of the table,
> - * set it to NULL.
> + * If column list includes all the columns of the table
> + * and there are no generated columns, set it to NULL.
>   */
> - if (bms_num_members(cols) == nliveatts)
> + if (bms_num_members(cols) == nliveatts && !gencolpresent)
>   {
>
> Something seems not quite right (or maybe redundant) with this logic.
> For example, because you unconditionally 'continue' for generated
> columns, then AFAICT it is just not possible for bms_num_members(cols)
> == nliveatts and at the same time 'gencolpresent' to be true. So you
> could've just Asserted (!gencolpresent) instead of checking it in the
> condition and mentioning it in the comment.
>
> ======
> src/test/regress/expected/publication.out
>
> 5.
> --- error: generated column "d" can't be in list
> +-- ok: generated column "d" can be in the list too
>  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
> -ERROR:  cannot use generated column "d" in publication column list
>
> By allowing the above to work without giving ERROR, I think you've
> broken many subsequent test expected results. e.g. I don't trust these
> "expected" results anymore because I didn't think these next test
> errors should have been affected, right?
>
>  -- error: system attributes "ctid" not allowed in column list
>  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
> -ERROR:  cannot use system column "ctid" in publication column list
> +ERROR:  relation "testpub_tbl5" is already member of publication
> "testpub_fortable"
>
> Hmm - looks like a wrong expected result to me.
>
> ~
>
>  -- error: duplicates not allowed in column list
>  ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, a);
> -ERROR:  duplicate column "a" in publication column list
> +ERROR:  relation "testpub_tbl5" is already member of publication
> "testpub_fortable"
>
> Hmm - looks like a wrong expected result to me.
>
> probably more like this...
>
> ======
> src/test/subscription/t/031_column_list.pl
>
> 6.
> +$node_subscriber->safe_psql(
> + 'postgres', qq(
> + CREATE TABLE test_gen (a int PRIMARY KEY, b int);
> +));
> +
> +$node_subscriber->safe_psql(
> + 'postgres', qq(
> + CREATE SUBSCRIPTION sub_gen CONNECTION '$publisher_connstr'
> PUBLICATION pub_gen;
> +));
>
> Should combine these.
>
> ~~~
>
> 7.
> +$node_publisher->wait_for_catchup('sub_gen');
> +
> +is( $node_subscriber->safe_psql(
> + 'postgres', "SELECT * FROM test_gen ORDER BY a"),
> + qq(1|2),
> + 'replication with generated columns in column list');
> +
>
> But, this is only testing normal replication. You should also include
> some initial table data so you can test that the initial table
> synchronization works too. Otherwise, I think current this patch has
> no proof that the initial 'copy_data' even works at all.
>

All the agreed comments have been addressed. Please find the attached
v44 Patches for the required changes.

Thanks and Regards,
Shubham Khanna.

  227. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-28T13:10:51Z

    On Mon, Oct 28, 2024 at 8:47 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Shubham,
>
> Thanks for updating the patch! I resumed reviewing the patch set.
> Here are only cosmetic comments as my rehabilitation.
>
> 01. getPublications()
>
> I feel we could follow the notation like getSubscriptions(), because number of
> parameters became larger. How do you feel like attached?
>

I will handle this comment in a later set of patches.

> 02. fetch_remote_table_info()
>
> ```
>                           "SELECT DISTINCT"
> -                         "  (CASE WHEN (array_length(gpt.attrs, 1) = c.relnatts)"
> -                         "   THEN NULL ELSE gpt.attrs END)"
> +                         "  (gpt.attrs)"
> ```
>
> I think no need to separate lines and add bracket. How about like below?
>
> ```
>                                                  "SELECT DISTINCT gpt.attrs"
> ```
>
Fixed this.

The v44 version patches attached at [1] have the changes for the same.
[1] - https://www.postgresql.org/message-id/CAHv8RjLvr8ZxX-1TcaxrZns1nwgrVUTO_2jhDdOPys0WgrDyKQ%40mail.gmail.com

Thanks and Regards,
Shubham Khanna.

  228. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-29T02:14:17Z

    Here are my review comments for v44-0001.

======
doc/src/sgml/ref/create_publication.sgml

1.
-      When a column list is specified, only the named columns are replicated.
+      When a column list is specified, all columns (except generated columns)
+      of the table are replicated.
       If no column list is specified, all columns of the table are replicated
       through this publication, including any columns added later. It has no

Huh? This seems very wrong.

I think it should have been like:
When a column list is specified, only the named columns are
replicated. If no column list is specified, all table columns (except
generated columns) are replicated...

======
src/backend/replication/logical/proto.c

2.
+bool
+logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
+{
+ if (att->attisdropped)
+ return false;
+
+ /*
+ * Skip publishing generated columns if they are not included in the
+ * column list.
+ */
+ if (!columns && att->attgenerated)
+ return false;
+
+ /*
+ * Check if a column is covered by a column list.
+ */
+ if (columns && !bms_is_member(att->attnum, columns))
+ return false;
+
+ return true;
+}

I thought this could be more simply written as:

{
if (att->attisdropped)
  return false;

/* If a column list was specified only publish the specified columns. */
if (columns)
  return bms_is_member(att->attnum, columns);

/* If a column list was not specified publish everything except
generated columns. */
return !att->attgenerated;
}

======
src/backend/replication/pgoutput/pgoutput.c

3.
- if (att->attisdropped || att->attgenerated)
+ if (att->attisdropped)
+ continue;
+
+ if (att->attgenerated)
+ {
+ if (bms_is_member(att->attnum, cols))
+ gencolpresent = true;
+
  continue;
+ }
+

  nliveatts++;
  }

  /*
- * If column list includes all the columns of the table,
- * set it to NULL.
+ * If column list includes all the columns of the table
+ * and there are no generated columns, set it to NULL.
  */
- if (bms_num_members(cols) == nliveatts)
+ if (bms_num_members(cols) == nliveatts && !gencolpresent)
  {
  bms_free(cols);
  cols = NULL;
~

That code still looks strange to me. I think that unconditional
'continue' for attgenerated is breaking the meaning of 'nliveattrs'
(which I take as meaning 'count-of-the-attrs-to-be-published').

AFAICT the code should be more like this:

if (att->attgenerated)
{
  /* Generated cols are skipped unless they are present in a column list. */
  if (!bms_is_member(att->attnum, cols))
    continue;

  gencolpresent = true;
}

======
src/test/regress/sql/publication.sql

4.
 ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;

+-- ok: generated column "d" can be in the list too
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (d);
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;

Maybe you can change this test to do "SET TABLE testpub_tbl5 (a,d);"
instead of ADD TABLE, so then you can remove the earlier DROP and DROP
the table only once.

======
src/test/subscription/t/031_column_list.pl

5.
+# TEST: Dropped columns are not considered for the column list, and generated
+# columns are not replicated if they are not explicitly included in the column
+# list. So, the publication having a column list except for those columns and a
+# publication without any column (aka all columns as part of the columns list)
+# are considered to have the same column list.

Hmm. I don't think this wording is quite right "without any column".
AFAIK the original intent of this test was to prove only that
dropped/generated columns were ignored for the NULL column list logic.

That last sentence maybe should say more like:

So a publication with a column list specifying all table columns
(excluding only dropped and generated columns) is considered to be the
same as a publication that has no column list at all for that table.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  229. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-29T04:23:46Z

    On Tue, Oct 29, 2024 at 7:44 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> ======
> src/backend/replication/logical/proto.c
>
> 2.
> +bool
> +logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> +{
> + if (att->attisdropped)
> + return false;
> +
> + /*
> + * Skip publishing generated columns if they are not included in the
> + * column list.
> + */
> + if (!columns && att->attgenerated)
> + return false;
> +
> + /*
> + * Check if a column is covered by a column list.
> + */
> + if (columns && !bms_is_member(att->attnum, columns))
> + return false;
> +
> + return true;
> +}
>
> I thought this could be more simply written as:
>
> {
> if (att->attisdropped)
>   return false;
>
> /* If a column list was specified only publish the specified columns. */
> if (columns)
>   return bms_is_member(att->attnum, columns);
>
> /* If a column list was not specified publish everything except
> generated columns. */
> return !att->attgenerated;
> }
>

Your version is difficult to follow compared to what is proposed in
the current patch. It is a matter of personal choice, so I leave it to
the author (or others) which one they prefer. However, I suggest that
we add extra comments in the current patch where we return true at the
end of the function and also at the top of the function.

>
> ======
> src/test/regress/sql/publication.sql
>
> 4.
>  ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
>
> +-- ok: generated column "d" can be in the list too
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (d);
> +ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
>
> Maybe you can change this test to do "SET TABLE testpub_tbl5 (a,d);"
> instead of ADD TABLE, so then you can remove the earlier DROP and DROP
> the table only once.
>

Yeah, we can do that if we want, but let's not add the dependency of
the previous test. Separate tests make it easier to extend the tests
in the future. Now, if it would have saved a noticeable amount of
time, then we could have considered it. Having said that, we can keep
both columns a and d in the column list.

> ======
> src/test/subscription/t/031_column_list.pl
>
> 5.
> +# TEST: Dropped columns are not considered for the column list, and generated
> +# columns are not replicated if they are not explicitly included in the column
> +# list. So, the publication having a column list except for those columns and a
> +# publication without any column (aka all columns as part of the columns list)
> +# are considered to have the same column list.
>
> Hmm. I don't think this wording is quite right "without any column".
> AFAIK the original intent of this test was to prove only that
> dropped/generated columns were ignored for the NULL column list logic.
>
> That last sentence maybe should say more like:
>
> So a publication with a column list specifying all table columns
> (excluding only dropped and generated columns) is considered to be the
> same as a publication that has no column list at all for that table.
>

I think you are saying the same thing in slightly different words.
Both of those sound correct to me. So not sure if we get any advantage
by changing it.

-- 
With Regards,
Amit Kapila.

  230. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-10-29T05:49:38Z

    Dear Shubham,

Thanks for updating the patch! Here are my comments for v44.

01. fetch_remote_table_info()

`bool *remotegencolpresent` is accessed unconditionally, but it can cause crash
if NULL is passed to the function. Should we add an Assert to verify it?

02. fetch_remote_table_info()

```
+        if (server_version >= 180000)
+            *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
+
```

Can we add Assert(!isnull) like other parts?

03. fetch_remote_table_info()

Also, we do not have to reach here once *remotegencolpresent becomes true.
Based on 02 and 03, how about below?

```
		if (server_version >= 180000 && !(*remotegencolpresent))
		{
			*remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
			Assert(!isnull);
		}
```

04. pgoutput_column_list_init()

+                        if (att->attgenerated)
+                        {
+                            if (bms_is_member(att->attnum, cols))
+                                gencolpresent = true;
+
                             continue;
+                        }

I'm not sure it is correct. Why do you skip the generated column even when it is in
the column list? Also, can you add comments what you want to do?

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  231. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-29T06:00:02Z

    Here are my review comments for patch v44-0002.

======
Commit message.

1.
The commit message is missing.

======
src/backend/replication/logical/tablesync.c

fetch_remote_table_info:

2.
+fetch_remote_table_info(char *nspname, char *relname, LogicalRepRelation *lrel,
+ List **qual, bool *remotegencolpresent)

The name 'remotegencolpresent' sounds like it means a generated col is
present in the remote table, but don't we only care when it is being
published? So, would a better parameter name be more like
'remote_gencol_published'?

~~~

3.
Would it be better to introduce a new human-readable variable like:
bool check_for_published_gencols = (server_version >= 180000);

because then you could use that instead of having the 180000 check in
multiple places.

~~~

4.
-   lengthof(attrRow), attrRow);
+   server_version >= 180000 ? lengthof(attrRow) : lengthof(attrRow) -
1, attrRow);

If you wish, that length calculation could be written more concisely like:
lengthof(attrow) - (server_version >= 180000 ? 0 : 1)

~~~

5.
+ if (server_version >= 180000)
+ *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
+

Should this also say Assert(!isnull)?

======
src/test/subscription/t/031_column_list.pl

6.
+ qq(0|1),
+ 'replication with generated columns in column list');

Perhaps this message should be worded slightly differently, to
distinguish it from the "normal" replication message.

/replication with generated columns in column list/initial replication
with generated columns in column list/

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  232. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-29T06:15:16Z

    On Tue, Oct 29, 2024 at 11:19 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> 01. fetch_remote_table_info()
>
> `bool *remotegencolpresent` is accessed unconditionally, but it can cause crash
> if NULL is passed to the function. Should we add an Assert to verify it?
>

This is a static function being called from just one place, so don't
think this is required.

> 02. fetch_remote_table_info()
>
> ```
> +        if (server_version >= 180000)
> +            *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
> +
> ```
>
> Can we add Assert(!isnull) like other parts?
>
> 03. fetch_remote_table_info()
>
> Also, we do not have to reach here once *remotegencolpresent becomes true.
> Based on 02 and 03, how about below?
>
> ```
>                 if (server_version >= 180000 && !(*remotegencolpresent))
>                 {
>                         *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
>                         Assert(!isnull);
>                 }
> ```
>

Yeah, we can follow this suggestion but better to add a comment for the same.

-- 
With Regards,
Amit Kapila.

  233. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-29T06:28:24Z

    On Tue, Oct 29, 2024 at 11:30 AM Peter Smith <smithpb2250@gmail.com> wrote:
> ======
> src/backend/replication/logical/tablesync.c
>
> fetch_remote_table_info:
>
> 2.
> +fetch_remote_table_info(char *nspname, char *relname, LogicalRepRelation *lrel,
> + List **qual, bool *remotegencolpresent)
>
> The name 'remotegencolpresent' sounds like it means a generated col is
> present in the remote table, but don't we only care when it is being
> published? So, would a better parameter name be more like
> 'remote_gencol_published'?
>

I feel no need to add a 'remote' to this variable name as the function
name itself clarifies the same. Both in the function definition and at
the caller site, we can name it 'gencol_published'.

> ~~~
>
> 3.
> Would it be better to introduce a new human-readable variable like:
> bool check_for_published_gencols = (server_version >= 180000);
>
> because then you could use that instead of having the 180000 check in
> multiple places.
>

It is better to add a comment because it makes this part of the code
difficult to enhance in the same version (18) if required.

> ~~~
>
> 4.
> -   lengthof(attrRow), attrRow);
> +   server_version >= 180000 ? lengthof(attrRow) : lengthof(attrRow) -
> 1, attrRow);
>
> If you wish, that length calculation could be written more concisely like:
> lengthof(attrow) - (server_version >= 180000 ? 0 : 1)
>

The current way of the patch seems easier to follow.

-- 
With Regards,
Amit Kapila.

  234. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-29T09:35:57Z

    On Tue, 29 Oct 2024 at 07:44, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for v44-0001.
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 1.
> -      When a column list is specified, only the named columns are replicated.
> +      When a column list is specified, all columns (except generated columns)
> +      of the table are replicated.
>        If no column list is specified, all columns of the table are replicated
>        through this publication, including any columns added later. It has no
>
> Huh? This seems very wrong.
>
> I think it should have been like:
> When a column list is specified, only the named columns are
> replicated. If no column list is specified, all table columns (except
> generated columns) are replicated...

Modified

> ======
> src/backend/replication/logical/proto.c
>
> 2.
> +bool
> +logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> +{
> + if (att->attisdropped)
> + return false;
> +
> + /*
> + * Skip publishing generated columns if they are not included in the
> + * column list.
> + */
> + if (!columns && att->attgenerated)
> + return false;
> +
> + /*
> + * Check if a column is covered by a column list.
> + */
> + if (columns && !bms_is_member(att->attnum, columns))
> + return false;
> +
> + return true;
> +}
>
> I thought this could be more simply written as:
>
> {
> if (att->attisdropped)
>   return false;
>
> /* If a column list was specified only publish the specified columns. */
> if (columns)
>   return bms_is_member(att->attnum, columns);
>
> /* If a column list was not specified publish everything except
> generated columns. */
> return !att->attgenerated;
> }

I preferred the earlier code as it is more simple, added a few
comments for the same to avoid confusion.

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 3.
> - if (att->attisdropped || att->attgenerated)
> + if (att->attisdropped)
> + continue;
> +
> + if (att->attgenerated)
> + {
> + if (bms_is_member(att->attnum, cols))
> + gencolpresent = true;
> +
>   continue;
> + }
> +
>
>   nliveatts++;
>   }
>
>   /*
> - * If column list includes all the columns of the table,
> - * set it to NULL.
> + * If column list includes all the columns of the table
> + * and there are no generated columns, set it to NULL.
>   */
> - if (bms_num_members(cols) == nliveatts)
> + if (bms_num_members(cols) == nliveatts && !gencolpresent)
>   {
>   bms_free(cols);
>   cols = NULL;
> ~
>
> That code still looks strange to me. I think that unconditional
> 'continue' for attgenerated is breaking the meaning of 'nliveattrs'
> (which I take as meaning 'count-of-the-attrs-to-be-published').
>
> AFAICT the code should be more like this:
>
> if (att->attgenerated)
> {
>   /* Generated cols are skipped unless they are present in a column list. */
>   if (!bms_is_member(att->attnum, cols))
>     continue;
>
>   gencolpresent = true;
> }

Modified

> ======
> src/test/regress/sql/publication.sql
>
> 4.
>  ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
>
> +-- ok: generated column "d" can be in the list too
> +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (d);
> +ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
>
> Maybe you can change this test to do "SET TABLE testpub_tbl5 (a,d);"
> instead of ADD TABLE, so then you can remove the earlier DROP and DROP
> the table only once.

I did not make this change as Amit also felt that way, added column a
also mentioned in [1].

> ======
> src/test/subscription/t/031_column_list.pl
>
> 5.
> +# TEST: Dropped columns are not considered for the column list, and generated
> +# columns are not replicated if they are not explicitly included in the column
> +# list. So, the publication having a column list except for those columns and a
> +# publication without any column (aka all columns as part of the columns list)
> +# are considered to have the same column list.
>
> Hmm. I don't think this wording is quite right "without any column".
> AFAIK the original intent of this test was to prove only that
> dropped/generated columns were ignored for the NULL column list logic.
>
> That last sentence maybe should say more like:
>
> So a publication with a column list specifying all table columns
> (excluding only dropped and generated columns) is considered to be the
> same as a publication that has no column list at all for that table.

I have just changed "publication without any column" to "publication
without any column list" as the rest looks ok to me.

The attached v45 version patch has the changes for the same.  I have
also merged the 0002 patch as the patch looks fairly stable now.

 [1] - https://www.postgresql.org/message-id/CAA4eK1K31%3D1draCJE0ng3Drt8C9D65qPppwK%3D-V64YMiDyRziA%40mail.gmail.com

Regards,
Vignesh

  235. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-29T09:40:44Z

    On Tue, 29 Oct 2024 at 11:19, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Shubham,
>
> Thanks for updating the patch! Here are my comments for v44.
>
> 01. fetch_remote_table_info()
>
> `bool *remotegencolpresent` is accessed unconditionally, but it can cause crash
> if NULL is passed to the function. Should we add an Assert to verify it?

I have not made any changes for this as I felt it is not required.
Also Amit felt the same way as in [1].

> 02. fetch_remote_table_info()
>
> ```
> +        if (server_version >= 180000)
> +            *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
> +
> ```
>
> Can we add Assert(!isnull) like other parts?

Included it.

> 03. fetch_remote_table_info()
>
> Also, we do not have to reach here once *remotegencolpresent becomes true.
> Based on 02 and 03, how about below?
>
> ```
>                 if (server_version >= 180000 && !(*remotegencolpresent))
>                 {
>                         *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
>                         Assert(!isnull);
>                 }
> ```

Modified

> 04. pgoutput_column_list_init()
>
> +                        if (att->attgenerated)
> +                        {
> +                            if (bms_is_member(att->attnum, cols))
> +                                gencolpresent = true;
> +
>                              continue;
> +                        }
>
> I'm not sure it is correct. Why do you skip the generated column even when it is in
> the column list? Also, can you add comments what you want to do?

Modified it now and added comments.

The v45 version attached has the changes for the same.

[1] - https://www.postgresql.org/message-id/CAA4eK1Keq7hewXGe4mUHuCzEA5%3DZR5wQK0%2BL5TU%2BzVFUMrmOFw%40mail.gmail.com

Regards,
Vignesh

  236. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-29T09:48:16Z

    On Tue, 29 Oct 2024 at 11:30, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for patch v44-0002.
>
> ======
> Commit message.
>
> 1.
> The commit message is missing.

This patch is now merged, so no change required.

> ======
> src/backend/replication/logical/tablesync.c
>
> fetch_remote_table_info:
>
> 2.
> +fetch_remote_table_info(char *nspname, char *relname, LogicalRepRelation *lrel,
> + List **qual, bool *remotegencolpresent)
>
> The name 'remotegencolpresent' sounds like it means a generated col is
> present in the remote table, but don't we only care when it is being
> published? So, would a better parameter name be more like
> 'remote_gencol_published'?

I have changed it to gencol_published based on Amit's suggestion at [1].

> ~~~
>
> 3.
> Would it be better to introduce a new human-readable variable like:
> bool check_for_published_gencols = (server_version >= 180000);
>
> because then you could use that instead of having the 180000 check in
> multiple places.

I felt this is not required, so not making any change for this.

> ~~~
>
> 4.
> -   lengthof(attrRow), attrRow);
> +   server_version >= 180000 ? lengthof(attrRow) : lengthof(attrRow) -
> 1, attrRow);
>
> If you wish, that length calculation could be written more concisely like:
> lengthof(attrow) - (server_version >= 180000 ? 0 : 1)

I felt the current one is better, also Amit feels the same way as in
[1]. Not making any change for this.

> ~~~
>
> 5.
> + if (server_version >= 180000)
> + *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
> +
>
> Should this also say Assert(!isnull)?

Added an assert

> ======
> src/test/subscription/t/031_column_list.pl
>
> 6.
> + qq(0|1),
> + 'replication with generated columns in column list');
>
> Perhaps this message should be worded slightly differently, to
> distinguish it from the "normal" replication message.
>
> /replication with generated columns in column list/initial replication
> with generated columns in column list/

Modified

The v45 version patch attached at [2] has the changes for the same.

[1] - https://www.postgresql.org/message-id/CAA4eK1Lpzy3eqd2AOM%2BTXp80SFL1cCfX3cf9thjL-hJxn%2BAYGA%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/CALDaNm1oc-%2Buav380Z1k6gCZY5GJn5ZYKRexwM%2BqqGiRinUS-Q%40mail.gmail.com

Regards,
Vignesh

  237. Re: Pgoutput not capturing the generated columns

    Shubham Khanna <khannashubham1197@gmail.com> — 2024-10-29T11:38:50Z

    On Tue, Oct 29, 2024 at 3:18 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 29 Oct 2024 at 11:30, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Here are my review comments for patch v44-0002.
> >
> > ======
> > Commit message.
> >
> > 1.
> > The commit message is missing.
>
> This patch is now merged, so no change required.
>
> > ======
> > src/backend/replication/logical/tablesync.c
> >
> > fetch_remote_table_info:
> >
> > 2.
> > +fetch_remote_table_info(char *nspname, char *relname, LogicalRepRelation *lrel,
> > + List **qual, bool *remotegencolpresent)
> >
> > The name 'remotegencolpresent' sounds like it means a generated col is
> > present in the remote table, but don't we only care when it is being
> > published? So, would a better parameter name be more like
> > 'remote_gencol_published'?
>
> I have changed it to gencol_published based on Amit's suggestion at [1].
>
> > ~~~
> >
> > 3.
> > Would it be better to introduce a new human-readable variable like:
> > bool check_for_published_gencols = (server_version >= 180000);
> >
> > because then you could use that instead of having the 180000 check in
> > multiple places.
>
> I felt this is not required, so not making any change for this.
>
> > ~~~
> >
> > 4.
> > -   lengthof(attrRow), attrRow);
> > +   server_version >= 180000 ? lengthof(attrRow) : lengthof(attrRow) -
> > 1, attrRow);
> >
> > If you wish, that length calculation could be written more concisely like:
> > lengthof(attrow) - (server_version >= 180000 ? 0 : 1)
>
> I felt the current one is better, also Amit feels the same way as in
> [1]. Not making any change for this.
>
> > ~~~
> >
> > 5.
> > + if (server_version >= 180000)
> > + *remotegencolpresent |= DatumGetBool(slot_getattr(slot, 5, &isnull));
> > +
> >
> > Should this also say Assert(!isnull)?
>
> Added an assert
>
> > ======
> > src/test/subscription/t/031_column_list.pl
> >
> > 6.
> > + qq(0|1),
> > + 'replication with generated columns in column list');
> >
> > Perhaps this message should be worded slightly differently, to
> > distinguish it from the "normal" replication message.
> >
> > /replication with generated columns in column list/initial replication
> > with generated columns in column list/
>
> Modified
>
> The v45 version patch attached at [2] has the changes for the same.
>
> [1] - https://www.postgresql.org/message-id/CAA4eK1Lpzy3eqd2AOM%2BTXp80SFL1cCfX3cf9thjL-hJxn%2BAYGA%40mail.gmail.com
> [2] - https://www.postgresql.org/message-id/CALDaNm1oc-%2Buav380Z1k6gCZY5GJn5ZYKRexwM%2BqqGiRinUS-Q%40mail.gmail.com
>

While performing the Backward Compatibility Test, I found that
'tablesync' is not working for the older versions i.e., from
version-12 till version-15.
I created 2 nodes ; PUBLISHER on old versions and SUBSCRIBER on HEAD +
v45 Patch for testing.
Following was done on the PUBLISHER node:
CREATE TABLE t1 (c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED);
INSERT INTO t1 (c1) VALUES (1), (2);
CREATE PUBLICATION pub1 for table t1;

Following  was done on the SUBSCRIBER node:
CREATE TABLE t1 (c1 int, c2 int);
CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres' PUBLICATION pub1;

Following Error occurs repeatedly in the Subscriber log files:
ERROR:  could not start initial contents copy for table "public.t1":
ERROR:  column "c2" is a generated column
DETAIL:  Generated columns cannot be used in COPY.

Thanks and Regards,
Shubham Khanna.

  238. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-29T15:20:15Z

    On Tue, 29 Oct 2024 at 17:09, Shubham Khanna
<khannashubham1197@gmail.com> wrote:
>
> While performing the Backward Compatibility Test, I found that
> 'tablesync' is not working for the older versions i.e., from
> version-12 till version-15.
> I created 2 nodes ; PUBLISHER on old versions and SUBSCRIBER on HEAD +
> v45 Patch for testing.
> Following was done on the PUBLISHER node:
> CREATE TABLE t1 (c1 int, c2 int GENERATED ALWAYS AS (c1 * 2) STORED);
> INSERT INTO t1 (c1) VALUES (1), (2);
> CREATE PUBLICATION pub1 for table t1;
>
> Following  was done on the SUBSCRIBER node:
> CREATE TABLE t1 (c1 int, c2 int);
> CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres' PUBLICATION pub1;
>
> Following Error occurs repeatedly in the Subscriber log files:
> ERROR:  could not start initial contents copy for table "public.t1":
> ERROR:  column "c2" is a generated column
> DETAIL:  Generated columns cannot be used in COPY.

Thank you for reporting this issue. The attached v46 patch addresses
the problem and includes some adjustments to the comments. Thanks to
Amit for sharing the comment changes offline.

Regards,
Vignesh

  239. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-10-30T09:36:34Z

    On Tue, Oct 29, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Thank you for reporting this issue. The attached v46 patch addresses
> the problem and includes some adjustments to the comments. Thanks to
> Amit for sharing the comment changes offline.
>

Pushed. Kindly rebase and send the remaining patches.

-- 
With Regards,
Amit Kapila.

  240. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-30T16:16:06Z

    On Wed, 30 Oct 2024 at 15:06, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Oct 29, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thank you for reporting this issue. The attached v46 patch addresses
> > the problem and includes some adjustments to the comments. Thanks to
> > Amit for sharing the comment changes offline.
> >
>
> Pushed. Kindly rebase and send the remaining patches.

Thanks for committing this patch, here is a rebased version of the
remaining patches.

Regards,
Vignesh

  241. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-30T23:11:46Z

    On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 30 Oct 2024 at 15:06, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Oct 29, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Thank you for reporting this issue. The attached v46 patch addresses
> > > the problem and includes some adjustments to the comments. Thanks to
> > > Amit for sharing the comment changes offline.
> > >
> >
> > Pushed. Kindly rebase and send the remaining patches.
>
> Thanks for committing this patch, here is a rebased version of the
> remaining patches.
>

Hi,

I found that the docs of src/sgml/ddl.sgml [1] are still saying:

     <para>
      Generated columns are skipped for logical replication and cannot be
      specified in a <command>CREATE PUBLICATION</command> column list.
     </para>

But that is contrary to the new behaviour after the "Replicate
generated columns when specified in the column list." commit yesterday
[2].

It looks like an oversight. I think updating that paragraph should
have been included with yesterday's commit.

======
[1] https://github.com/postgres/postgres/blob/master/doc/src/sgml/ddl.sgml
[2] https://github.com/postgres/postgres/commit/745217a051a9341e8c577ea59a87665d331d4af0

Kind Regards,
Peter Smith.
Fujitsu Australia

  242. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-10-31T02:13:28Z

    On Thu, 31 Oct 2024 at 04:42, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 30 Oct 2024 at 15:06, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Tue, Oct 29, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > Thank you for reporting this issue. The attached v46 patch addresses
> > > > the problem and includes some adjustments to the comments. Thanks to
> > > > Amit for sharing the comment changes offline.
> > > >
> > >
> > > Pushed. Kindly rebase and send the remaining patches.
> >
> > Thanks for committing this patch, here is a rebased version of the
> > remaining patches.
> >
>
> Hi,
>
> I found that the docs of src/sgml/ddl.sgml [1] are still saying:
>
>      <para>
>       Generated columns are skipped for logical replication and cannot be
>       specified in a <command>CREATE PUBLICATION</command> column list.
>      </para>
>
> But that is contrary to the new behaviour after the "Replicate
> generated columns when specified in the column list." commit yesterday
> [2].
>
> It looks like an oversight. I think updating that paragraph should
> have been included with yesterday's commit.

Thanks for the findings, the attached patch has the changes for the same.

Regards,
Vignesh

  243. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-10-31T02:38:58Z

    On Thu, Oct 31, 2024 at 1:14 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 31 Oct 2024 at 04:42, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, 30 Oct 2024 at 15:06, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Tue, Oct 29, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > Thank you for reporting this issue. The attached v46 patch addresses
> > > > > the problem and includes some adjustments to the comments. Thanks to
> > > > > Amit for sharing the comment changes offline.
> > > > >
> > > >
> > > > Pushed. Kindly rebase and send the remaining patches.
> > >
> > > Thanks for committing this patch, here is a rebased version of the
> > > remaining patches.
> > >
> >
> > Hi,
> >
> > I found that the docs of src/sgml/ddl.sgml [1] are still saying:
> >
> >      <para>
> >       Generated columns are skipped for logical replication and cannot be
> >       specified in a <command>CREATE PUBLICATION</command> column list.
> >      </para>
> >
> > But that is contrary to the new behaviour after the "Replicate
> > generated columns when specified in the column list." commit yesterday
> > [2].
> >
> > It looks like an oversight. I think updating that paragraph should
> > have been included with yesterday's commit.
>
> Thanks for the findings, the attached patch has the changes for the same.
>

LGTM.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  244. Re: Pgoutput not capturing the generated columns

    Ajin Cherian <itsajin@gmail.com> — 2024-10-31T10:55:49Z

    I ran some tests and verified that the patch works with previous versions
of PG12 and PG17
1. Verified with publications with generated columns and without generated
columns on patched code and subscriptions on PG12 and PG17
Observations:
    a. If publication is created with publish_generated_columns=true or
with generated columns mentioned explicitly, then tablesync will not copy
generated columns but post tablesync the generated columns are replicated
    b. Column list override (publish_generated_columns=false) behaviour

These seem expected.

2. Publication on PG12 and PG17 with subscription on patched code:
Observation:
Behaves as if without patch.

3. Pg_dump - confirmed that the new version correctly dumps the new syntax
4. Pg_upgrade - confirmed that when updating from previous version to the
latest the "Generated columns" field default to false.
5. Verified that publications with different column list are disallowed to
be subscribed by one subscription
   a. PUB_A(column list = (a, b)) PUB_B(no column list, with
publish_generated_column) - OK
   b. PUB_A(column list = (a, b)) PUB_B(no column list, without
publish_generated_column) - FAIL
   c.  PUB_A(no column list, without publish_generated_column) PUB_B(no
column list, with publish_generated_column) - FAIL

Tests did not show any unexpected behaviour.

regards,
Ajin Cherian
Fujitsu Australia

  245. Re: Pgoutput not capturing the generated columns

    Ajin Cherian <itsajin@gmail.com> — 2024-10-31T11:14:37Z

    On Thu, Oct 31, 2024 at 9:55 PM Ajin Cherian <itsajin@gmail.com> wrote:

> I ran some tests and verified that the patch works with previous versions
> of PG12 and PG17
> 1. Verified with publications with generated columns and without generated
> columns on patched code and subscriptions on PG12 and PG17
> Observations:
>     a. If publication is created with publish_generated_columns=true or
> with generated columns mentioned explicitly, then tablesync will not copy
> generated columns but post tablesync the generated columns are replicated
>     b. Column list override (publish_generated_columns=false) behaviour
>
> These seem expected.
>
>
Currently the documentation does not talk about this behaviour, I suggest
this be added similar to how such a behaviour was documented when the
original row-filter version was committed.
Suggestion:
"If a subscriber is a pre-18 version, the initial table synchronization
won't publish generated columns even if they are defined in the publisher."

regards,
Ajin Cherian
Fujitsu Australia

  246. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-01T01:40:23Z

    On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
> Thanks for committing this patch, here is a rebased version of the
> remaining patches.
>

Hi Vignesh, thanks for the rebased patches.

Here are my review comments for patch v1-0001.

======
Commit message.

1.
The commit message text is stale, so needs some updates.

For example, it is still saying "Generated column values are not
currently replicated..." but that is not correct anymore since the
recent push of the previous v46-0001 patch [1], which already
implemented replication of generated columns when they are specified
in a publication column list..

======
doc/src/sgml/ddl.sgml

2.
      <para>
-      Generated columns are skipped for logical replication and cannot be
-      specified in a <command>CREATE PUBLICATION</command> column list.
+      Generated columns are allowed to be replicated during logical replication
+      according to the <command>CREATE PUBLICATION</command> option
+      <link linkend="sql-createpublication-params-with-publish-generated-columns">
+      <literal>include_generated_columns</literal></link>.
      </para>

This explanation is incomplete because generated columns can also be
specified in a publication column list which has nothing to do with
the new option. In fact, lack of mention about the column lists seems
like an oversight which should have been in the previous patch [1]. I
already posted another mail about this [2].

======
doc/src/sgml/protocol.sgml

3.
      <para>
-      Next, one of the following submessages appears for each column:
+      Next, one of the following submessages appears for each column
(except generated columns):

Hmm. Now that generated column replication is supported is this change
still required?

======
doc/src/sgml/ref/create_publication.sgml

4.
+
+       <varlistentry
id="sql-createpublication-params-with-publish-generated-columns">
+        <term><literal>publish_generated_columns</literal>
(<type>boolean</type>)</term>
+        <listitem>
+         <para>
+          Specifies whether the generated columns present in the tables
+          associated with the publication should be replicated.
+          The default is <literal>false</literal>.
+         </para>
+        </listitem>
+       </varlistentry>
+

I know that the subsequent DOCS patch V1-0002 will explain more about
this, but as a stand-alone patch 0001 maybe you need to clarify that a
publication column list will override this 'publish_generated_columns'
parameter.

======
src/backend/catalog/pg_publication.c

has_column_list_defined:

5.
+/*
+ * Returns true if the relation has column list associated with the
publication,
+ * false otherwise.
+ */
+bool
+has_column_list_defined(Publication *pub, Oid relid)
+{
+ HeapTuple cftuple = NULL;
+ bool isnull = true;
+
+ if (pub->alltables)
+ return false;
+
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+   ObjectIdGetDatum(relid),
+   ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(cftuple))
+ {
+ /* Lookup the column list attribute. */
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+    Anum_pg_publication_rel_prattrs,
+    &isnull);
+ if (!isnull)
+ {
+ ReleaseSysCache(cftuple);
+ return true;
+ }
+
+ ReleaseSysCache(cftuple);
+ }
+
+ return false;
+}
+

5a.
It might be tidier if you checked for !HeapTupleIsValid(cftuple) and
do early return false, instead of needing an indented if-block.

~

5b.
The code can be rearranged and simplified -- you don't need multiple
calls to ReleaseSysCache.

SUGGESTION:

/* Lookup the column list attribute. */
(void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
  Anum_pg_publication_rel_prattrs,
  &isnull);

ReleaseSysCache(cftuple);
/* Was a column list found? */
return isnull ? false : true;

~~~

pub_getallcol_bitmapset:

6.
+/*
+ * Return a column list bitmap for the specified table.
+ *
+ * Generated columns are included if pubgencols is true.
+ *
+ * If mcxt isn't NULL, build the bitmapset in that context.
+ */
+Bitmapset *
+pub_getallcol_bitmapset(Relation relation, bool pubgencols,
+ MemoryContext mcxt)

IIUC this is a BMS of the table columns to be published. The function
comment seems confusing to me when it says "column list bitmap"
because IIUC this function is not really anything to do with a
publication "column list", which is an entirely different thing.

======
src/backend/replication/logical/proto.c

7.
 static void logicalrep_write_attrs(StringInfo out, Relation rel,
-    Bitmapset *columns);
+    Bitmapset *columns, bool pubgencols);
 static void logicalrep_write_tuple(StringInfo out, Relation rel,
     TupleTableSlot *slot,
-    bool binary, Bitmapset *columns);
+    bool binary, Bitmapset *columns,
+    bool pubgencols);

The meaning of all these new 'pubgencols' are ambiguous. e.g. Are they
(a) The value of the CREATE PUBLICATION 'publish_generate_columns'
parameter, or does it mean (b) Just some generated column is being
published (maybe via column list or maybe not).

I think it means (a) but, if true, that could be made much more clear
by changing all of these names to 'pubgencols_option' or something
similar.  Actually, now I have doubts about that also -- I think this
might be magically assigned to false if no generated columns exist in
the table. Anyway, please do whatever you can to disambiguate this.

~~~

logicalrep_should_publish_column:

8.
The function comment is stale. It is still only talking about
generated columns in column lists.

SUGGESTION
Note that generated columns can be published only when present in a
publication column list, or (if there is no column list), when the
publication parameter 'publish_generated_columns' is true.

~~~

9.
bool
logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
bool pubgencols)
{
if (att->attisdropped)
return false;

/*
* Skip publishing generated columns if they are not included in the
* column list or if the option is not specified.
*/
if (!columns && !pubgencols && att->attgenerated)
return false;

/*
* Check if a column is covered by a column list.
*/
if (columns && !bms_is_member(att->attnum, columns))
return false;

return true;
}

Same as mentioned before in my previous v46-0001 review comments, I
feel that the conditionals of this function are over-complicated and
that there are more 'return' points than necessary. The alternative
code below looks simpler to me.

SUGGESTION
bool
logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
bool pubgencols_option)
{
  if (att->attisdropped)
    return false;

  if (columns)
  {
    /*
* Has a column list:
* Publish only cols named in that list.
*/
    return bms_is_member(att->attnum, columns);
  }
  else
  {
    /*
     * Has no column list:
* Publish generated cols only if 'publish_generated_cols' is true.
     * Publish all non-generated cols.
    */
    return att->attgenerated ? pubgencols_option : true;
  }
}

======
src/backend/replication/pgoutput/pgoutput.c

10.
+ /* Include publishing generated columns */
+ bool pubgencols;
+

There is similar ambiguity here with this field-name as was mentioned
about for other 'pbgencols' function params. I had initially thought
that this this just caries around same value as the publication option
'publish_generated_columns' but now (after looking at function
check_and_init_gencol) I think that might not be the case because I
saw it can be assigned false (overriding the publication option?).

Anyway, the comment needs to be made much clearer about what is the
true meaning of this field. Or, rename it if there is a better name.

~~~

11.
+static void send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx,
+ RelationSyncEntry *relentry);

Was there some reason to move this static? Maybe it is better just to
change the existing static in-place rather than moving code around at
the same time.

~~~

send_relation_and_attrs:

12.
- if (!logicalrep_should_publish_column(att, columns))
+ if (!logicalrep_should_publish_column(att, columns, relentry->pubgencols))
  continue;
It seemed a bit strange/inconsistent that 'columns' was assigned to a
local var, but 'pubgencols' was not, given they are both fields of the
same struct. Maybe removing this 'columns' var would be consistent
with other code in this patch.

~~~

13.
check_and_init_gencol:

nit - missing periods for comments.

~~~

14.
+ /* There is no generated columns to be published *

/There is no generated columns/There are no generated columns/

~~~

15.
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);

AFAIK this can be re-written using a different macro to avoid needing
the 'lc' var.

~~~

pgoutput_column_list_init:

16.
+ bool collistpubexist = false;

This seemed like not a very good name, How about 'found_pub_with_collist';

~~~

17.
bool pub_no_list = true;

nit - Not caused by this patch, but it's closely related; In passing
we should declare this variable at a lower scope, and rename it to
'isnull' which is more in keeping with the comments around it.

~~~

18.
+ /*
+ * For non-column list publications—such as TABLE (without a column
+ * list), ALL TABLES, or ALL TABLES IN SCHEMA publications consider
+ * all columns of the table, including generated columns, based on the
+ * pubgencols option.
+ */

Some minor tweaks.

SUGGESTION
For non-column list publications — e.g. TABLE (without a column list),
ALL TABLES, or ALL TABLES IN SCHEMA, we consider all columns of the
table (including generated columns when 'publish_generated_columns'
option is true).

~~~

19.
+ Assert(pub->pubgencols == entry->pubgencols);
+
+ /*
+ * Retrieve the columns if they haven't been prepared yet, or if
+ * there are multiple publications.
+ */
+ if (!relcols && (list_length(publications) > 1))
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+ relcols = pub_getallcol_bitmapset(relation, entry->pubgencols,
+   entry->entry_cxt);
+ }

19a.
Is that Assert correct? I ask only because AFAICT I saw in previous
function (check_and_init_gencol) there is code that might change
entry->pubgencols = false; even if the 'publish_generated_columns'
option is true but there were not generated columns found in the
table.

~

19b.
The comment says "or if there are multiple publications" but the code
says &&. Something seems wrong.

~~~

20.
+ /*
+ * If no column list publications exit, columns will be selected later
+ * according to the generated columns option.
+ */

20a.
typo - /exit/exist/

~

20b.
There is a GENERAL problem that applies for lots of comments of this
patch (including this comment) because the new publication option is
referred to inconsistently in many different ways:

e.g.
- the generated columns option.
- if the option is not specified
- publish_generated_columns option.
- the pubgencols option
- 'publish_generated_columns' option

All these references should be made the same. My personal preference
is the last one ('publish_generated_columns' option).

~~~

get_rel_sync_entry:

21.
+ /* Check whether to publish to generated columns. */
+ check_and_init_gencol(data, rel_publications, entry);
+

typo in comment - "publishe to"?

======
src/include/catalog/pg_publication.h

22.
 extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
     MemoryContext mcxt);
+extern Bitmapset *pub_getallcol_bitmapset(Relation relation, bool pubgencols,
+   MemoryContext mcxt);

Maybe a better name for this new function is 'pub_allcols_bitmapse'.
That would then be consistent with the other BMS function which
doesn't include the word "get".


======
Some of my suggested updates above are already implemented in the
attached nitpicks diff. Please refer to it.

======
[1] push support for gencols in column list --
https://github.com/postgres/postgres/commit/745217a051a9341e8c577ea59a87665d331d4af0
[2] docs oversight in commit --
https://www.postgresql.org/message-id/CAHut%2BPtBVSxNDph-mHP_SE4%2BWw%2BxJ0SKhVupnx5uVhK3V_SDHw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  247. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-01T03:52:37Z

    On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:

> Thanks for committing this patch, here is a rebased version of the
> remaining patches.
>

Hi Vignesh.

Here are my review comments for the docs patch v1-0002.

======
Commit message

1.
This patch updates docs to describe the new feature allowing
replication of generated
columns. This includes addition of a new section "Generated Column
Replication" to the
"Logical Replication" documentation chapter.

~

That first sentence was correct previously when this patch contained
*all* the gencols documentation, but now some of the feature docs are
already handled by previous patches, so the first sentence can be
removed.

Now patch 0002 is only for adding the new chapter, plus the references to it.

~

/This includes addition of a new section/This patch adds a new section/

======
doc/src/sgml/protocol.sgml

2.
      <para>
-      Next, one of the following submessages appears for each column
(except generated columns):
+      Next, one of the following submessages appears for each column:

AFAIK this simply cancels out a change from the v1-0001 patch which
IMO should have not been there in the first place. Please refer to my
v1-0001 review for the same.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  248. RE: Pgoutput not capturing the generated columns

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2024-11-01T07:57:03Z

    Dear Vignesh,

Thanks for rebasing the patch! Before reviewing deeply, I want to confirm the specification.
I understood like below based on the documentation.

- If publish_generated_columns is false, the publication won't replicate generated columns
- If publish_generated_columns is true, the behavior on the subscriber depends on the table column:
  - If it is a generated column even on the subscriber, it causes an ERROR.
  - If it is a regular column, the generated value is replicated.
  - If the column is missing, it causes an ERROR.

However, below test implies that generated columns can be replicated even if
publish_generated_columns is false. Not sure...

```
# Verify that incremental replication of generated columns occurs
# when they are included in the column list, regardless of the
# publish_generated_columns option.
$result =
  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3 ORDER BY a");
is( $result, qq(|2
|4
|6
|8),
	'tab3 incremental replication, when publish_generated_columns=false');
```

Also, I've tested the case both pub and sub had the generated column, but the ERROR was strange for me.

```
test_pub=# CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
test_pub=# CREATE PUBLICATION pub FOR TABLE gencoltable(a, b) WITH (publish_generated_columns = true);
test_pub=# INSERT INTO gencoltable (a) VALUES (generate_series(1, 10));

test_sub=# CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
test_sub=# CREATE SUBSCRIPTION sub CONNECTION ... PUBLICATION pub;

-> ERROR: logical replication target relation "public.gencoltable" is missing replicated column: "b"
```

The attribute existed on the sub but it was reported as missing column. I think
we should somehow report like "generated column on publisher is replicated the
generated column on the subscriber".

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  249. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-03T13:27:58Z

    On Thu, 31 Oct 2024 at 16:44, Ajin Cherian <itsajin@gmail.com> wrote:
>
>
>
> On Thu, Oct 31, 2024 at 9:55 PM Ajin Cherian <itsajin@gmail.com> wrote:
>>
>> I ran some tests and verified that the patch works with previous versions of PG12 and PG17
>> 1. Verified with publications with generated columns and without generated columns on patched code and subscriptions on PG12 and PG17
>> Observations:
>>     a. If publication is created with publish_generated_columns=true or with generated columns mentioned explicitly, then tablesync will not copy generated columns but post tablesync the generated columns are replicated
>>     b. Column list override (publish_generated_columns=false) behaviour
>>
>> These seem expected.
>>
>
> Currently the documentation does not talk about this behaviour, I suggest this be added similar to how such a behaviour was documented when the original row-filter version was committed.
> Suggestion:
> "If a subscriber is a pre-18 version, the initial table synchronization won't publish generated columns even if they are defined in the publisher."

The updated patch has the changes for the same.

Regards,
Vignesh

  250. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-03T15:22:26Z

    On Fri, 1 Nov 2024 at 13:27, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Vignesh,
>
> Thanks for rebasing the patch! Before reviewing deeply, I want to confirm the specification.
> I understood like below based on the documentation.
>
> - If publish_generated_columns is false, the publication won't replicate generated columns
> - If publish_generated_columns is true, the behavior on the subscriber depends on the table column:
>   - If it is a generated column even on the subscriber, it causes an ERROR.
>   - If it is a regular column, the generated value is replicated.
>   - If the column is missing, it causes an ERROR.

This is correct.

> However, below test implies that generated columns can be replicated even if
> publish_generated_columns is false. Not sure...
>
> ```
> # Verify that incremental replication of generated columns occurs
> # when they are included in the column list, regardless of the
> # publish_generated_columns option.
> $result =
>   $node_subscriber->safe_psql('postgres', "SELECT * FROM tab3 ORDER BY a");
> is( $result, qq(|2
> |4
> |6
> |8),
>         'tab3 incremental replication, when publish_generated_columns=false');
> ```

Yes, this is a special case where the column list will take priority
over the publish_generated_columns option. The same was discussed at
[1].

> Also, I've tested the case both pub and sub had the generated column, but the ERROR was strange for me.
>
> ```
> test_pub=# CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
> test_pub=# CREATE PUBLICATION pub FOR TABLE gencoltable(a, b) WITH (publish_generated_columns = true);
> test_pub=# INSERT INTO gencoltable (a) VALUES (generate_series(1, 10));
>
> test_sub=# CREATE TABLE gencoltable (a int PRIMARY KEY, b int GENERATED ALWAYS AS (a * 2) STORED);
> test_sub=# CREATE SUBSCRIPTION sub CONNECTION ... PUBLICATION pub;
>
> -> ERROR: logical replication target relation "public.gencoltable" is missing replicated column: "b"
> ```
>
> The attribute existed on the sub but it was reported as missing column. I think
> we should somehow report like "generated column on publisher is replicated the
> generated column on the subscriber".

Agree on this, we will include a fix for this in one of the upcoming versions.

[1] - https://www.postgresql.org/message-id/CAA4eK1JgdyLYGo%2BG%3Db90VCqpbtwGMV8Su5Cuafo_hByWNTbkBg%40mail.gmail.com

Regards,
Vignesh

  251. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-04T02:12:48Z

    On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 30 Oct 2024 at 15:06, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Oct 29, 2024 at 8:50 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Thank you for reporting this issue. The attached v46 patch addresses
> > > the problem and includes some adjustments to the comments. Thanks to
> > > Amit for sharing the comment changes offline.
> > >
> >
> > Pushed. Kindly rebase and send the remaining patches.
>
> Thanks for committing this patch, here is a rebased version of the
> remaining patches.
>

Here are some review comments for the patch v1-0003 (tap tests)

======
src/test/subscription/t/011_generated.pl

1.
+# The following combinations are tested:
+# - Publication pub1 on the 'postgres' database with the option
+#   publish_generated_columns set to false.
+# - Publication pub2 on the 'postgres' database with the option
+#   publish_generated_columns set to true.
+# - Subscription sub1 on the 'postgres' database for publication pub1.
+# - Subscription sub2 on the 'test_pgc_true' database for publication pub2.

Those aren't really "combinations" anymore. That's just describing how
these pub/sub tests are configured.

/The following combinations are tested:/The test environment is set up
as follows:/

~~~

2.
+# Wait for the initial synchronization of the 'regress_sub1_gen_to_nogen'
+# subscription in the 'postgres' database.
+$node_subscriber->wait_for_subscription_sync($node_publisher,
+ 'regress_sub1_gen_to_nogen', 'postgres');
+
+# Wait for the initial synchronization of the 'regress_sub2_gen_to_nogen'
+# subscription in the 'test_pgc_true' database.
+$node_subscriber->wait_for_subscription_sync($node_publisher,
+ 'regress_sub2_gen_to_nogen', 'test_pgc_true');
+

These detailed descriptions are not adding much value here. Just
combining these and saying "Wait for the initial synchronization of
both subscriptions" would have been enough, I think.

~~~

3.
+# =============================================================================
+# The following test cases demonstrate the behavior of generated column
+# replication with publish_generated_columns set to false and true:
+# Test: Publication column list includes generated columns when
+# publish_generated_columns is set to false.
+# Test: Publication column list excludes generated columns when
+# publish_generated_columns is set to false.
+# Test: Publication column list includes generated columns when
+# publish_generated_columns is set to true.
+# Test: Publication column list excludes generated columns when
+# publish_generated_columns is set to true.
+# =============================================================================

Some extra spacing and minor rewording would make this unreadable
comment readable. e.g.

# =============================================================================
# The following test cases demonstrate the behavior of generated column
# replication with publish_generated_columns set to false and true:
#
# When publish_generated_columns is set to false...
# Test: Publication column list includes generated columns
# Test: Publication column list excludes generated columns
#
# When publish_generated_columns is set to true...
# Test: Publication column list includes generated columns
# Test: Publication column list excludes generated columns
# =============================================================================


====

1st test:

4.
+# Create table and publications.
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab2 (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
+ CREATE TABLE tab3 (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
+ CREATE PUBLICATION pub1 FOR table tab2, tab3(gen1) WITH
(publish_generated_columns=false);
+));
+

4a.
/Create table/Create tables/

~

4b.
TBH, I am not sure why you are including the table 'tab2' like this,
because the test case for replication without any column list at all
was already tested in your earlier tests. AFAICT 'tab2' should also
have a column list, but one that *exlcudes* the gencols. After all,
that's what the main comment said you were going to test.

~~~

5.
+# Create table and subscription.
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE tab2 (a int, gen1 int);
+ CREATE TABLE tab3 (a int, gen1 int);
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION
pub1 WITH (copy_data = true);
+));

/Create table/Create tables/

~~~

6.
+# Verify that the initial synchronization of generated columns is not
replicated
+# when they are not included in the column list, regardless of the
+# publish_generated_columns option.
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2 ORDER BY a");
+is( $result, qq(1|
+2|),
+ 'tab2 initial sync, when publish_generated_columns=false');
+

This comment doesn't make much sense to me. E.g.
a) IIUC tab2 should have used a column list that "excludes generated
columns". After all, that's what the main comment said you were going
to test.
b) the option is already false, so saying "... regardless of the
publish_generated_columns option" doesn't really mean anything

~~~

7.
+# Verify that incremental replication of generated columns does not occur
+# when they are not included in the column list, regardless of the
+# publish_generated_columns option.
+$node_publisher->wait_for_catchup('sub1');
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab2 ORDER BY a");
+is( $result, qq(1|
+2|
+3|
+4|),
+ 'tab2 incremental replication, when publish_generated_columns=false');
+

This comment has the same issues described in the above review comment #6.

====

2nd test:

8.
+# --------------------------------------------------
+# Test Case: Even when publish_generated_columns is set to true, the publisher
+# only publishes the data of columns specified in the column list,
+# skipping other generated and non-generated columns.
+# --------------------------------------------------
+

This 2nd test has lots of the same problems as the first test.

For example:
- /# Create table and publications./# Create tables and publications./
- IMO 'tab4' should also have a publications column list, but one that
does not include gencols.
- /# Create table and subscription./# Create tables and subscription./

~~~

9.
+# Initial sync test when publish_generated_columns=true.

Why did you use a simple comment here, but a much more complicated
comment for the same scenario in the 1st test?

~~~

10.
+# Incremental replication test when publish_generated_columns=true.
+# Verify that column 'gen1' is replicated.
+$node_publisher->wait_for_catchup('sub1');
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab4 ORDER BY a");
+is( $result, qq(1|2
+2|4
+3|6
+4|8),
+ 'tab4 incremental replication, when publish_generated_columns=true');
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT * FROM tab5 ORDER BY a");
+is( $result, qq(|2
+|4
+|6
+|8),
+ 'tab5 incremental replication, when publish_generated_columns=true');
+

AFAICT the table 'tab4' also should have a column list, but one that
excludes the gencol. After all, that's what the main comment said you
were going to test. So this test comment and the tab4 test part is
currently broken.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  252. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-04T03:20:26Z

    On Thu, Oct 31, 2024 at 4:26 PM Ajin Cherian <itsajin@gmail.com> wrote:
>
> 5. Verified that publications with different column list are disallowed to be subscribed by one subscription
>    a. PUB_A(column list = (a, b)) PUB_B(no column list, with publish_generated_column) - OK
>    b. PUB_A(column list = (a, b)) PUB_B(no column list, without publish_generated_column) - FAIL
>    c.  PUB_A(no column list, without publish_generated_column) PUB_B(no column list, with publish_generated_column) - FAIL
>
> Tests did not show any unexpected behaviour.
>

Thanks for the tests, but the results of step 5 do not clearly show
whether they are correct because you haven't shared the table schema.

-- 
With Regards,
Amit Kapila.

  253. Re: Pgoutput not capturing the generated columns

    Ajin Cherian <itsajin@gmail.com> — 2024-11-04T04:30:29Z

    On Mon, Nov 4, 2024 at 2:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 31, 2024 at 4:26 PM Ajin Cherian <itsajin@gmail.com> wrote:
> >
> > 5. Verified that publications with different column list are disallowed to be subscribed by one subscription
> >    a. PUB_A(column list = (a, b)) PUB_B(no column list, with publish_generated_column) - OK
> >    b. PUB_A(column list = (a, b)) PUB_B(no column list, without publish_generated_column) - FAIL
> >    c.  PUB_A(no column list, without publish_generated_column) PUB_B(no column list, with publish_generated_column) - FAIL
> >
> > Tests did not show any unexpected behaviour.
> >
>
> Thanks for the tests, but the results of step 5 do not clearly show
> whether they are correct because you haven't shared the table schema.
>

Here are the tests:
5. Verified that publications with different column list are
disallowed to be subscribed by one subscription
   a. PUB_A(column list = (a, b)) PUB_B(no column list, with
publish_generated_column) - OK
PUB:
CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
CREATE PUBLICATION pub1 FOR table gencols with (publish_generated_columns=true);
CREATE PUBLICATION pub2 FOR table gencols(a,gen1);

SUB:
postgres=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres
host=localhost port=6972' PUBLICATION pub1, pub2;
NOTICE:  created replication slot "sub1" on publisher
CREATE SUBSCRIPTION


   b. PUB_A(column list = (a, b)) PUB_B(no column list, without
publish_generated_column) - FAIL
PUB:
CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
CREATE PUBLICATION pub1 FOR table gencols with
(publish_generated_columns=false);
CREATE PUBLICATION pub2 FOR table gencols(a,gen1);

SUB:
postgres=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres
host=localhost port=6972' PUBLICATION pub1, pub2;
ERROR:  cannot use different column lists for table "public.gencols"
in different publications

   c.  PUB_A(no column list, without publish_generated_column)
PUB_B(no column list, with publish_generated_column) - FAIL
PUB:
CREATE TABLE gencols (a int, gen1 int GENERATED ALWAYS AS (a * 2) STORED);
CREATE PUBLICATION pub1 FOR table gencols with
(publish_generated_columns=false);
CREATE PUBLICATION pub2 FOR table gencols with (publish_generated_columns=true);

SUB:
postgres=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=postgres
host=localhost port=6972' PUBLICATION pub1, pub2;
ERROR:  cannot use different column lists for table "public.gencols"
in different publications

regards,
Ajin Cherian
Fujitsu Australia

  254. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-04T04:59:58Z

    On Mon, Nov 4, 2024 at 12:28 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 31 Oct 2024 at 16:44, Ajin Cherian <itsajin@gmail.com> wrote:
> >
> >
> >
> > On Thu, Oct 31, 2024 at 9:55 PM Ajin Cherian <itsajin@gmail.com> wrote:
> >>
> >> I ran some tests and verified that the patch works with previous versions of PG12 and PG17
> >> 1. Verified with publications with generated columns and without generated columns on patched code and subscriptions on PG12 and PG17
> >> Observations:
> >>     a. If publication is created with publish_generated_columns=true or with generated columns mentioned explicitly, then tablesync will not copy generated columns but post tablesync the generated columns are replicated
> >>     b. Column list override (publish_generated_columns=false) behaviour
> >>
> >> These seem expected.
> >>
> >
> > Currently the documentation does not talk about this behaviour, I suggest this be added similar to how such a behaviour was documented when the original row-filter version was committed.
> > Suggestion:
> > "If a subscriber is a pre-18 version, the initial table synchronization won't publish generated columns even if they are defined in the publisher."
>
> The updated patch has the changes for the same.

Hi Vignesh,

Thanks for the latest doc v2 "fix" patch. Here are my review comments about it.

======
src/sgml/logical-replication.sgml

1.
    During initial data synchronization, only the published columns are
    copied.  However, if the subscriber is from a release prior to 15, then
    all the columns in the table are copied during initial data synchronization,
-   ignoring any column lists.
+   ignoring any column lists. If the subscriber is from a release prior to 18,
+   then initial table synchronization won't copy generated columns data even if
+   they are defined in the publisher.

There are some inconsistencies with the markup etc.

a) For publication row filters the text about Initial Synchronization
version differences is using SGML <Note> markup. But, for "Column
Lists" the similar text about Initial Synchronization version
differences is just plain paragraph text. So, shouldn't this also be
using a <Note> markup for better documentation consistency?

b) I also thought "even if they are defined in the publisher" wording
seems like it is referring about the table definition, but IMO it
needs to convey something more like "even when they are published"

SUGGESTION
If the subscriber is from a release prior to 18, copy pre-existing
data does not copy generated columns even when they are published.
This is because old releases ignore generated table data during the
copy.

~~

Furthermore, we will have to write something more about this in the
main patch still being developed, because the same initial sync caveat
is true even for publication of generated columns published *without*
column lists.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  255. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-04T05:26:22Z

    On Mon, Nov 4, 2024 at 10:30 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Mon, Nov 4, 2024 at 12:28 AM vignesh C <vignesh21@gmail.com> wrote:
>
> Thanks for the latest doc v2 "fix" patch. Here are my review comments about it.
>
> ======
> src/sgml/logical-replication.sgml
>
> 1.
>     During initial data synchronization, only the published columns are
>     copied.  However, if the subscriber is from a release prior to 15, then
>     all the columns in the table are copied during initial data synchronization,
> -   ignoring any column lists.
> +   ignoring any column lists. If the subscriber is from a release prior to 18,
> +   then initial table synchronization won't copy generated columns data even if
> +   they are defined in the publisher.
>
> There are some inconsistencies with the markup etc.
>
> a) For publication row filters the text about Initial Synchronization
> version differences is using SGML <Note> markup. But, for "Column
> Lists" the similar text about Initial Synchronization version
> differences is just plain paragraph text. So, shouldn't this also be
> using a <Note> markup for better documentation consistency?
>

I don't think both are comparable as the row filters section has a
separate sub-section for Initial Data Synchronization. In general, I
find the way things are described in the Column Lists sub-section more
like other parts of the documentation. Moreover, this patch has just
extended the existing docs.

> b) I also thought "even if they are defined in the publisher" wording
> seems like it is referring about the table definition, but IMO it
> needs to convey something more like "even when they are published"
>
> SUGGESTION
> If the subscriber is from a release prior to 18, copy pre-existing
> data does not copy generated columns even when they are published.
> This is because old releases ignore generated table data during the
> copy.
>

The second line says something obvious and doesn't seem to be
required. The change "even when they are published" is debatable as I
didn't read the way you read Vignesh's proposed wording, to me it was
clear what the doc is saying. I have already pushed Vignesh's version
with a minor modification.

-- 
With Regards,
Amit Kapila.

  256. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-04T10:01:09Z

    On Fri, Nov 1, 2024 at 7:10 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> ======
> doc/src/sgml/protocol.sgml
>
> 3.
>       <para>
> -      Next, one of the following submessages appears for each column:
> +      Next, one of the following submessages appears for each column
> (except generated columns):
>
> Hmm. Now that generated column replication is supported is this change
> still required?
>

How about changing it to: "Next, one of the following submessages
appears for each published column:"? This is because the column may
not be sent because either it is not in the column list or a generated
one (with publish_generated_columns as false for respective
publication).

> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 4.
> +
> +       <varlistentry
> id="sql-createpublication-params-with-publish-generated-columns">
> +        <term><literal>publish_generated_columns</literal>
> (<type>boolean</type>)</term>
> +        <listitem>
> +         <para>
> +          Specifies whether the generated columns present in the tables
> +          associated with the publication should be replicated.
> +          The default is <literal>false</literal>.
> +         </para>
> +        </listitem>
> +       </varlistentry>
> +
>
> I know that the subsequent DOCS patch V1-0002 will explain more about
> this, but as a stand-alone patch 0001 maybe you need to clarify that a
> publication column list will override this 'publish_generated_columns'
> parameter.
>

It is better to leave it to 0002 patch. But note in that patch, we
should add some reference link for the column_list behavior in the
create publication page as well.


> ======
> src/backend/catalog/pg_publication.c
>
>
> pub_getallcol_bitmapset:
>
> 6.
> +/*
> + * Return a column list bitmap for the specified table.
> + *
> + * Generated columns are included if pubgencols is true.
> + *
> + * If mcxt isn't NULL, build the bitmapset in that context.
> + */
> +Bitmapset *
> +pub_getallcol_bitmapset(Relation relation, bool pubgencols,
> + MemoryContext mcxt)
>
> IIUC this is a BMS of the table columns to be published. The function
> comment seems confusing to me when it says "column list bitmap"
> because IIUC this function is not really anything to do with a
> publication "column list", which is an entirely different thing.
>

We can probably name it pub_form_cols_map() and change the comments accordingly.

> ======
> src/backend/replication/logical/proto.c
>
> 7.
>  static void logicalrep_write_attrs(StringInfo out, Relation rel,
> -    Bitmapset *columns);
> +    Bitmapset *columns, bool pubgencols);
>  static void logicalrep_write_tuple(StringInfo out, Relation rel,
>      TupleTableSlot *slot,
> -    bool binary, Bitmapset *columns);
> +    bool binary, Bitmapset *columns,
> +    bool pubgencols);
>
> The meaning of all these new 'pubgencols' are ambiguous. e.g. Are they
> (a) The value of the CREATE PUBLICATION 'publish_generate_columns'
> parameter, or does it mean (b) Just some generated column is being
> published (maybe via column list or maybe not).
>
> I think it means (a) but, if true, that could be made much more clear
> by changing all of these names to 'pubgencols_option' or something
> similar.  Actually, now I have doubts about that also -- I think this
> might be magically assigned to false if no generated columns exist in
> the table. Anyway, please do whatever you can to disambiguate this.
>

To make it clear we can name this parameter as include_gencols.
Similarly, change the name of RelationSyncEntry's new member.
> ~~~
>
> 9.
> bool
> logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
> bool pubgencols)
> {
> if (att->attisdropped)
> return false;
>
> /*
> * Skip publishing generated columns if they are not included in the
> * column list or if the option is not specified.
> */
> if (!columns && !pubgencols && att->attgenerated)
> return false;
>
> /*
> * Check if a column is covered by a column list.
> */
> if (columns && !bms_is_member(att->attnum, columns))
> return false;
>
> return true;
> }
>
> Same as mentioned before in my previous v46-0001 review comments, I
> feel that the conditionals of this function are over-complicated and
> that there are more 'return' points than necessary. The alternative
> code below looks simpler to me.
>
> SUGGESTION
> bool
> logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
> bool pubgencols_option)
> {
>   if (att->attisdropped)
>     return false;
>
>   if (columns)
>   {
>     /*
> * Has a column list:
> * Publish only cols named in that list.
> */
>     return bms_is_member(att->attnum, columns);
>   }
>   else
>   {
>     /*
>      * Has no column list:
> * Publish generated cols only if 'publish_generated_cols' is true.
>      * Publish all non-generated cols.
>     */
>     return att->attgenerated ? pubgencols_option : true;
>   }
> }
>

Fair enough but do we need else in the above code?

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 10.
> + /* Include publishing generated columns */
> + bool pubgencols;
> +
>
> There is similar ambiguity here with this field-name as was mentioned
> about for other 'pbgencols' function params. I had initially thought
> that this this just caries around same value as the publication option
> 'publish_generated_columns' but now (after looking at function
> check_and_init_gencol) I think that might not be the case because I
> saw it can be assigned false (overriding the publication option?).
>
> Anyway, the comment needs to be made much clearer about what is the
> true meaning of this field. Or, rename it if there is a better name.
>

As suggested above, we can name it as include_gencols.

>
> send_relation_and_attrs:
>
> 12.
> - if (!logicalrep_should_publish_column(att, columns))
> + if (!logicalrep_should_publish_column(att, columns, relentry->pubgencols))
>   continue;
> It seemed a bit strange/inconsistent that 'columns' was assigned to a
> local var, but 'pubgencols' was not, given they are both fields of the
> same struct. Maybe removing this 'columns' var would be consistent
> with other code in this patch.
>

I think the other way would be better. I mean take another local
variable for this function. We don't need to always do the same in
such cases.

> ~~~
>
> 13.
> check_and_init_gencol:
>
> nit - missing periods for comments.
>
> ~~~
>
> 14.
> + /* There is no generated columns to be published *
>
> /There is no generated columns/There are no generated columns/
>
> ~~~
>
> 15.
> + foreach(lc, publications)
> + {
> + Publication *pub = lfirst(lc);
>
> AFAIK this can be re-written using a different macro to avoid needing
> the 'lc' var.
>
> ~~~
>
> pgoutput_column_list_init:
>
> 16.
> + bool collistpubexist = false;
>
> This seemed like not a very good name, How about 'found_pub_with_collist';
>
> ~~~
>
> 17.
> bool pub_no_list = true;
>
> nit - Not caused by this patch, but it's closely related; In passing
> we should declare this variable at a lower scope, and rename it to
> 'isnull' which is more in keeping with the comments around it.
>

Moving to local scope is okay but doing more than that in this patch
is not advisable even if your suggestion is a good idea which I am not
sure.


> ~
>
> 20b.
> There is a GENERAL problem that applies for lots of comments of this
> patch (including this comment) because the new publication option is
> referred to inconsistently in many different ways:
>
> e.g.
> - the generated columns option.
> - if the option is not specified
> - publish_generated_columns option.
> - the pubgencols option
> - 'publish_generated_columns' option
>
> All these references should be made the same. My personal preference
> is the last one ('publish_generated_columns' option).
>

I have responded with a better name for other places. Here, the
proposed name seems okay to me.

-- 
With Regards,
Amit Kapila.

  257. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-04T10:54:47Z

    On Wed, Oct 30, 2024 at 9:46 PM vignesh C <vignesh21@gmail.com> wrote:
>
...
+ /*
+ * For non-column list publications—such as TABLE (without a column
+ * list), ALL TABLES, or ALL TABLES IN SCHEMA publications consider
+ * all columns of the table, including generated columns, based on the
+ * pubgencols option.
+ */
+ if (!cols)
+ {
+ Assert(pub->pubgencols == entry->pubgencols);
+
+ /*
+ * Retrieve the columns if they haven't been prepared yet, or if
+ * there are multiple publications.
+ */
+ if (!relcols && (list_length(publications) > 1))
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+ relcols = pub_getallcol_bitmapset(relation, entry->pubgencols,
+   entry->entry_cxt);
+ }
+
+ cols = relcols;

Don't we need this only when generated column(s) are present, if so,
we can get that as an input to pgoutput_column_list_init()? We have
already computed that in the function check_and_init_gencol() which is
invoked just before pgoutput_column_list_init().

-- 
With Regards,
Amit Kapila.

  258. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-04T14:58:53Z

    On Fri, 1 Nov 2024 at 07:10, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
> > Thanks for committing this patch, here is a rebased version of the
> > remaining patches.
> >
>
> Hi Vignesh, thanks for the rebased patches.
>
> Here are my review comments for patch v1-0001.
>
> ======
> Commit message.
>
> 1.
> The commit message text is stale, so needs some updates.
>
> For example, it is still saying "Generated column values are not
> currently replicated..." but that is not correct anymore since the
> recent push of the previous v46-0001 patch [1], which already
> implemented replication of generated columns when they are specified
> in a publication column list..

Modified

> ======
> doc/src/sgml/ddl.sgml
>
> 2.
>       <para>
> -      Generated columns are skipped for logical replication and cannot be
> -      specified in a <command>CREATE PUBLICATION</command> column list.
> +      Generated columns are allowed to be replicated during logical replication
> +      according to the <command>CREATE PUBLICATION</command> option
> +      <link linkend="sql-createpublication-params-with-publish-generated-columns">
> +      <literal>include_generated_columns</literal></link>.
>       </para>
>
> This explanation is incomplete because generated columns can also be
> specified in a publication column list which has nothing to do with
> the new option. In fact, lack of mention about the column lists seems
> like an oversight which should have been in the previous patch [1]. I
> already posted another mail about this [2].

Modified

> ======
> doc/src/sgml/protocol.sgml
>
> 3.
>       <para>
> -      Next, one of the following submessages appears for each column:
> +      Next, one of the following submessages appears for each column
> (except generated columns):
>
> Hmm. Now that generated column replication is supported is this change
> still required?

Modified

> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 4.
> +
> +       <varlistentry
> id="sql-createpublication-params-with-publish-generated-columns">
> +        <term><literal>publish_generated_columns</literal>
> (<type>boolean</type>)</term>
> +        <listitem>
> +         <para>
> +          Specifies whether the generated columns present in the tables
> +          associated with the publication should be replicated.
> +          The default is <literal>false</literal>.
> +         </para>
> +        </listitem>
> +       </varlistentry>
> +
>
> I know that the subsequent DOCS patch V1-0002 will explain more about
> this, but as a stand-alone patch 0001 maybe you need to clarify that a
> publication column list will override this 'publish_generated_columns'
> parameter.

I felt it is better to keep it in 0002 patch itself.

>
> ======
> src/backend/catalog/pg_publication.c
>
> has_column_list_defined:
>
> 5.
> +/*
> + * Returns true if the relation has column list associated with the
> publication,
> + * false otherwise.
> + */
> +bool
> +has_column_list_defined(Publication *pub, Oid relid)
> +{
> + HeapTuple cftuple = NULL;
> + bool isnull = true;
> +
> + if (pub->alltables)
> + return false;
> +
> + cftuple = SearchSysCache2(PUBLICATIONRELMAP,
> +   ObjectIdGetDatum(relid),
> +   ObjectIdGetDatum(pub->oid));
> + if (HeapTupleIsValid(cftuple))
> + {
> + /* Lookup the column list attribute. */
> + (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
> +    Anum_pg_publication_rel_prattrs,
> +    &isnull);
> + if (!isnull)
> + {
> + ReleaseSysCache(cftuple);
> + return true;
> + }
> +
> + ReleaseSysCache(cftuple);
> + }
> +
> + return false;
> +}
> +
>
> 5a.
> It might be tidier if you checked for !HeapTupleIsValid(cftuple) and
> do early return false, instead of needing an indented if-block.

I preferred the existing, I did not see any advantage by doing this.

> ~
>
> 5b.
> The code can be rearranged and simplified -- you don't need multiple
> calls to ReleaseSysCache.
>
> SUGGESTION:
>
> /* Lookup the column list attribute. */
> (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
>   Anum_pg_publication_rel_prattrs,
>   &isnull);
>
> ReleaseSysCache(cftuple);
> /* Was a column list found? */
> return isnull ? false : true;

Modified

> ~~~
>
> pub_getallcol_bitmapset:
>
> 6.
> +/*
> + * Return a column list bitmap for the specified table.
> + *
> + * Generated columns are included if pubgencols is true.
> + *
> + * If mcxt isn't NULL, build the bitmapset in that context.
> + */
> +Bitmapset *
> +pub_getallcol_bitmapset(Relation relation, bool pubgencols,
> + MemoryContext mcxt)
>
> IIUC this is a BMS of the table columns to be published. The function
> comment seems confusing to me when it says "column list bitmap"
> because IIUC this function is not really anything to do with a
> publication "column list", which is an entirely different thing.

Changed the function name and updated comments based on Amit's
suggestion from [1]

> ======
> src/backend/replication/logical/proto.c
>
> 7.
>  static void logicalrep_write_attrs(StringInfo out, Relation rel,
> -    Bitmapset *columns);
> +    Bitmapset *columns, bool pubgencols);
>  static void logicalrep_write_tuple(StringInfo out, Relation rel,
>      TupleTableSlot *slot,
> -    bool binary, Bitmapset *columns);
> +    bool binary, Bitmapset *columns,
> +    bool pubgencols);
>
> The meaning of all these new 'pubgencols' are ambiguous. e.g. Are they
> (a) The value of the CREATE PUBLICATION 'publish_generate_columns'
> parameter, or does it mean (b) Just some generated column is being
> published (maybe via column list or maybe not).
>
> I think it means (a) but, if true, that could be made much more clear
> by changing all of these names to 'pubgencols_option' or something
> similar.  Actually, now I have doubts about that also -- I think this
> might be magically assigned to false if no generated columns exist in
> the table. Anyway, please do whatever you can to disambiguate this.

Changed it to include_gencols based on Amit's suggestion from [1]

> ~~~
>
> logicalrep_should_publish_column:
>
> 8.
> The function comment is stale. It is still only talking about
> generated columns in column lists.
>
> SUGGESTION
> Note that generated columns can be published only when present in a
> publication column list, or (if there is no column list), when the
> publication parameter 'publish_generated_columns' is true.

Modified

> ~~~
>
> 9.
> bool
> logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
> bool pubgencols)
> {
> if (att->attisdropped)
> return false;
>
> /*
> * Skip publishing generated columns if they are not included in the
> * column list or if the option is not specified.
> */
> if (!columns && !pubgencols && att->attgenerated)
> return false;
>
> /*
> * Check if a column is covered by a column list.
> */
> if (columns && !bms_is_member(att->attnum, columns))
> return false;
>
> return true;
> }
>
> Same as mentioned before in my previous v46-0001 review comments, I
> feel that the conditionals of this function are over-complicated and
> that there are more 'return' points than necessary. The alternative
> code below looks simpler to me.
>
> SUGGESTION
> bool
> logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
> bool pubgencols_option)
> {
>   if (att->attisdropped)
>     return false;
>
>   if (columns)
>   {
>     /*
> * Has a column list:
> * Publish only cols named in that list.
> */
>     return bms_is_member(att->attnum, columns);
>   }
>   else
>   {
>     /*
>      * Has no column list:
> * Publish generated cols only if 'publish_generated_cols' is true.
>      * Publish all non-generated cols.
>     */
>     return att->attgenerated ? pubgencols_option : true;
>   }
> }

Modified

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 10.
> + /* Include publishing generated columns */
> + bool pubgencols;
> +
>
> There is similar ambiguity here with this field-name as was mentioned
> about for other 'pbgencols' function params. I had initially thought
> that this this just caries around same value as the publication option
> 'publish_generated_columns' but now (after looking at function
> check_and_init_gencol) I think that might not be the case because I
> saw it can be assigned false (overriding the publication option?).
>
> Anyway, the comment needs to be made much clearer about what is the
> true meaning of this field. Or, rename it if there is a better name.

Changed the variable to avoid confusion.

> ~~~
>
> 11.
> +static void send_relation_and_attrs(Relation relation, TransactionId xid,
> + LogicalDecodingContext *ctx,
> + RelationSyncEntry *relentry);
>
> Was there some reason to move this static? Maybe it is better just to
> change the existing static in-place rather than moving code around at
> the same time.

This function now requires the RelationSyncEntry parameter to publish
generated columns. Since this structure was defined after the previous
function prototype, it has been moved below the prototype.

> ~~~
>
> send_relation_and_attrs:
>
> 12.
> - if (!logicalrep_should_publish_column(att, columns))
> + if (!logicalrep_should_publish_column(att, columns, relentry->pubgencols))
>   continue;
> It seemed a bit strange/inconsistent that 'columns' was assigned to a
> local var, but 'pubgencols' was not, given they are both fields of the
> same struct. Maybe removing this 'columns' var would be consistent
> with other code in this patch.

Modified to use a local variable.

> ~~~
>
> 13.
> check_and_init_gencol:
>
> nit - missing periods for comments.

Modified

> ~~~
>
> 14.
> + /* There is no generated columns to be published *
>
> /There is no generated columns/There are no generated columns/

Modified

> ~~~
>
> 15.
> + foreach(lc, publications)
> + {
> + Publication *pub = lfirst(lc);
>
> AFAIK this can be re-written using a different macro to avoid needing
> the 'lc' var.

Modified

> ~~~
>
> pgoutput_column_list_init:
>
> 16.
> + bool collistpubexist = false;
>
> This seemed like not a very good name, How about 'found_pub_with_collist';

Modified

> ~~~
>
> 17.
> bool pub_no_list = true;
>
> nit - Not caused by this patch, but it's closely related; In passing
> we should declare this variable at a lower scope, and rename it to
> 'isnull' which is more in keeping with the comments around it.

I was not sure if this should be done in this patch, there was a
similar opinion from Amit also at [1].

> ~~~
>
> 18.
> + /*
> + * For non-column list publications—such as TABLE (without a column
> + * list), ALL TABLES, or ALL TABLES IN SCHEMA publications consider
> + * all columns of the table, including generated columns, based on the
> + * pubgencols option.
> + */
>
> Some minor tweaks.
>
> SUGGESTION
> For non-column list publications — e.g. TABLE (without a column list),
> ALL TABLES, or ALL TABLES IN SCHEMA, we consider all columns of the
> table (including generated columns when 'publish_generated_columns'
> option is true).

Modified

> ~~~
>
> 19.
> + Assert(pub->pubgencols == entry->pubgencols);
> +
> + /*
> + * Retrieve the columns if they haven't been prepared yet, or if
> + * there are multiple publications.
> + */
> + if (!relcols && (list_length(publications) > 1))
> + {
> + pgoutput_ensure_entry_cxt(data, entry);
> + relcols = pub_getallcol_bitmapset(relation, entry->pubgencols,
> +   entry->entry_cxt);
> + }
>
> 19a.
> Is that Assert correct? I ask only because AFAICT I saw in previous
> function (check_and_init_gencol) there is code that might change
> entry->pubgencols = false; even if the 'publish_generated_columns'
> option is true but there were not generated columns found in the
> table.

Removed Assert as we don't set the entry->pubgencols to the same value
as pubgencols when the table has no generated columns.

> ~
>
> 19b.
> The comment says "or if there are multiple publications" but the code
> says &&. Something seems wrong.

Modified the comment

> ~~~
>
> 20.
> + /*
> + * If no column list publications exit, columns will be selected later
> + * according to the generated columns option.
> + */
>
> 20a.
> typo - /exit/exist/

Modified

> ~
>
> 20b.
> There is a GENERAL problem that applies for lots of comments of this
> patch (including this comment) because the new publication option is
> referred to inconsistently in many different ways:
>
> e.g.
> - the generated columns option.
> - if the option is not specified
> - publish_generated_columns option.
> - the pubgencols option
> - 'publish_generated_columns' option
>
> All these references should be made the same. My personal preference
> is the last one ('publish_generated_columns' option).

Modified

> ~~~
>
> get_rel_sync_entry:
>
> 21.
> + /* Check whether to publish to generated columns. */
> + check_and_init_gencol(data, rel_publications, entry);
> +
>
> typo in comment - "publishe to"?

Modified

> ======
> src/include/catalog/pg_publication.h
>
> 22.
>  extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
>      MemoryContext mcxt);
> +extern Bitmapset *pub_getallcol_bitmapset(Relation relation, bool pubgencols,
> +   MemoryContext mcxt);
>
> Maybe a better name for this new function is 'pub_allcols_bitmapse'.
> That would then be consistent with the other BMS function which
> doesn't include the word "get".

Changed it to  pub_form_cols_map

The attached v47 version patch has the changes for the same.

[1] - https://www.postgresql.org/message-id/CAA4eK1%2B%2Bae6KqKv35b%2BQODPn1PxuiGCnL-B_m4T678XfLX0qXw%40mail.gmail.com

Regards,
Vignesh

  259. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-04T15:11:55Z

    On Mon, 4 Nov 2024 at 16:25, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 30, 2024 at 9:46 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> ...
> + /*
> + * For non-column list publications—such as TABLE (without a column
> + * list), ALL TABLES, or ALL TABLES IN SCHEMA publications consider
> + * all columns of the table, including generated columns, based on the
> + * pubgencols option.
> + */
> + if (!cols)
> + {
> + Assert(pub->pubgencols == entry->pubgencols);
> +
> + /*
> + * Retrieve the columns if they haven't been prepared yet, or if
> + * there are multiple publications.
> + */
> + if (!relcols && (list_length(publications) > 1))
> + {
> + pgoutput_ensure_entry_cxt(data, entry);
> + relcols = pub_getallcol_bitmapset(relation, entry->pubgencols,
> +   entry->entry_cxt);
> + }
> +
> + cols = relcols;
>
> Don't we need this only when generated column(s) are present, if so,
> we can get that as an input to pgoutput_column_list_init()?

We will use this in all cases i.e. irrespective of generated columns present:
ex:
CREATE TABLE t1(c1 int, c2 int);
create publication pub1 for table t1(c1);
create publication pub2 for table t1;

Create subscription ... publication pub1,pub2;

Even in this case we will have to identify that column list is not
matching and throw:
2024-11-04 20:35:58.199 IST [492190] 492190 sub1 ERROR:  cannot use
different column lists for table "public.t1" in different publications

Regards,
Vignesh

  260. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-04T15:16:25Z

    On Fri, 1 Nov 2024 at 09:23, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Thu, Oct 31, 2024 at 3:16 AM vignesh C <vignesh21@gmail.com> wrote:
>
> > Thanks for committing this patch, here is a rebased version of the
> > remaining patches.
> >
>
> Hi Vignesh.
>
> Here are my review comments for the docs patch v1-0002.
>
> ======
> Commit message
>
> 1.
> This patch updates docs to describe the new feature allowing
> replication of generated
> columns. This includes addition of a new section "Generated Column
> Replication" to the
> "Logical Replication" documentation chapter.
>
> ~
>
> That first sentence was correct previously when this patch contained
> *all* the gencols documentation, but now some of the feature docs are
> already handled by previous patches, so the first sentence can be
> removed.
>
> Now patch 0002 is only for adding the new chapter, plus the references to it.
>
> ~
>
> /This includes addition of a new section/This patch adds a new section/

Modified

> ======
> doc/src/sgml/protocol.sgml
>
> 2.
>       <para>
> -      Next, one of the following submessages appears for each column
> (except generated columns):
> +      Next, one of the following submessages appears for each column:
>
> AFAIK this simply cancels out a change from the v1-0001 patch which
> IMO should have not been there in the first place. Please refer to my
> v1-0001 review for the same.

Removed it.

The changes for the same are available at v47 version patch attached
at [1]. I have not included the 0003 patch for now, I will include
once these two patch stabilizes.
[1] - https://www.postgresql.org/message-id/CALDaNm2sNfZoFfqOKq9GAjQZd3isqosij9iHaJjn7oQVmLLNYw%40mail.gmail.com

Regards,
Vignesh

  261. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-05T01:29:59Z

    Hi Vignesh,

Here are my review comments for your latest patch v47-0001.

======
doc/src/sgml/ddl.sgml

1.
      <para>
-      Generated columns can be replicated during logical replication by
-      including them in the column list of the
-      <command>CREATE PUBLICATION</command> command.
+      Generated columns are allowed to be replicated during logical replication
+      according to the <command>CREATE PUBLICATION</command> option
+      <link linkend="sql-createpublication-params-with-publish-generated-columns">
+      <literal>include_generated_columns</literal></link> or by including them
+      in the column list of the <command>CREATE PUBLICATION</command> command.
      </para>

1a.
This text gives the wrong name for the new parameter.
/include_generated_columns/publish_generated_columns/

~

1b.
Everywhere in this patch (except here), this is called the
'publish_generated_columns' parameter (not "option") so it should be
called a parameter here also. Anyway, apparently that is the docs rule
-- see [1].

BTW, the same applies for the commit message 1st line of this patch:
[PATCH v47 1/2] Enable support for 'publish_generated_columns' option.
Should be
[PATCH v47 1/2] Enable support for 'publish_generated_columns' parameter.

======
doc/src/sgml/protocol.sgml

2.
-      Next, one of the following submessages appears for each column:
+      Next, one of the following submessages appears for each published column:

The change is OK. But, note that there are other descriptions just
like this one on the same page, so if you are going to say "published"
here, then to be consistent you probably want to consider updating the
other places as well.

======
src/backend/catalog/pg_publication.c

3.
+bool
+has_column_list_defined(Publication *pub, Oid relid)
+{
+ HeapTuple cftuple = NULL;
+ bool isnull = true;

Since you chose not to rearrange the HeapTupleIsValid check, this
'isnull' declaration should be relocated within the if-block.

======
src/backend/replication/logical/proto.c

4.
 /*
  * Check if the column 'att' of a table should be published.
  *
- * 'columns' represents the column list specified for that table in the
- * publication.
+ * 'columns' represents the publication column list (if any) for that table.
  *
- * Note that generated columns can be present only in 'columns' list.
+ * Note that generated columns can be published only when present in a
+ * publication column list, or when include_gencols is true.
  */
 bool
-logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
+logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
+ bool include_gencols)

The function comment describes 'columns' but it doesn't describe
'include_gencols'. I think knowing more about that parameter would be
helpful.

SUGGESTION:
The 'include_gencols' flag indicates whether generated columns should
be published when there is no column list. Typically, this will have
the same value as the 'publish_generated_columns' publication
parameter.

======
src/backend/replication/logical/relation.c

5.
@@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode)
  int attnum;
  Form_pg_attribute attr = TupleDescAttr(desc, i);

- if (attr->attisdropped || attr->attgenerated)
+ if (attr->attisdropped)
  {
  entry->attrmap->attnums[i] = -1;
  continue;
@@ -432,7 +432,15 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode)

  entry->attrmap->attnums[i] = attnum;
  if (attnum >= 0)
+ {
+ if (attr->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("replicating to a target relation's generated column \"%s\"
for \"%s.%s\" is not supported",
+    NameStr(attr->attname), remoterel->nspname, remoterel->relname));
+
  missingatts = bms_del_member(missingatts, attnum);
+ }

Hmm. I think this more descriptive error is a good improvement over
the previous "missing" error, but I just don't think it belongs in
this patch. This is impacting the existing "regular" ==> "generated"
replication as well, which seems out-of-scope for this gencols patch.

IMO this ought to be made as a separate patch that can be pushed to
master separately/independently *before* any of this new gencols
stuff.

Also, you already said in the commit message:
* Publisher not-generated column => subscriber generated column:
  This will give ERROR (not changed by this patch).

So the "not changed by this patch" part is not true if these changes
are included.

======
src/backend/replication/pgoutput/pgoutput.c

6.
+ /*
+ * Include publishing generated columns if 'publish_generated_columns'
+ * parameter is set to true, this will be set only if the relation
+ * contains any generated column.
+ */
+ bool include_gencols;
+

Minor rewording.

SUGGESTION:
Include generated columns for publication is set true if
'publish_generated_columns' parameter is true, and the relation
contains generated columns.

~~~

7.
+ /*
+ * Retrieve the columns if they haven't been prepared yet, and
+ * only if multiple publications exist.
+ */
+ if (!relcols && (list_length(publications) > 1))
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+ relcols = pub_form_cols_map(relation, entry->include_gencols,
+ entry->entry_cxt);
+ }

IIUC the purpose of this is for ensuring that the column lists are
consistent across all publications. That is why we only do this when
there are > 1 publications. For the 1st publication with no column
list we cache all the columns (in 'relcols') so later the cols of the
*current* publication (in 'cols') can be checked to see if they are
the same.

TBH, I think this part needs to have more explanation because it's a
bit too subtle; you have to read between the lines to figure out what
it is doing instead of just having a comment to clearly describe the
logic up-front.

======
[1] option versus parameter -
https://www.postgresql.org/message-id/CAKFQuwZVJ%2B_Z0pMX%3DBBKF9A6skVqiv89gxEgFOX7cwtWJj-Ccw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  262. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-05T02:24:33Z

    Hi Vignesh,

Here are my review comments for the v47-0002 (DOCS) patch.

======
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 577bcb4b71..a13f19bdbe 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -517,7 +517,8 @@ CREATE TABLE people (
       Generated columns are allowed to be replicated during logical replication
       according to the <command>CREATE PUBLICATION</command> option
       <link linkend="sql-createpublication-params-with-publish-generated-columns">
-      <literal>include_generated_columns</literal></link>.
+      <literal>include_generated_columns</literal></link>. See
+      <xref linkend="logical-replication-gencols"/> for details.
      </para>
     </listitem>
    </itemizedlist>

Previously (in v1-0002) above there was a link to the new gencols
section ("See XXX for details"), but in v47 that link is no longer
included. Why not?

======
doc/src/sgml/ref/create_publication.sgml

-      lists.
+      lists. See <xref linkend="logical-replication-gencols-howto"/> for more
+      information on the logical replication of generated columns using a
+      column list publication.
      </para>

I don't really think this change is necessary.

The existing paragraph already says "When a column list is specified,
only the named columns are replicated.", so there is nothing special
more than that which we really need to say for generated columns.

Also, this paragraph already has a link to the "Column List" chapter
for more details, so if the user really wants to learn about column
lists which happen to have generated columns in them, then that's
where they should look. and there is a link to the new chapter 29.6
from there.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  263. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-05T05:13:05Z

    On Tue, Nov 5, 2024 at 7:00 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
>
> ~
>
> 1b.
> Everywhere in this patch (except here), this is called the
> 'publish_generated_columns' parameter (not "option") so it should be
> called a parameter here also. Anyway, apparently that is the docs rule
> -- see [1].
>

In the thread you linked, we have decided to name 'failover' an
option. I feel the same should be followed here but I agree that we
should spell it consistently throughout the patch.

>
> ======
> doc/src/sgml/protocol.sgml
>
> 2.
> -      Next, one of the following submessages appears for each column:
> +      Next, one of the following submessages appears for each published column:
>
> The change is OK. But, note that there are other descriptions just
> like this one on the same page, so if you are going to say "published"
> here, then to be consistent you probably want to consider updating the
> other places as well.
>

Are you referring to the existing message: "Next, the following
message part appears for each column included in the publication:"? If
so, we can change it to make it the same but the current one also
looks okay. We can consider changing it separately if required after
this patch.

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 6.
> + /*
> + * Include publishing generated columns if 'publish_generated_columns'
> + * parameter is set to true, this will be set only if the relation
> + * contains any generated column.
> + */
> + bool include_gencols;
> +
>
> Minor rewording.
>
> SUGGESTION:
> Include generated columns for publication is set true if
>

/set true/set to true

>
> ======
> [1] option versus parameter -
> https://www.postgresql.org/message-id/CAKFQuwZVJ%2B_Z0pMX%3DBBKF9A6skVqiv89gxEgFOX7cwtWJj-Ccw%40mail.gmail.com
>

-- 
With Regards,
Amit Kapila.

  264. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-05T05:47:02Z

    On Tue, 5 Nov 2024 at 07:00, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are my review comments for your latest patch v47-0001.
>
> ======
> doc/src/sgml/ddl.sgml
>
> 1.
>       <para>
> -      Generated columns can be replicated during logical replication by
> -      including them in the column list of the
> -      <command>CREATE PUBLICATION</command> command.
> +      Generated columns are allowed to be replicated during logical replication
> +      according to the <command>CREATE PUBLICATION</command> option
> +      <link linkend="sql-createpublication-params-with-publish-generated-columns">
> +      <literal>include_generated_columns</literal></link> or by including them
> +      in the column list of the <command>CREATE PUBLICATION</command> command.
>       </para>
>
> 1a.
> This text gives the wrong name for the new parameter.
> /include_generated_columns/publish_generated_columns/
>
> ~
>
> 1b.
> Everywhere in this patch (except here), this is called the
> 'publish_generated_columns' parameter (not "option") so it should be
> called a parameter here also. Anyway, apparently that is the docs rule
> -- see [1].
>
> BTW, the same applies for the commit message 1st line of this patch:
> [PATCH v47 1/2] Enable support for 'publish_generated_columns' option.
> Should be
> [PATCH v47 1/2] Enable support for 'publish_generated_columns' parameter.

Modified to keep it consistent

> ======
> doc/src/sgml/protocol.sgml
>
> 2.
> -      Next, one of the following submessages appears for each column:
> +      Next, one of the following submessages appears for each published column:
>
> The change is OK. But, note that there are other descriptions just
> like this one on the same page, so if you are going to say "published"
> here, then to be consistent you probably want to consider updating the
> other places as well.

This can be done later after this patch is committed

> ======
> src/backend/catalog/pg_publication.c
>
> 3.
> +bool
> +has_column_list_defined(Publication *pub, Oid relid)
> +{
> + HeapTuple cftuple = NULL;
> + bool isnull = true;
>
> Since you chose not to rearrange the HeapTupleIsValid check, this
> 'isnull' declaration should be relocated within the if-block.

Modified

> ======
> src/backend/replication/logical/proto.c
>
> 4.
>  /*
>   * Check if the column 'att' of a table should be published.
>   *
> - * 'columns' represents the column list specified for that table in the
> - * publication.
> + * 'columns' represents the publication column list (if any) for that table.
>   *
> - * Note that generated columns can be present only in 'columns' list.
> + * Note that generated columns can be published only when present in a
> + * publication column list, or when include_gencols is true.
>   */
>  bool
> -logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> +logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
> + bool include_gencols)
>
> The function comment describes 'columns' but it doesn't describe
> 'include_gencols'. I think knowing more about that parameter would be
> helpful.
>
> SUGGESTION:
> The 'include_gencols' flag indicates whether generated columns should
> be published when there is no column list. Typically, this will have
> the same value as the 'publish_generated_columns' publication
> parameter.

Modified

> ======
> src/backend/replication/logical/relation.c
>
> 5.
> @@ -421,7 +421,7 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
> LOCKMODE lockmode)
>   int attnum;
>   Form_pg_attribute attr = TupleDescAttr(desc, i);
>
> - if (attr->attisdropped || attr->attgenerated)
> + if (attr->attisdropped)
>   {
>   entry->attrmap->attnums[i] = -1;
>   continue;
> @@ -432,7 +432,15 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
> LOCKMODE lockmode)
>
>   entry->attrmap->attnums[i] = attnum;
>   if (attnum >= 0)
> + {
> + if (attr->attgenerated)
> + ereport(ERROR,
> + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("replicating to a target relation's generated column \"%s\"
> for \"%s.%s\" is not supported",
> +    NameStr(attr->attname), remoterel->nspname, remoterel->relname));
> +
>   missingatts = bms_del_member(missingatts, attnum);
> + }
>
> Hmm. I think this more descriptive error is a good improvement over
> the previous "missing" error, but I just don't think it belongs in
> this patch. This is impacting the existing "regular" ==> "generated"
> replication as well, which seems out-of-scope for this gencols patch.
>
> IMO this ought to be made as a separate patch that can be pushed to
> master separately/independently *before* any of this new gencols
> stuff.
>
> Also, you already said in the commit message:
> * Publisher not-generated column => subscriber generated column:
>   This will give ERROR (not changed by this patch).
>
> So the "not changed by this patch" part is not true if these changes
> are included.

That makes sense, I will post a separate patch for this after this work is done.

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 6.
> + /*
> + * Include publishing generated columns if 'publish_generated_columns'
> + * parameter is set to true, this will be set only if the relation
> + * contains any generated column.
> + */
> + bool include_gencols;
> +
>
> Minor rewording.
>
> SUGGESTION:
> Include generated columns for publication is set true if
> 'publish_generated_columns' parameter is true, and the relation
> contains generated columns.

Modified

> ~~~
>
> 7.
> + /*
> + * Retrieve the columns if they haven't been prepared yet, and
> + * only if multiple publications exist.
> + */
> + if (!relcols && (list_length(publications) > 1))
> + {
> + pgoutput_ensure_entry_cxt(data, entry);
> + relcols = pub_form_cols_map(relation, entry->include_gencols,
> + entry->entry_cxt);
> + }
>
> IIUC the purpose of this is for ensuring that the column lists are
> consistent across all publications. That is why we only do this when
> there are > 1 publications. For the 1st publication with no column
> list we cache all the columns (in 'relcols') so later the cols of the
> *current* publication (in 'cols') can be checked to see if they are
> the same.
>
> TBH, I think this part needs to have more explanation because it's a
> bit too subtle; you have to read between the lines to figure out what
> it is doing instead of just having a comment to clearly describe the
> logic up-front.

Added comments.

The attached v48 version has the changes for the same.

Regards,
Vignesh

  265. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-05T05:48:41Z

    On Tue, 5 Nov 2024 at 07:55, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are my review comments for the v47-0002 (DOCS) patch.
>
> ======
> diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
> index 577bcb4b71..a13f19bdbe 100644
> --- a/doc/src/sgml/ddl.sgml
> +++ b/doc/src/sgml/ddl.sgml
> @@ -517,7 +517,8 @@ CREATE TABLE people (
>        Generated columns are allowed to be replicated during logical replication
>        according to the <command>CREATE PUBLICATION</command> option
>        <link linkend="sql-createpublication-params-with-publish-generated-columns">
> -      <literal>include_generated_columns</literal></link>.
> +      <literal>include_generated_columns</literal></link>. See
> +      <xref linkend="logical-replication-gencols"/> for details.
>       </para>
>      </listitem>
>     </itemizedlist>
>
> Previously (in v1-0002) above there was a link to the new gencols
> section ("See XXX for details"), but in v47 that link is no longer
> included. Why not?

Included it now.

> ======
> doc/src/sgml/ref/create_publication.sgml
>
> -      lists.
> +      lists. See <xref linkend="logical-replication-gencols-howto"/> for more
> +      information on the logical replication of generated columns using a
> +      column list publication.
>       </para>
>
> I don't really think this change is necessary.
>
> The existing paragraph already says "When a column list is specified,
> only the named columns are replicated.", so there is nothing special
> more than that which we really need to say for generated columns.
>
> Also, this paragraph already has a link to the "Column List" chapter
> for more details, so if the user really wants to learn about column
> lists which happen to have generated columns in them, then that's
> where they should look. and there is a link to the new chapter 29.6
> from there.

Removed it.

The v48 version patch attached at [1] has the changes for the same.

[1] - https://www.postgresql.org/message-id/CALDaNm3Ha5t9bOLJ7OBnaMRgYHX_Q4j9k3EbRsX%3D%2B1mxUo5BZw%40mail.gmail.com

Regards,
Vignesh

  266. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-05T07:02:24Z

    Hi Vignesh,

Here are my review comments for patch v48-0001.

======
src/backend/catalog/pg_publication.c

has_column_list_defined:

1.
+ if (HeapTupleIsValid(cftuple))
+ {
+ bool isnull = true;
+
+ /* Lookup the column list attribute. */
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+    Anum_pg_publication_rel_prattrs,
+    &isnull);

AFAIK it is not necessary to assign a default value to 'isnull' here.
e.g. most of the other 100s of calls to SysCacheGetAttr elsewhere in
PostgreSQL source don't bother to do this.

//////////

I also checked the docs patch v48-0002. That now looks good to me.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  267. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-05T10:55:35Z

    On Tue, Nov 5, 2024 at 12:32 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> has_column_list_defined:
>
> 1.
> + if (HeapTupleIsValid(cftuple))
> + {
> + bool isnull = true;
> +
> + /* Lookup the column list attribute. */
> + (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
> +    Anum_pg_publication_rel_prattrs,
> +    &isnull);
>
> AFAIK it is not necessary to assign a default value to 'isnull' here.
> e.g. most of the other 100s of calls to SysCacheGetAttr elsewhere in
> PostgreSQL source don't bother to do this.
>

Can we try to reuse this new function has_column_list_defined() in
pgoutput_column_list_init() where we are trying to check and form a
column list? For that, we need to change this function such that it
also returns a column list when requested.

Some more comments:
1.
 extern bool is_schema_publication(Oid pubid);
+extern bool has_column_list_defined(Publication *pub, Oid relid);

The order of declaration doesn't match with its definition in .c file.
It would be better to define this function after
is_schema_publication().

2.
+ * 'include_gencols' flag indicates whether generated columns should be
+ * published when there is no column list. Typically, this will have the same
+ * value as the 'publish_generated_columns' publication parameter.
+ *
+ * Note that generated columns can be published only when present in a
+ * publication column list, or when include_gencols is true.
  */
 bool
-logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
+logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
+ bool include_gencols)
 {
  if (att->attisdropped)
  return false;

- /*
- * Skip publishing generated columns if they are not included in the
- * column list.
- */
- if (!columns && att->attgenerated)
- return false;
+ /* If a column list is provided, publish only the cols in that list. */
+ if (columns)
+ return bms_is_member(att->attnum, columns);

  /*
- * Check if a column is covered by a column list.
+ * If no column list is provided, generated columns will be published only
+ * if include_gencols is true, while all non-generated columns will always
+ * be published.

The similar information is repeated multiple times in different words.
I have adjusted the comments in this part of the code to avoid
repetition.

I have changed comments at a few other places in the patch. See attached.

-- 
With Regards,
Amit Kapila.

  268. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-05T14:39:56Z

    On Tue, 5 Nov 2024 at 16:25, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Nov 5, 2024 at 12:32 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > has_column_list_defined:
> >
> > 1.
> > + if (HeapTupleIsValid(cftuple))
> > + {
> > + bool isnull = true;
> > +
> > + /* Lookup the column list attribute. */
> > + (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
> > +    Anum_pg_publication_rel_prattrs,
> > +    &isnull);
> >
> > AFAIK it is not necessary to assign a default value to 'isnull' here.
> > e.g. most of the other 100s of calls to SysCacheGetAttr elsewhere in
> > PostgreSQL source don't bother to do this.
> >
>
> Can we try to reuse this new function has_column_list_defined() in
> pgoutput_column_list_init() where we are trying to check and form a
> column list? For that, we need to change this function such that it
> also returns a column list when requested.

Modified

> Some more comments:
> 1.
>  extern bool is_schema_publication(Oid pubid);
> +extern bool has_column_list_defined(Publication *pub, Oid relid);
>
> The order of declaration doesn't match with its definition in .c file.
> It would be better to define this function after
> is_schema_publication().

Modified

> 2.
> + * 'include_gencols' flag indicates whether generated columns should be
> + * published when there is no column list. Typically, this will have the same
> + * value as the 'publish_generated_columns' publication parameter.
> + *
> + * Note that generated columns can be published only when present in a
> + * publication column list, or when include_gencols is true.
>   */
>  bool
> -logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns)
> +logicalrep_should_publish_column(Form_pg_attribute att, Bitmapset *columns,
> + bool include_gencols)
>  {
>   if (att->attisdropped)
>   return false;
>
> - /*
> - * Skip publishing generated columns if they are not included in the
> - * column list.
> - */
> - if (!columns && att->attgenerated)
> - return false;
> + /* If a column list is provided, publish only the cols in that list. */
> + if (columns)
> + return bms_is_member(att->attnum, columns);
>
>   /*
> - * Check if a column is covered by a column list.
> + * If no column list is provided, generated columns will be published only
> + * if include_gencols is true, while all non-generated columns will always
> + * be published.
>
> The similar information is repeated multiple times in different words.
> I have adjusted the comments in this part of the code to avoid
> repetition.

I have taken the changes

> I have changed comments at a few other places in the patch. See attached.

I have taken the changes

The attached v49 patch has the changes for the same.

Regards,
Vignesh

  269. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-05T14:41:24Z

    On Tue, 5 Nov 2024 at 12:32, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are my review comments for patch v48-0001.
>
> ======
> src/backend/catalog/pg_publication.c
>
> has_column_list_defined:
>
> 1.
> + if (HeapTupleIsValid(cftuple))
> + {
> + bool isnull = true;
> +
> + /* Lookup the column list attribute. */
> + (void) SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
> +    Anum_pg_publication_rel_prattrs,
> +    &isnull);
>
> AFAIK it is not necessary to assign a default value to 'isnull' here.
> e.g. most of the other 100s of calls to SysCacheGetAttr elsewhere in
> PostgreSQL source don't bother to do this.

This is fixed in the v49 version patch attached at [1].

[1] - https://www.postgresql.org/message-id/CALDaNm3XV5mAeZzZMkOPSPieANMaxOH8xAydLqf8X5PQn%2Ba5EA%40mail.gmail.com

Regards,
Vignesh

  270. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-06T02:04:15Z

    Hi Vignesh,

Here are my review comments for patch v49-0001.

======
src/backend/catalog/pg_publication.c

1. check_fetch_column_list

+bool
+check_fetch_column_list(Publication *pub, Oid relid, MemoryContext mcxt,
+ Bitmapset **cols)
+{
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+ bool found = false;
+

1a.
The 'cftuple' is unconditionally assigned; the default assignment
seems unnecessary.

~

1b.
The 'cfdatum' can be declared in a lower scope (in the if-block).
The 'cfdatum' is unconditionally assigned; the default assignment
seems unnecessary.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  271. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-06T04:26:18Z

    On Wed, Nov 6, 2024 at 7:34 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are my review comments for patch v49-0001.
>

I have a question on the display of this new parameter.

postgres=# \dRp+
                                      Publication pub_gen
  Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
root | Generated columns
----------+------------+---------+---------+---------+-----------+----------+-------------------
 KapilaAm | f          | t       | t       | t       | t         | f        | t
Tables:
    "public.test_gen"

The current theory for the display of the "Generated Columns" option
could be that let's add new parameters at the end which sounds
reasonable. The other way to look at it is how it would be easier for
users to interpret. I think the value of the "Via root" option should
be either after "All tables" or at the end as that is higher level
table information than operations or column-level information. As
currently, it is at the end, so "Generated Columns" should be added
before.

Thoughts?

-- 
With Regards,
Amit Kapila.

  272. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-06T04:55:48Z

    On Wed, Nov 6, 2024 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Nov 6, 2024 at 7:34 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh,
> >
> > Here are my review comments for patch v49-0001.
> >
>
> I have a question on the display of this new parameter.
>
> postgres=# \dRp+
>                                       Publication pub_gen
>   Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
> root | Generated columns
> ----------+------------+---------+---------+---------+-----------+----------+-------------------
>  KapilaAm | f          | t       | t       | t       | t         | f        | t
> Tables:
>     "public.test_gen"
>
> The current theory for the display of the "Generated Columns" option
> could be that let's add new parameters at the end which sounds
> reasonable. The other way to look at it is how it would be easier for
> users to interpret. I think the value of the "Via root" option should
> be either after "All tables" or at the end as that is higher level
> table information than operations or column-level information. As
> currently, it is at the end, so "Generated Columns" should be added
> before.
>
> Thoughts?
>

FWIW, I've always felt the CREATE PUBLICATION parameters
publish
publish_via_root
publish_generated_columns

Should be documented (e.g. on CREATE PUBLICATION page) in alphabetical order:
publish
publish_generated_columns
publish_via_root

~

Following on from that. IMO it will make sense for the describe
(\dRp+) columns for those parameters to be in the same order as the
parameters in the documentation. So the end result would be the same
order as what you are wanting, even though the reason might be
different.

======
Kind Regards,
Peter Smith.
Fujitsu Australia.

  273. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-06T05:22:58Z

    On Wed, Nov 6, 2024 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Wed, Nov 6, 2024 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Nov 6, 2024 at 7:34 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Vignesh,
> > >
> > > Here are my review comments for patch v49-0001.
> > >
> >
> > I have a question on the display of this new parameter.
> >
> > postgres=# \dRp+
> >                                       Publication pub_gen
> >   Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
> > root | Generated columns
> > ----------+------------+---------+---------+---------+-----------+----------+-------------------
> >  KapilaAm | f          | t       | t       | t       | t         | f        | t
> > Tables:
> >     "public.test_gen"
> >
> > The current theory for the display of the "Generated Columns" option
> > could be that let's add new parameters at the end which sounds
> > reasonable. The other way to look at it is how it would be easier for
> > users to interpret. I think the value of the "Via root" option should
> > be either after "All tables" or at the end as that is higher level
> > table information than operations or column-level information. As
> > currently, it is at the end, so "Generated Columns" should be added
> > before.
> >
> > Thoughts?
> >
>
> FWIW, I've always felt the CREATE PUBLICATION parameters
> publish
> publish_via_root
> publish_generated_columns
>
> Should be documented (e.g. on CREATE PUBLICATION page) in alphabetical order:
> publish
> publish_generated_columns
> publish_via_root
>
> ~
>
> Following on from that. IMO it will make sense for the describe
> (\dRp+) columns for those parameters to be in the same order as the
> parameters in the documentation. So the end result would be the same
> order as what you are wanting, even though the reason might be
> different.
>

Sounds reasonable to me.

I have made some minor comments and function name changes in the
attached. Please include in the next version.

-- 
With Regards,
Amit Kapila.

  274. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2024-11-06T06:05:00Z

    Hi Vignesh,

I am observing some unexpected errors with the following scenario.

======
Tables:

Publisher table:
test_pub=# create table t1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
test_pub=# insert into t1 values (1);
INSERT 0 1

~

And Subscriber table:
test_sub=# create table t1(a int, b int);
CREATE TABLE

======
TEST PART 1.

I create 2 publications, having different parameter values.

test_pub=# create publication pub1 for table t1 with
(publish_generated_columns=true);
CREATE PUBLICATION
test_pub=# create publication pub2 for table t1 with
(publish_generated_columns=false);
CREATE PUBLICATION

~

And I try creating a subscription simultaneously subscribing to both
of these publications. This fails with an expected error.

test_sub=# create subscription sub1 connection 'dbname=test_pub'
publication pub1, pub2;
ERROR:  cannot use different column lists for table "public.t1" in
different publications

======
TEST PART 2.

Now on publisher set parameter for pub2 to be true;

test_pub=# alter publication pub2 set (publish_generated_columns);
ALTER PUBLICATION
test_pub=# \dRp+
                                        Publication pub1
  Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
root | Genera
ted columns
----------+------------+---------+---------+---------+-----------+----------+-------
------------
 postgres | f          | t       | t       | t       | t         | f        | t
Tables:
    "public.t1"

                                        Publication pub2
  Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
root | Genera
ted columns
----------+------------+---------+---------+---------+-----------+----------+-------
------------
 postgres | f          | t       | t       | t       | t         | f        | t
Tables:
    "public.t1"

~

Now the create subscriber works OK.

test_sub=# create subscription sub1 connection 'dbname=test_pub'
publication pub1,pub2;
NOTICE:  created replication slot "sub1" on publisher
CREATE SUBSCRIPTION

======
TEST PART 3.

Now on Publisher let's alter that parameter back to false again...

test_pub=# alter publication pub2 set (publish_generated_columns=false);
ALTER PUBLICATION

And insert some data.

test_pub=# insert into t1 values (2);
INSERT 0 1

~

Now the subscriber starts failing again...

ERROR:  cannot use different values of publish_generated_columns for
table "public.t1" in different publications
etc...

======
TEST PART 4.

Finally, on the Publisher alter that parameter back to true again!

test_pub=# alter publication pub2 set (publish_generated_columns);
ALTER PUBLICATION
test_pub=# \dRp+
                                        Publication pub1
  Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
root | Genera
ted columns
----------+------------+---------+---------+---------+-----------+----------+-------
------------
 postgres | f          | t       | t       | t       | t         | f        | t
Tables:
    "public.t1"

                                        Publication pub2
  Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
root | Genera
ted columns
----------+------------+---------+---------+---------+-----------+----------+-------
------------
 postgres | f          | t       | t       | t       | t         | f        | t
Tables:
    "public.t1"


~~

Unfortunately, even though the publication parameters are the same
again, the subscription seems to continue forever failing....

ERROR:  cannot use different values of publish_generated_columns for
table "public.t1" in different publications

~~

I didn't think a REFRESH PUBLICATION was necessary for this case, but
anyway that does not seem to make any difference.

test_sub=# alter subscription sub1 refresh publication;
ALTER SUBSCRIPTION

... still getting repeating error
2024-11-06 16:54:44.839 AEDT [5659] ERROR:  could not receive data
from WAL stream: ERROR:  cannot use different values of
publish_generated_columns for table "public.t1" in different
publications

======

To summarize -- Altering the publication parameter combination from
good to bad has an immediate effect on breaking the subscription, but
then altering it back again from bad to good seemed to do nothing  at
all (the subscription just remains broken).

======
Kind Regards,
Peter Smith.
Fujitsu Australia.

  275. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-06T06:25:04Z

    On Wed, Nov 6, 2024 at 11:35 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> I am observing some unexpected errors with the following scenario.
>

You are getting an expected ERROR. It is because of the design of
logical decoding which relies on historic snapshots.

> ======
> Tables:
>
> Publisher table:
> test_pub=# create table t1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
> CREATE TABLE
> test_pub=# insert into t1 values (1);
> INSERT 0 1
>
> ~
>
> And Subscriber table:
> test_sub=# create table t1(a int, b int);
> CREATE TABLE
>
> ======
> TEST PART 1.
>
> I create 2 publications, having different parameter values.
>
> test_pub=# create publication pub1 for table t1 with
> (publish_generated_columns=true);
> CREATE PUBLICATION
> test_pub=# create publication pub2 for table t1 with
> (publish_generated_columns=false);
> CREATE PUBLICATION
>
> ~
>
> And I try creating a subscription simultaneously subscribing to both
> of these publications. This fails with an expected error.
>
> test_sub=# create subscription sub1 connection 'dbname=test_pub'
> publication pub1, pub2;
> ERROR:  cannot use different column lists for table "public.t1" in
> different publications
>
> ======
> TEST PART 2.
>
> Now on publisher set parameter for pub2 to be true;
>
> test_pub=# alter publication pub2 set (publish_generated_columns);
> ALTER PUBLICATION
> test_pub=# \dRp+
>                                         Publication pub1
>   Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
> root | Genera
> ted columns
> ----------+------------+---------+---------+---------+-----------+----------+-------
> ------------
>  postgres | f          | t       | t       | t       | t         | f        | t
> Tables:
>     "public.t1"
>
>                                         Publication pub2
>   Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
> root | Genera
> ted columns
> ----------+------------+---------+---------+---------+-----------+----------+-------
> ------------
>  postgres | f          | t       | t       | t       | t         | f        | t
> Tables:
>     "public.t1"
>
> ~
>
> Now the create subscriber works OK.
>
> test_sub=# create subscription sub1 connection 'dbname=test_pub'
> publication pub1,pub2;
> NOTICE:  created replication slot "sub1" on publisher
> CREATE SUBSCRIPTION
>
> ======
> TEST PART 3.
>
> Now on Publisher let's alter that parameter back to false again...
>
> test_pub=# alter publication pub2 set (publish_generated_columns=false);
> ALTER PUBLICATION
>
> And insert some data.
>
> test_pub=# insert into t1 values (2);
> INSERT 0 1
>
> ~
>
> Now the subscriber starts failing again...
>
> ERROR:  cannot use different values of publish_generated_columns for
> table "public.t1" in different publications
> etc...
>
> ======
> TEST PART 4.
>
> Finally, on the Publisher alter that parameter back to true again!
>
> test_pub=# alter publication pub2 set (publish_generated_columns);
> ALTER PUBLICATION
...
>
>
> ~~
>
> Unfortunately, even though the publication parameters are the same
> again, the subscription seems to continue forever failing....
>
> ERROR:  cannot use different values of publish_generated_columns for
> table "public.t1" in different publications
>

The reason is that the failing 'insert' uses a historic snapshot,
which has a catalog state where 'publish_generated_columns' is still
false. So, you are seeing that error repeatedly. This behavior exists
from the very beginning of logical replication and another issue due
to the same reason was reported recently [1] which is actually a setup
issue. We should improve this situation some day but it is not the
responsibility of this patch.

[1] - https://www.postgresql.org/message-id/18683-a98f79c0673be358%40postgresql.org

-- 
With Regards,
Amit Kapila.

  276. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2024-11-06T10:48:29Z

    On Wed, 6 Nov 2024 at 10:53, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Nov 6, 2024 at 10:26 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Wed, Nov 6, 2024 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Nov 6, 2024 at 7:34 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Vignesh,
> > > >
> > > > Here are my review comments for patch v49-0001.
> > > >
> > >
> > > I have a question on the display of this new parameter.
> > >
> > > postgres=# \dRp+
> > >                                       Publication pub_gen
> > >   Owner   | All tables | Inserts | Updates | Deletes | Truncates | Via
> > > root | Generated columns
> > > ----------+------------+---------+---------+---------+-----------+----------+-------------------
> > >  KapilaAm | f          | t       | t       | t       | t         | f        | t
> > > Tables:
> > >     "public.test_gen"
> > >
> > > The current theory for the display of the "Generated Columns" option
> > > could be that let's add new parameters at the end which sounds
> > > reasonable. The other way to look at it is how it would be easier for
> > > users to interpret. I think the value of the "Via root" option should
> > > be either after "All tables" or at the end as that is higher level
> > > table information than operations or column-level information. As
> > > currently, it is at the end, so "Generated Columns" should be added
> > > before.
> > >
> > > Thoughts?
> > >
> >
> > FWIW, I've always felt the CREATE PUBLICATION parameters
> > publish
> > publish_via_root
> > publish_generated_columns
> >
> > Should be documented (e.g. on CREATE PUBLICATION page) in alphabetical order:
> > publish
> > publish_generated_columns
> > publish_via_root
> >
> > ~
> >
> > Following on from that. IMO it will make sense for the describe
> > (\dRp+) columns for those parameters to be in the same order as the
> > parameters in the documentation. So the end result would be the same
> > order as what you are wanting, even though the reason might be
> > different.
> >
>
> Sounds reasonable to me.

Updated the documentation and describe output accordingly.

> I have made some minor comments and function name changes in the
> attached. Please include in the next version.

Thanks, I have included the changes.

The attached v50 version patch has the changes for the same.

Regards,
Vignesh

  277. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-07T04:13:52Z

    On Wed, Nov 6, 2024 at 4:18 PM vignesh C <vignesh21@gmail.com> wrote:
>
> The attached v50 version patch has the changes for the same.
>

Pushed.

-- 
With Regards,
Amit Kapila.

  278. Re: Pgoutput not capturing the generated columns

    Peter Eisentraut <peter@eisentraut.org> — 2024-11-07T06:33:51Z

    On 07.11.24 05:13, Amit Kapila wrote:
> On Wed, Nov 6, 2024 at 4:18 PM vignesh C <vignesh21@gmail.com> wrote:
>>
>> The attached v50 version patch has the changes for the same.

Could you (everybody on this thread) please provide guidance how this 
feature is supposed to interact with virtual generated columns [0].  I 
don't think it's reasonably possible to replicate virtual generated 
columns.  I had previously suggested to make it more explicit that this 
feature only works for stored generated columns (e.g., name the option 
like that), but I don't see that this was considered.

[0]: 
https://www.postgresql.org/message-id/flat/a368248e-69e4-40be-9c07-6c3b5880b0a6@eisentraut.org

  279. RE: Pgoutput not capturing the generated columns

    Shinoda, Noriyoshi <noriyoshi.shinoda@hpe.com> — 2024-11-07T09:15:17Z

    Hi, Hackers.

Thanks for developing this great feature.
There seems to be a missing description of the "pubgencols" column added to the "pg_publication" catalog. The attached patch adds the description to the catalog.sgml file.
Please fix the patch if you have a better explanation.

Regards,
Noriyoshi Shinoda

-----Original Message-----
From: Amit Kapila <amit.kapila16@gmail.com> 
Sent: Thursday, November 7, 2024 1:14 PM
To: vignesh C <vignesh21@gmail.com>
Cc: Peter Smith <smithpb2250@gmail.com>; Shubham Khanna <khannashubham1197@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Rajendra Kumar Dangwal <dangwalrajendra888@gmail.com>; pgsql-hackers@lists.postgresql.org; euler@eulerto.com
Subject: Re: Pgoutput not capturing the generated columns

On Wed, Nov 6, 2024 at 4:18 PM vignesh C <vignesh21@gmail.com> wrote:
>
> The attached v50 version patch has the changes for the same.
>

Pushed.

-- 
With Regards,
Amit Kapila.

  280. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-07T09:50:04Z

    On Thu, Nov 7, 2024 at 12:03 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 07.11.24 05:13, Amit Kapila wrote:
> > On Wed, Nov 6, 2024 at 4:18 PM vignesh C <vignesh21@gmail.com> wrote:
> >>
> >> The attached v50 version patch has the changes for the same.
>
> Could you (everybody on this thread) please provide guidance how this
> feature is supposed to interact with virtual generated columns [0].  I
> don't think it's reasonably possible to replicate virtual generated
> columns.
>

I haven't studied the patch but can't we think of a way where we can
compute the value of the virtual generated column on the fly (say by
evaluating the required expression) before sending it to the client?
We do evaluate the expressions during the row filter, so can't we do
it for virtual-generated columns? I think we need some more work
similar to row filter/column list where we need to ensure that the
columns used in expressions for virtual generated columns must be part
of replica identity. I haven't thought about all the details so I may
be missing something.

>
  I had previously suggested to make it more explicit that this
> feature only works for stored generated columns (e.g., name the option
> like that), but I don't see that this was considered.
>

It was considered in earlier versions of the patch like [1] but later
we focussed more on getting key parts of the feature ready. Sorry for
missing that part but we can do it now. The idea is that we explicitly
mention in docs that the new option 'publish_generated_columns' will
replicate only STORED generated columns and also explicitly compare
'attgenerated' as ATTRIBUTE_GENERATED_STORED during decoding and
adjust comments. I suggest we do that for now. We could also consider
naming the option as publish_stored_generated_columns but that way the
name would be too long. The other idea could be to make the new option
as a string but that would be useful only if we decide to replicate
virtual generated columns.

[1] - https://www.postgresql.org/message-id/CAHv8RjJsGWETA9U53iRiV2%2BVGtnHamEJ5PKMHUcfat269kQaSQ%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  281. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2024-11-07T10:16:54Z

    On Thu, Nov 7, 2024 at 2:45 PM Shinoda, Noriyoshi (SXD Japan FSIP)
<noriyoshi.shinoda@hpe.com> wrote:
>
> Hi, Hackers.
>
> Thanks for developing this great feature.
> There seems to be a missing description of the "pubgencols" column added to the "pg_publication" catalog. The attached patch adds the description to the catalog.sgml file.
> Please fix the patch if you have a better explanation.
>

Can we slightly modify it as: "If true, this publication replicates
the generated columns in the tables associated with the publication."?
BTW, we might want to say: "If true, this publication replicates the
stored generated columns in the tables associated with the
publication." depending on the point Peter E. has raised, so, let's
wait for the conclusion.

-- 
With Regards,
Amit Kapila.

  282. Re: Pgoutput not capturing the generated columns

    Michael Paquier <michael@paquier.xyz> — 2024-12-10T06:50:43Z

    On Thu, Nov 07, 2024 at 03:46:54PM +0530, Amit Kapila wrote:
> Can we slightly modify it as: "If true, this publication replicates
> the generated columns in the tables associated with the publication."?
> BTW, we might want to say: "If true, this publication replicates the
> stored generated columns in the tables associated with the
> publication." depending on the point Peter E. has raised, so, let's
> wait for the conclusion.

Yeah, the state of things is unclear to me..  As of now this thread is
just waiting on author and idle in the CF app for no real purpose.  I
have marked it as returned with feedback, and I'd suggest to come back
to it and resubmit once the horizon is clear (one item being related
to virtual generated columns).
--
Michael

  283. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-12T23:55:24Z

    Hi,

Some patches of this thread have fallen though the cracks for more
than 2 months now, so I am re-posting them so that do not get
overlooked any longer.

For v49 [1] there were 2 patches:
v49-0001-Enable-support-for-publish_generated_columns-par
v49-0002-DOCS-Generated-Column-Replication

Then, at v50 [2] there were 2 patches:
v50-0001-Replicate-generated-columns-when-publish_generat
v50-0002-Tap-tests-for-generated-columns

Notice the DOCS patch is missing in v50.

Those v50* patches were then pushed to master. But, that was 2 months ago.

~~

So, the v49-0002-DOCS patch (which added a new section to Chapter 29)
remains still unpushed.

Meanwhile, it was reported [3] that pubgencols attribute of the
pg_publication catalog is missing from the docs.

~~

In this post, I have attached v51* patches where:

v51-0001 - This is a rebase of the missing DOCS patch from v49.

v51-0002 - This is the pg_publication catalog missing pubgencols
attribute patch.

~~~

Future -- there probably need to be further clarifications/emphasis to
describe how the generated column replication feature only works for
STORED generated columns (not VIRTUAL ones), but IMO it is better to
address that separately *after* dealing with these missing
documentation patches.

======
[1] v49 - https://www.postgresql.org/message-id/CALDaNm3XV5mAeZzZMkOPSPieANMaxOH8xAydLqf8X5PQn%2Ba5EA%40mail.gmail.com
[2] v50 - https://www.postgresql.org/message-id/CALDaNm1hR9-xFZXiK0it_ohn%2BPvfKTLvoOFhBi7p9oTSRCPJRg%40mail.gmail.com
[3] pg_publication -
https://www.postgresql.org/message-id/DM4PR84MB1734D79B25D5B427F5B6101AEE5C2%40DM4PR84MB1734.NAMPRD84.PROD.OUTLOOK.COM

Kind Regards,
Peter Smith.
Fujitsu Australia

  284. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-13T00:15:16Z

    On Mon, Jan 13, 2025 at 10:55 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> Some patches of this thread have fallen though the cracks for more
> than 2 months now, so I am re-posting them so that do not get
> overlooked any longer.
>
> For v49 [1] there were 2 patches:
> v49-0001-Enable-support-for-publish_generated_columns-par
> v49-0002-DOCS-Generated-Column-Replication
>
> Then, at v50 [2] there were 2 patches:
> v50-0001-Replicate-generated-columns-when-publish_generat
> v50-0002-Tap-tests-for-generated-columns
>
> Notice the DOCS patch is missing in v50.
>
> Those v50* patches were then pushed to master. But, that was 2 months ago.
>
> ~~
>
> So, the v49-0002-DOCS patch (which added a new section to Chapter 29)
> remains still unpushed.
>
> Meanwhile, it was reported [3] that pubgencols attribute of the
> pg_publication catalog is missing from the docs.
>
> ~~
>
> In this post, I have attached v51* patches where:
>
> v51-0001 - This is a rebase of the missing DOCS patch from v49.
>
> v51-0002 - This is the pg_publication catalog missing pubgencols
> attribute patch.
>

Added a new commitfest entry for this.

======
[1] https://commitfest.postgresql.org/52/5502/

Kind Regards,
Peter Smith.
Fujitsu Australia

  285. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-13T09:26:51Z

    On Mon, Jan 13, 2025 at 5:25 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Future -- there probably need to be further clarifications/emphasis to
> describe how the generated column replication feature only works for
> STORED generated columns (not VIRTUAL ones), but IMO it is better to
> address that separately *after* dealing with these missing
> documentation patches.
>

I thought it was better to deal with the ambiguity related to the
'virtual' part first. I have summarized the options we have regarding
this in an email [1]. I prefer to extend the current option to take
values as 's', and 'n'. This will keep the room open to extending it
with a new value 'v'. The primary reason to go this way is to avoid
adding new options in the future. It is better to keep the number of
subscription options under control. Do you have any preference?

[1] - https://www.postgresql.org/message-id/CAA4eK1JfEZUdtC5896vwEZFXBZnQ4aTDDXQxv3NOaosYu973Pw%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

  286. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-13T18:21:47Z

    On Mon, 13 Jan 2025 at 14:57, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 13, 2025 at 5:25 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Future -- there probably need to be further clarifications/emphasis to
> > describe how the generated column replication feature only works for
> > STORED generated columns (not VIRTUAL ones), but IMO it is better to
> > address that separately *after* dealing with these missing
> > documentation patches.
> >
>
> I thought it was better to deal with the ambiguity related to the
> 'virtual' part first. I have summarized the options we have regarding
> this in an email [1]. I prefer to extend the current option to take
> values as 's', and 'n'. This will keep the room open to extending it
> with a new value 'v'. The primary reason to go this way is to avoid
> adding new options in the future. It is better to keep the number of
> subscription options under control. Do you have any preference?

Yes, this seems a better approach. Here is the attached patch which
handles the same.

Regards,
Vignesh

  287. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-15T05:47:17Z

    Hi Vignesh.

Some review comments for patch 0001

======
GENERAL

1.
AFAIK there are still many places in the docs where there is no
distinction made between the stored/virtual generated cols. Maybe we
need another patch in this patch set to address all of these?

For example, CREATE PUBLICATION says "The column list can contain
generated columns as well.". Is that really true? What happens if we
try to publish a VIRTUAL column via a column list?

~~

2.
As suggested in more detail below, I think it would be better if you
can define a C code enum type for these potential values instead of
just using #define macros and char. I guess that will impact a lot of
the APIs.

======
doc/src/sgml/ref/create_publication.sgml

CREATE PUBLICATION, publish_generated_columns (enum)

3.
+         <para>
+          If set to <literal>stored</literal>, the generated columns present in
+          the tables associated with publication will be replicated.
          </para>

Maybe that say should say /the generated columns/the STORED generated columns/

~~~

4.
The next NOTE part is still referring wrongly to a boolean option:

If the subscriber is from a release prior to 18, then initial table
synchronization won't copy generated columns even if parameter
publish_generated_columns is true in the publisher.

SUGGESTION
If the subscriber is from a release prior to 18, then initial table
synchronization won't copy generated columns even if parameter
publish_generated_columns is not <literal>none</literal> in the
publisher.

======
src/backend/catalog/pg_publication.c

pub_form_cols_map:

5.
 /*
  * Returns a bitmap representing the columns of the specified table.
  *
- * Generated columns are included if include_gencols is true.
+ * Generated columns are included if gencols_type is 'stored'.
  */

The code is only checking 'none' but in future just because it is not
'none' does not mean it must be 'stored'. So maybe better to make this
comment future proof.

SUGGESTION
Generated columns are included if gencols_type is not PUBLISH_GENCOLS_NONE.

======
src/backend/commands/publicationcmds.c

defGetGeneratedColsOption:

6.
+ /*
+ * If no parameter value given, assume "stored" is meant.
+ */
+ if (!def->arg)
+ return PUBLISH_GENCOL_STORED;

Really, I think if no parameter is given it should mean "all". Now,
today there is only one valid value so 'all' and 'stored' are
equivalent, but I was wondering should you add another 'a'/'all' enum
so this meaning will already there for the future? Otherwise the no
parameter implementIion will have a slightly different meaning later.

~~

7.
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires a \"none\" or \"stored\"",
+    def->defname));
+ return PUBLISH_GENCOL_NONE; /* keep compiler quiet */

Put a blank line after the ereport.

======
src/backend/replication/logical/proto.c

8. General for all this file

 static void logicalrep_write_attrs(StringInfo out, Relation rel,
-    Bitmapset *columns, bool include_gencols);
+    Bitmapset *columns, char gencols_type);
 static void logicalrep_write_tuple(StringInfo out, Relation rel,
     TupleTableSlot *slot,
     bool binary, Bitmapset *columns,
-    bool include_gencols);
+    char gencols_type);

IMO all these parameter names changes all throughout this file should
be more like "include_gencols_type"

~~~

9.
 * 'gencols_type' value indicates whether generated columns should be
 * published when there is no column list. Typically, this will have the same
 * value as the 'publish_generated_columns' publication parameter.
 *
 * Note that generated columns can be published only when present in a
 * publication column list, or when gencols_type is 'stored'.

9a.
/gencols_type/include_gencols_type/

~

9b..
To future-proof this comment it might be more generic to say:

"...or when include_gencols_type is not PUBLISH_GENCOLS_NONE"

~~

10.
  /* All non-generated columns are always published. */
- return att->attgenerated ? include_gencols : true;
+ return att->attgenerated ? (gencols_type == PUBLISH_GENCOL_STORED) : true;

Would it be better (more future-proof) to express this like:

return att->attgenerated ? (gencols_type != PUBLISH_GENCOLS_NONE) : true;

======
src/backend/replication/pgoutput/pgoutput.c

11.
As mentioned elsewhere, IMO it might be more meaningful to name the
param/field/variable with an appropriate verb -- e.g.
"include_gencols_type" or "pubgencols_type", instead of just
"gencols_type".

~~~

typedef struct RelationSyncEntry:

12.
  /*
- * This is set if the 'publish_generated_columns' parameter is true, and
- * the relation contains generated columns.
+ * This is set if the 'publish_generated_columns' parameter is 'stored',
+ * and the relation contains generated columns.
  */
- bool include_gencols;
+ char gencols_type;

The comment doesn't make sense to me -- I think something got lost
when translating it from a boolean. e.g. I thought this field should
*always* be set, even if it is set to a value of PUBLISH_GENCOLS_NONE.

======
src/backend/utils/cache/relcache.c
======
src/bin/pg_dump/pg_dump.c

13.
  if (fout->remoteVersion >= 180000)
- appendPQExpBufferStr(query, "p.pubgencols ");
+ appendPQExpBufferStr(query, "p.pubgencols_type ");
  else
- appendPQExpBufferStr(query, "false AS pubgencols ");
+ appendPQExpBufferStr(query, "false AS pubgencols_type ");

Is that a bug? I thought it should be "'n' AS pubgencols_type"

~~~

14.
- i_pubgencols = PQfnumber(res, "pubgencols");
+ i_pubgencols = PQfnumber(res, "pubgencols_type");

Shouldn't we also rename the variable being assigned -- e.g. i_pubgencols_type

======
src/bin/pg_dump/pg_dump.h
======
src/bin/pg_dump/t/002_pg_dump.pl
======
src/bin/psql/describe.c
======
src/include/catalog/pg_publication.h

15.
 - /* true if generated columns data should be published */
- bool pubgencols;
+ /* 's'(stored) if generated columns data should be published */
+ char pubgencols_type;

I thought the comment should document all the possible values -- e.g.
what about 'n'?

~~~

16.
- bool pubgencols;
+ char pubgencols_type;
  PublicationActions pubactions;
 } Publication;

@@ -124,6 +124,16 @@ typedef struct PublicationRelInfo
  List    *columns;
 } PublicationRelInfo;

IMO all the APIs for functions that use this could be improved if you
are able to define a proper C enum for this field instead of just
using chars.

Anyway, you can still assigned the enum values to 'n' and 's' if it helps.

~~~

17,
+#ifdef EXPOSE_TO_CLIENT_CODE
+
+/* Generated columns present should not be replicated. */
+#define PUBLISH_GENCOL_NONE 'n'
+
+/* Generated columns present should be replicated. */
+#define PUBLISH_GENCOL_STORED 's'
+
+#endif /* EXPOSE_TO_CLIENT_CODE */

These values (which I thought ought to be enum values) should be
PUBLISH_GENCOLS_NONE, and PUBLISH_GENCOLS_STORED (e.g. _GENCOLS_
instead of _GENCOL_).

======
src/include/commands/publicationcmds.h

18.
 extern bool pub_contains_invalid_column(Oid pubid, Relation relation,
  List *ancestors, bool pubviaroot,
- bool pubgencols,
+ char gencols_type,
  bool *invalid_column_list,
  bool *invalid_gen_col);
Should new param name be 'pubgencols_type'?

======
src/include/replication/logicalproto.h

19.
- bool include_gencols);
+ char gencols_type);

Should all the changes like this be named "include_gencols_type" ?

======
src/test/regress/expected/publication.out

20.
\dRp
                                                        List of publications
        Name        |          Owner           | All tables | Inserts
| Updates | Deletes | Truncates | Generated columns | Via root
--------------------+--------------------------+------------+---------+---------+---------+-----------+-------------------+----------
 testpib_ins_trunct | regress_publication_user | f          | t
| f       | f       | f         | n                 | f
 testpub_default    | regress_publication_user | f          | f
| t       | f       | f         | n                 | f
(2 rows)

20a.
Why does that display 'n'? I expected it should say 'none'

~

20b.
Looks like a typo in the name /testpib_ins_trunct/testpub_ins_trunct/

~~~

21.
\dRp+ testpub_default
                                          Publication testpub_default
          Owner           | All tables | Inserts | Updates | Deletes |
Truncates | Generated columns | Via root
--------------------------+------------+---------+---------+---------+-----------+-------------------+----------
 regress_publication_user | f          | t       | t       | t       |
f         | none              | f
(1 row)

-- fail - must be owner of publication
SET ROLE regress_publication_user_dummy;
ALTER PUBLICATION testpub_default RENAME TO testpub_dummy;
ERROR:  must be owner of publication testpub_default
RESET ROLE;
ALTER PUBLICATION testpub_default RENAME TO testpub_foo;
\dRp testpub_foo
                                                     List of publications
    Name     |          Owner           | All tables | Inserts |
Updates | Deletes | Truncates | Generated columns | Via root
-------------+--------------------------+------------+---------+---------+---------+-----------+-------------------+----------
 testpub_foo | regress_publication_user | f          | t       | t
  | t       | f         | n                 | f
(1 row)

Notice there is a 'n' for \dRp but a 'none' for \dRp+. It looks like
only the \dRp is broken.

======
src/test/regress/sql/publication.sql

22. General

I found lots of places referring to 'publication_generate_columns'.
But there is no such thing. It is supposed to say
publish_generated_columns.

~~~

23. Missing test?

I think there now should be a test for a publication with
publish_gfenerated_column option, but having no value to verify that
it is the same as stored.

======
src/test/subscription/t/011_generated.pl

24.
 # =============================================================================
 # Exercise logical replication of a generated column to a subscriber side
 # regular column. This is done both when the publication parameter
-# 'publish_generated_columns' is set to false (to confirm existing default
+# 'publish_generated_columns' is set to 'none' (to confirm existing default
 # behavior), and is set to true (to confirm replication occurs).

Not fully updated. This is still saying "and is set to true"

~~~

25.
# Verify that the generated column data is not replicated during incremental
# replication when publish_generated_columns is set to false.

The above comment (still present in the file) still refers to the
boolean values of the option.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  288. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-15T06:29:41Z

    On Mon, Jan 13, 2025 at 8:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 13, 2025 at 5:25 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Future -- there probably need to be further clarifications/emphasis to
> > describe how the generated column replication feature only works for
> > STORED generated columns (not VIRTUAL ones), but IMO it is better to
> > address that separately *after* dealing with these missing
> > documentation patches.
> >
>
> I thought it was better to deal with the ambiguity related to the
> 'virtual' part first. I have summarized the options we have regarding
> this in an email [1]. I prefer to extend the current option to take
> values as 's', and 'n'. This will keep the room open to extending it
> with a new value 'v'. The primary reason to go this way is to avoid
> adding new options in the future. It is better to keep the number of
> subscription options under control. Do you have any preference?
>

Hi,

During my review of Vignesh's patch for the enum-version of
publish_generated_columns, I was thinking of yet another way to
specify which columns to replicate.

My idea below is analogous to the existing 'publish' option; Instead
of adding an option specific to generated column types why don't we
instead add a (string) option for controlling the publication of *all*
column types?

Synopsis:

publish_column_types = <col_types>[,...]
where <col_types> are 'normal', 'generated_stored', 'generated_virtual'.

The default value is 'normal', which just means everything that's not generated

~

This option would be overriding if a publication column list is
specified, same as the current implementation does.

~

And, just like the 'publish' option the effect is cumulative:

e.g.1. WITH (publish_column_types = 'normal') == default behavior.
publishes all normal columns same as PG17
e.g.2. WITH (publish_column_types = 'normal, generated_stored') ==
publishes all normal cols AND stored gencols
e.g.3. WITH (publish_column_types = 'generated_stored') == publishes
only the stored gencols and nothing else

Notice that some combinations (like example 3 above with a FOR ALL
TABLES) are not even possible using master, or Vignesh's patch. Maybe
having this extra flexibility is useful for someone?

~

Also, having a generic name 'publish_column_types' leaves this open to
be extended with more possible values in the future without
proliferating more publication options.

Thoughts?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  289. Re: Pgoutput not capturing the generated columns

    Shlok Kyal <shlok.kyal.oss@gmail.com> — 2025-01-15T09:11:18Z

    On Mon, 13 Jan 2025 at 23:52, vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 13 Jan 2025 at 14:57, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Jan 13, 2025 at 5:25 AM Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Future -- there probably need to be further clarifications/emphasis to
> > > describe how the generated column replication feature only works for
> > > STORED generated columns (not VIRTUAL ones), but IMO it is better to
> > > address that separately *after* dealing with these missing
> > > documentation patches.
> > >
> >
> > I thought it was better to deal with the ambiguity related to the
> > 'virtual' part first. I have summarized the options we have regarding
> > this in an email [1]. I prefer to extend the current option to take
> > values as 's', and 'n'. This will keep the room open to extending it
> > with a new value 'v'. The primary reason to go this way is to avoid
> > adding new options in the future. It is better to keep the number of
> > subscription options under control. Do you have any preference?
>
> Yes, this seems a better approach. Here is the attached patch which
> handles the same.
>
Hi Vignesh,

I have reviewed the patch and have following comments:

In file: create_publication.sgml

1.
+         <para>
+          If set to <literal>stored</literal>, the generated columns present in
+          the tables associated with publication will be replicated.
          </para>

Instead of 'generated columns' we should use 'stored generated columns'

======
In file: pg_publication.c

2. I feel this condition can be more specific. Since a new macro will
be introduced for upcoming Virtual Generated columns.

- if (att->attisdropped || (att->attgenerated && !include_gencols))
+ if (att->attisdropped ||
+ (att->attgenerated && (gencols_type == PUBLISH_GENCOL_NONE)))
  continue;

Something like:
    if (att->attisdropped)
        continue;
    if (att->attgenerated == ATTRIBUTE_GENERATED_STORED &&
gencols_type != PUBLISH_GENCOL_STORED)
        continue;

Thoughs?

3. Similarly this can be updated here as well:

- if (att->attisdropped || (att->attgenerated && !pub->pubgencols))
+ if (att->attisdropped ||
+ (att->attgenerated && pub->pubgencols_type == PUBLISH_GENCOL_NONE))
  continue;

=======
In file proto.c

4. I feel this condition should also be more specific:

  /* All non-generated columns are always published. */
- return att->attgenerated ? include_gencols : true;
+ return att->attgenerated ? (gencols_type == PUBLISH_GENCOL_STORED) : true;

We should return 'true' for 'gencols_type == PUBLISH_GENCOL_STORED'
only if 'att->attgenerated = ATTRIBUTE_GENERATED_STORED'

=======
In file publicationcmds.c

5.
  /*
  * As we don't allow a column list with REPLICA IDENTITY FULL, the
- * publish_generated_columns option must be set to true if the table
- * has any stored generated columns.
+ * publish_generated_columns option must be set to 's'(stored) if the
+ * table has any stored generated columns.
  */
- if (!pubgencols &&
+ if (gencols_type == PUBLISH_GENCOL_NONE &&
  relation->rd_att->constr &&

To be consistent with the comment, I think we should check if
'gencols_type != PUBLISH_GENCOL_STORED' instead of 'gencols_type ==
PUBLISH_GENCOL_NONE'.
Thoughts?

Thanks and Regards,
Shlok Kyal

  290. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-16T04:16:34Z

    On Wed, 15 Jan 2025 at 12:00, Peter Smith <smithpb2250@gmail.com> wrote:
>
> During my review of Vignesh's patch for the enum-version of
> publish_generated_columns, I was thinking of yet another way to
> specify which columns to replicate.
>
> My idea below is analogous to the existing 'publish' option; Instead
> of adding an option specific to generated column types why don't we
> instead add a (string) option for controlling the publication of *all*
> column types?
>
> Synopsis:
>
> publish_column_types = <col_types>[,...]
> where <col_types> are 'normal', 'generated_stored', 'generated_virtual'.
>
> The default value is 'normal', which just means everything that's not generated
>
> ~
>
> This option would be overriding if a publication column list is
> specified, same as the current implementation does.
>
> ~
>
> And, just like the 'publish' option the effect is cumulative:
>
> e.g.1. WITH (publish_column_types = 'normal') == default behavior.
> publishes all normal columns same as PG17
> e.g.2. WITH (publish_column_types = 'normal, generated_stored') ==
> publishes all normal cols AND stored gencols
> e.g.3. WITH (publish_column_types = 'generated_stored') == publishes
> only the stored gencols and nothing else
>
> Notice that some combinations (like example 3 above with a FOR ALL
> TABLES) are not even possible using master, or Vignesh's patch. Maybe
> having this extra flexibility is useful for someone?
>
> ~
>
> Also, having a generic name 'publish_column_types' leaves this open to
> be extended with more possible values in the future without
> proliferating more publication options.
>
> Thoughts?

I believe the existing enum is adequate for handling generated
columns. Ideally, users would prefer to either have only non-generated
columns or all columns in the table. However, since the implementation
of virtual generated columns will be phased—starting with the virtual
columns and followed by their replication in future updates—an enum is
necessary. This will allow users to choose between
publish_generated_columns set to none or publish_generated_columns set
to stored for now, and later switch to publish_generated_columns set
to all once the implementation is complete. Additionally, users who
want to select only generated columns can use column list publication.
Given these considerations, I believe the current approach is
appropriate.

Regards,
Vignesh

  291. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-16T08:47:08Z

    On Wed, 15 Jan 2025 at 11:17, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh.
>
> Some review comments for patch 0001
>
> ======
> GENERAL
>
> 1.
> AFAIK there are still many places in the docs where there is no
> distinction made between the stored/virtual generated cols. Maybe we
> need another patch in this patch set to address all of these?

> For example, CREATE PUBLICATION says "The column list can contain
> generated columns as well.". Is that really true? What happens if we
> try to publish a VIRTUAL column via a column list?.

Currently I have mentioned that only stored generated columns will be
published. Let's see if there are only a few changes we can keep with
this patch else we could merge with the other document patch.

> ~~
>
> 2.
> As suggested in more detail below, I think it would be better if you
> can define a C code enum type for these potential values instead of
> just using #define macros and char. I guess that will impact a lot of
> the APIs.

If we change it to enum, we will not be able to access
PUBLISH_GENCOLS_NONE and PUBLISH_GENCOLS_STORED from describe.c files.
Maybe that is the reason the macros were used in the case of
pg_subscription.h also.

> ======
> doc/src/sgml/ref/create_publication.sgml
>
> CREATE PUBLICATION, publish_generated_columns (enum)
>
> 3.
> +         <para>
> +          If set to <literal>stored</literal>, the generated columns present in
> +          the tables associated with publication will be replicated.
>           </para>
>
> Maybe that say should say /the generated columns/the STORED generated columns/

Modified

> ~~~
>
> 4.
> The next NOTE part is still referring wrongly to a boolean option:
>
> If the subscriber is from a release prior to 18, then initial table
> synchronization won't copy generated columns even if parameter
> publish_generated_columns is true in the publisher.
>
> SUGGESTION
> If the subscriber is from a release prior to 18, then initial table
> synchronization won't copy generated columns even if parameter
> publish_generated_columns is not <literal>none</literal> in the
> publisher.

Modified

> ======
> src/backend/catalog/pg_publication.c
>
> pub_form_cols_map:
>
> 5.
>  /*
>   * Returns a bitmap representing the columns of the specified table.
>   *
> - * Generated columns are included if include_gencols is true.
> + * Generated columns are included if gencols_type is 'stored'.
>   */
>
> The code is only checking 'none' but in future just because it is not
> 'none' does not mean it must be 'stored'. So maybe better to make this
> comment future proof.
>
> SUGGESTION
> Generated columns are included if gencols_type is not PUBLISH_GENCOLS_NONE.

The current approach is more appropriate here, as replication of
generated virtual columns will not be supported in the initial
version.

> ======
> src/backend/commands/publicationcmds.c
>
> defGetGeneratedColsOption:
>
> 6.
> + /*
> + * If no parameter value given, assume "stored" is meant.
> + */
> + if (!def->arg)
> + return PUBLISH_GENCOL_STORED;
>
> Really, I think if no parameter is given it should mean "all". Now,
> today there is only one valid value so 'all' and 'stored' are
> equivalent, but I was wondering should you add another 'a'/'all' enum
> so this meaning will already there for the future? Otherwise the no
> parameter implementIion will have a slightly different meaning later.

I believe the "all" option should be added when virtual columns are
introduced. Since the replication of virtual generated columns will
likely be phased—first with the addition of the virtual generated
column feature itself, followed later by support for logical
replication of these columns—it’s probable that logical replication of
virtual columns won’t be available in the initial release. Therefore,
the "all" option should be introduced when logical replication of
virtual generated columns is added.

> ~~
>
> 7.
> + ereport(ERROR,
> + errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("%s requires a \"none\" or \"stored\"",
> +    def->defname));
> + return PUBLISH_GENCOL_NONE; /* keep compiler quiet */
>
> Put a blank line after the ereport.

Modified

> ======
> src/backend/replication/logical/proto.c
>
> 8. General for all this file
>
>  static void logicalrep_write_attrs(StringInfo out, Relation rel,
> -    Bitmapset *columns, bool include_gencols);
> +    Bitmapset *columns, char gencols_type);
>  static void logicalrep_write_tuple(StringInfo out, Relation rel,
>      TupleTableSlot *slot,
>      bool binary, Bitmapset *columns,
> -    bool include_gencols);
> +    char gencols_type);
>
> IMO all these parameter names changes all throughout this file should
> be more like "include_gencols_type"

Modified

> ~~~
>
> 9.
>  * 'gencols_type' value indicates whether generated columns should be
>  * published when there is no column list. Typically, this will have the same
>  * value as the 'publish_generated_columns' publication parameter.
>  *
>  * Note that generated columns can be published only when present in a
>  * publication column list, or when gencols_type is 'stored'.
>
> 9a.
> /gencols_type/include_gencols_type/

Modified

> ~
>
> 9b..
> To future-proof this comment it might be more generic to say:
>
> "...or when include_gencols_type is not PUBLISH_GENCOLS_NONE"

The current approach is more appropriate here, when virtual generated
columns are added we don't want it to be published without special
handling for it.

> ~~
>
> 10.
>   /* All non-generated columns are always published. */
> - return att->attgenerated ? include_gencols : true;
> + return att->attgenerated ? (gencols_type == PUBLISH_GENCOL_STORED) : true;
>
> Would it be better (more future-proof) to express this like:
>
> return att->attgenerated ? (gencols_type != PUBLISH_GENCOLS_NONE) : true;

Modified

> ======
> src/backend/replication/pgoutput/pgoutput.c
>
> 11.
> As mentioned elsewhere, IMO it might be more meaningful to name the
> param/field/variable with an appropriate verb -- e.g.
> "include_gencols_type" or "pubgencols_type", instead of just
> "gencols_type".

Modified

> ~~~
>
> typedef struct RelationSyncEntry:
>
> 12.
>   /*
> - * This is set if the 'publish_generated_columns' parameter is true, and
> - * the relation contains generated columns.
> + * This is set if the 'publish_generated_columns' parameter is 'stored',
> + * and the relation contains generated columns.
>   */
> - bool include_gencols;
> + char gencols_type;
>
> The comment doesn't make sense to me -- I think something got lost
> when translating it from a boolean. e.g. I thought this field should
> *always* be set, even if it is set to a value of PUBLISH_GENCOLS_NONE.

Modified

> ======
> src/backend/utils/cache/relcache.c
> ======
> src/bin/pg_dump/pg_dump.c
>
> 13.
>   if (fout->remoteVersion >= 180000)
> - appendPQExpBufferStr(query, "p.pubgencols ");
> + appendPQExpBufferStr(query, "p.pubgencols_type ");
>   else
> - appendPQExpBufferStr(query, "false AS pubgencols ");
> + appendPQExpBufferStr(query, "false AS pubgencols_type ");
>
> Is that a bug? I thought it should be "'n' AS pubgencols_type"

Yes it should be n, modified

> ~~~
>
> 14.
> - i_pubgencols = PQfnumber(res, "pubgencols");
> + i_pubgencols = PQfnumber(res, "pubgencols_type");
>
> Shouldn't we also rename the variable being assigned -- e.g. i_pubgencols_type

Modified

> ======
> src/bin/pg_dump/pg_dump.h
> ======
> src/bin/pg_dump/t/002_pg_dump.pl
> ======
> src/bin/psql/describe.c
> ======
> src/include/catalog/pg_publication.h
>
> 15.
>  - /* true if generated columns data should be published */
> - bool pubgencols;
> + /* 's'(stored) if generated columns data should be published */
> + char pubgencols_type;
>
> I thought the comment should document all the possible values -- e.g.
> what about 'n'?

Modified

> ~~~
>
> 16.
> - bool pubgencols;
> + char pubgencols_type;
>   PublicationActions pubactions;
>  } Publication;
>
> @@ -124,6 +124,16 @@ typedef struct PublicationRelInfo
>   List    *columns;
>  } PublicationRelInfo;
>
> IMO all the APIs for functions that use this could be improved if you
> are able to define a proper C enum for this field instead of just
> using chars.
>
> Anyway, you can still assigned the enum values to 'n' and 's' if it helps.

If we change it to enum, we will not be able to access
PUBLISH_GENCOLS_NONE and PUBLISH_GENCOLS_STORED from describe.c files.
Maybe that is the reason the macros were used in the case of
pg_subscription.h also.

> ~~~
>
> 17,
> +#ifdef EXPOSE_TO_CLIENT_CODE
> +
> +/* Generated columns present should not be replicated. */
> +#define PUBLISH_GENCOL_NONE 'n'
> +
> +/* Generated columns present should be replicated. */
> +#define PUBLISH_GENCOL_STORED 's'
> +
> +#endif /* EXPOSE_TO_CLIENT_CODE */
>
> These values (which I thought ought to be enum values) should be
> PUBLISH_GENCOLS_NONE, and PUBLISH_GENCOLS_STORED (e.g. _GENCOLS_
> instead of _GENCOL_).

Modified

> ======
> src/include/commands/publicationcmds.h
>
> 18.
>  extern bool pub_contains_invalid_column(Oid pubid, Relation relation,
>   List *ancestors, bool pubviaroot,
> - bool pubgencols,
> + char gencols_type,
>   bool *invalid_column_list,
>   bool *invalid_gen_col);
> Should new param name be 'pubgencols_type'?

Modified

> ======
> src/include/replication/logicalproto.h
>
> 19.
> - bool include_gencols);
> + char gencols_type);
>
> Should all the changes like this be named "include_gencols_type" ?

Modified

> ======
> src/test/regress/expected/publication.out
>
> 20.
> \dRp
>                                                         List of publications
>         Name        |          Owner           | All tables | Inserts
> | Updates | Deletes | Truncates | Generated columns | Via root
> --------------------+--------------------------+------------+---------+---------+---------+-----------+-------------------+----------
>  testpib_ins_trunct | regress_publication_user | f          | t
> | f       | f       | f         | n                 | f
>  testpub_default    | regress_publication_user | f          | f
> | t       | f       | f         | n                 | f
> (2 rows)
>
> 20a.
> Why does that display 'n'? I expected it should say 'none'

Modified

> ~
>
> 20b.
> Looks like a typo in the name /testpib_ins_trunct/testpub_ins_trunct/

Modified

> ~~~
>
> 21.
> \dRp+ testpub_default
>                                           Publication testpub_default
>           Owner           | All tables | Inserts | Updates | Deletes |
> Truncates | Generated columns | Via root
> --------------------------+------------+---------+---------+---------+-----------+-------------------+----------
>  regress_publication_user | f          | t       | t       | t       |
> f         | none              | f
> (1 row)
>
> -- fail - must be owner of publication
> SET ROLE regress_publication_user_dummy;
> ALTER PUBLICATION testpub_default RENAME TO testpub_dummy;
> ERROR:  must be owner of publication testpub_default
> RESET ROLE;
> ALTER PUBLICATION testpub_default RENAME TO testpub_foo;
> \dRp testpub_foo
>                                                      List of publications
>     Name     |          Owner           | All tables | Inserts |
> Updates | Deletes | Truncates | Generated columns | Via root
> -------------+--------------------------+------------+---------+---------+---------+-----------+-------------------+----------
>  testpub_foo | regress_publication_user | f          | t       | t
>   | t       | f         | n                 | f
> (1 row)
>
> Notice there is a 'n' for \dRp but a 'none' for \dRp+. It looks like
> only the \dRp is broken.

Modified

> ======
> src/test/regress/sql/publication.sql
>
> 22. General
>
> I found lots of places referring to 'publication_generate_columns'.
> But there is no such thing. It is supposed to say
> publish_generated_columns.

Modified

> ~~
>
> 23. Missing test?
>
> I think there now should be a test for a publication with
> publish_gfenerated_column option, but having no value to verify that
> it is the same as stored.

Added a test for this

> ======
> src/test/subscription/t/011_generated.pl
>
> 24.
>  # =============================================================================
>  # Exercise logical replication of a generated column to a subscriber side
>  # regular column. This is done both when the publication parameter
> -# 'publish_generated_columns' is set to false (to confirm existing default
> +# 'publish_generated_columns' is set to 'none' (to confirm existing default
>  # behavior), and is set to true (to confirm replication occurs).
>
> Not fully updated. This is still saying "and is set to true"

Modified

> ~~~
>
> 25.
> # Verify that the generated column data is not replicated during incremental
> # replication when publish_generated_columns is set to false.
>
> The above comment (still present in the file) still refers to the
> boolean values of the option.

Modified

Thanks for the comments, the attached v2 version patch has the changes
for the same.

v52-0001 - There was an issue when we describe publication for a PG17
server, this 0001 patch fixes the same.
v52-0002 - One typo related to a publication name which Peter reported.
v52-0003 - Change publish_generated_columns option to use enum instead
of boolean.
v52-0004 and v52-0005 are the previous patches from [1].

[1] - https://www.postgresql.org/message-id/CAHut%2BPsPZVE8r1VeznuF3hSiTxcbprzaiBedSmMv39FYoHn4YA%40mail.gmail.com

Regards,
Vignesh

  292. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-16T08:49:31Z

    On Wed, 15 Jan 2025 at 14:41, Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> I have reviewed the patch and have following comments:
>
> In file: create_publication.sgml
>
> 1.
> +         <para>
> +          If set to <literal>stored</literal>, the generated columns present in
> +          the tables associated with publication will be replicated.
>           </para>
>
> Instead of 'generated columns' we should use 'stored generated columns'

Modified

> ======
> In file: pg_publication.c
>
> 2. I feel this condition can be more specific. Since a new macro will
> be introduced for upcoming Virtual Generated columns.
>
> - if (att->attisdropped || (att->attgenerated && !include_gencols))
> + if (att->attisdropped ||
> + (att->attgenerated && (gencols_type == PUBLISH_GENCOL_NONE)))
>   continue;
>
> Something like:
>     if (att->attisdropped)
>         continue;
>     if (att->attgenerated == ATTRIBUTE_GENERATED_STORED &&
> gencols_type != PUBLISH_GENCOL_STORED)
>         continue;
>
> Thoughs?

Modified

> 3. Similarly this can be updated here as well:
>
> - if (att->attisdropped || (att->attgenerated && !pub->pubgencols))
> + if (att->attisdropped ||
> + (att->attgenerated && pub->pubgencols_type == PUBLISH_GENCOL_NONE))
>   continue;

Modified

> =======
> In file proto.c
>
> 4. I feel this condition should also be more specific:
>
>   /* All non-generated columns are always published. */
> - return att->attgenerated ? include_gencols : true;
> + return att->attgenerated ? (gencols_type == PUBLISH_GENCOL_STORED) : true;
>
> We should return 'true' for 'gencols_type == PUBLISH_GENCOL_STORED'
> only if 'att->attgenerated = ATTRIBUTE_GENERATED_STORED'

Modified

> =======
> In file publicationcmds.c
>
> 5.
>   /*
>   * As we don't allow a column list with REPLICA IDENTITY FULL, the
> - * publish_generated_columns option must be set to true if the table
> - * has any stored generated columns.
> + * publish_generated_columns option must be set to 's'(stored) if the
> + * table has any stored generated columns.
>   */
> - if (!pubgencols &&
> + if (gencols_type == PUBLISH_GENCOL_NONE &&
>   relation->rd_att->constr &&
>
> To be consistent with the comment, I think we should check if
> 'gencols_type != PUBLISH_GENCOL_STORED' instead of 'gencols_type ==
> PUBLISH_GENCOL_NONE'.
> Thoughts?

Modified

The v52 version patch attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm3OcXdY0EzDEKAfaK9gq2B67Mfsgxu93%2B_249ohyts%3D0g%40mail.gmail.com

Regards,
Vignesh

  293. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-17T01:53:06Z

    Hi Vignesh.

Some review comments for v52-0001.

======
Commit message

1.
I guess the root cause is that in PG18 we decided to put the
"Generated columns" column to the left of the "Via Root" column
instead of to the right, and in doing so introduced a mistake ordering
the code.

The commit message can say that a bit more plainly -- e.g. you can
name those columns 'pubgencols' and 'pubviaroot' explicitly instead of
just referring to "the newly added column".

======
src/bin/psql/describe.c

describePublications:

2.
  if (has_pubgencols)
- printTableAddCell(&cont, PQgetvalue(res, i, 8), false, false);
- if (has_pubviaroot)
  printTableAddCell(&cont, PQgetvalue(res, i, 9), false, false);
+ if (has_pubviaroot)
+ printTableAddCell(&cont, PQgetvalue(res, i, 8), false, false);

It seems a bit tricky for the code to just have the PQgetvalue
hardwired indexes (8,9) reversed like that. I think at least this
reversal needs a comment to explain why it is done that way, so nobody
in future is tempted to change it and accidentally re-break this.

Maybe also, consider adding variables for those PQgetvalue indexes.
Maybe, something like this can work?

pubviaroot_idx = ncols;
pubgencols_idx = ncols + 1;

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  294. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-17T02:14:40Z

    On Thu, Jan 16, 2025 at 7:47 PM vignesh C <vignesh21@gmail.com> wrote:
>
...
>
> v52-0002 - One typo related to a publication name which Peter reported.

Hi Vignesh,

Patch v52-0002 LGTM.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  295. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-17T05:52:33Z

    Hi Vignesh.

Some review comments for patch v52-0003

======

1. GENERAL - change to use enum.

On Thu, Jan 16, 2025 at 7:47 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 15 Jan 2025 at 11:17, Peter Smith <smithpb2250@gmail.com> wrote:
> > 2.
> > As suggested in more detail below, I think it would be better if you
> > can define a C code enum type for these potential values instead of
> > just using #define macros and char. I guess that will impact a lot of
> > the APIs.
>
> If we change it to enum, we will not be able to access
> PUBLISH_GENCOLS_NONE and PUBLISH_GENCOLS_STORED from describe.c files.
> Maybe that is the reason the macros were used in the case of
> pg_subscription.h also.

Hm. I am not sure. Can't you just define the enum inside the #ifdef
EXPOSE_TO_CLIENT_CODE? I saw some examples of this already (see
src/include/catalog/pg_cast.h)

e.g. I tried following, which compiles for me:

#ifdef EXPOSE_TO_CLIENT_CODE

typedef enum PublishGencolsType
{
  /* Generated columns present should not be replicated. */
  PUBLISH_GENCOLS_NONE = 'n',

  /* Generated columns present should be replicated. */
  PUBLISH_GENCOLS_STORED = 's',

} PublishGencolsType;

#endif  /* EXPOSE_TO_CLIENT_CODE */

typedef struct Publication
{
  Oid    oid;
  char   *name;
  bool   alltables;
  bool   pubviaroot;
  PublishGencolsType pubgencols_type;
  PublicationActions pubactions;
} Publication;

======
doc/src/sgml/ref/create_publication.sgml

2.
          <para>
           Specifies whether the generated columns present in the tables
           associated with the publication should be replicated.
-          The default is <literal>false</literal>.
+          The default is <literal>none</literal> meaning the generated
+          columns present in the tables associated with publication will not be
+          replicated.
+         </para>

IMO it would be better to add another sentence before "The default
is..." to just spell out what the possible values are up-front. The
"The default is..." can also be preceded by a blank line.

SUGGESTION
Specifies whether the generated columns present in the tables
associated with the publication should be replicated. Possible values
are <literal>none</literal> and <literal>stored</literal>.

The default is <literal>none</literal> meaning...

======
src/backend/catalog/pg_publication.c

pub_form_cols_map:

3.
+ if (att->attgenerated)
+ {
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ else if (include_gencols_type != PUBLISH_GENCOLS_STORED)
+ continue;
+ }
+

I find it easier to read without the 'else' keyword. Also, I think
these can be commented on.

SUGGESTION
/* We only support replication of STORED generated cols. */
if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
  continue;

/* User hasn't requested to replicate STORED generated cols. */
if (include_gencols_type != PUBLISH_GENCOLS_STORED)
  continue;

~~~

pg_get_publication_tables:

4.
- if (att->attisdropped || (att->attgenerated && !pub->pubgencols))
+ if (att->attisdropped)
  continue;

+ if (att->attgenerated)
+ {
+ if (att->attgenerated != ATTRIBUTE_GENERATED_STORED)
+ continue;
+ else if (pub->pubgencols_type != PUBLISH_GENCOLS_STORED)
+ continue;
+ }
+


Ditto previous review comment #3 above.

======
src/backend/commands/publicationcmds.c

pub_contains_invalid_column:

5.
The function comment is still referring to "enabling
publish_generated_columns option" which kind of sounds like a boolean.

~~~

6.
  /*
- * The publish_generated_columns option must be set to true if the
- * REPLICA IDENTITY contains any stored generated column.
+ * The publish_generated_columns option must be set to 's'(stored)
+ * if the REPLICA IDENTITY contains any stored generated column.
  */
- if (!pubgencols && att->attgenerated)
+ if (pubgencols_type == PUBLISH_GENCOLS_NONE && att->attgenerated)

The comment and the code seem a bit out of sync; the comment says must
be 's', so to match that the code ought to be checking !=
PUBLISH_GENCOLS_STORED.

~~~

defGetGeneratedColsOption:

7.
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires a \"none\" or \"stored\"",
+    def->defname));
+

Should that have the word "value"?
e.g. "%s requires a \"none\" or \"stored\" value

======
src/backend/replication/logical/proto.c

logicalrep_should_publish_column:

8.
+ return (att->attgenerated == ATTRIBUTE_GENERATED_STORED) ?
(include_gencols_type == PUBLISH_GENCOLS_STORED) : false;

The ternary is fine, but it might be neater to split it out so you can
comment on it

SUGGESTION
/* Stored generated columns are only published when the user sets
publish_generated_columns = stored. */
if (att->attgenerated == ATTRIBUTE_GENERATED_STORED)
  return include_gencols_type == PUBLISH_GENCOLS_STORED;

return false;

======
src/backend/replication/pgoutput/pgoutput.c

typedef struct RelationSyncEntry:

9.
  /*
- * This is set if the 'publish_generated_columns' parameter is true, and
- * the relation contains generated columns.
+ * This will be 's'(stored) if the relation contains generated columns and
+ * the 'publish_generated_columns' parameter is set to 's'(stored).
+ * Otherwise, it will be 'n'(none), indicating that no generated columns
+ * should be published.
  */
- bool include_gencols;
+ char include_gencols_type;

Is the comment strictly correct? What about overriding column lists
where the option is set to 'none'?

======
src/include/catalog/pg_publication.h

10.
- /* true if generated columns data should be published */
- bool pubgencols;
+ /*
+ * 's'stored) if generated column data should be published, 'n'(none) if
+ * it should not be published
+ */
+ char pubgencols_type;

Typo. Missing '('

Also better to put each value on a separate line, and explicitly
mention 'stored', etc.

SUGGESTION
 /*
  * 'n'(none) if generated column data should not be published.
  * 's'(stored) if stored generated column data should be published.
  */

======
src/test/subscription/t/011_generated.pl

11.
 # =============================================================================
 # Exercise logical replication of a generated column to a subscriber side
 # regular column. This is done both when the publication parameter
-# 'publish_generated_columns' is set to false (to confirm existing default
-# behavior), and is set to true (to confirm replication occurs).
+# 'publish_generated_columns' is set to 'none' (to confirm existing default
+# behavior), and is set to stored (to confirm replication occurs).

Why 'none' in quotes but stored not in quotes?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  296. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-19T01:09:11Z

    Hi Vignesh,

I was having some second thoughts about this patch and my previous suggestion.

Currently the code is current written something like:

printfPQExpBuffer(&buf,
 "SELECT oid, pubname,\n"
 "  pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
 "  puballtables, pubinsert, pubupdate, pubdelete");

if (has_pubtruncate)
 appendPQExpBufferStr(&buf, ", pubtruncate");

if (has_pubgencols)
 appendPQExpBufferStr(&buf, ", pubgencols");

if (has_pubviaroot)
 appendPQExpBufferStr(&buf, ", pubviaroot");

~~

IIUC the variable number of result columns (for different server
versions) is what is causing all the subsequent hassles.

So, wouldn't the easiest fix be to change the code by adding the
appropriate 'else' alias for when the column is not available?

Like this:

printfPQExpBuffer(&buf,
 "SELECT oid, pubname,\n"
 "  pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
 "  puballtables, pubinsert, pubupdate, pubdelete");

if (has_pubtruncate)
 appendPQExpBufferStr(&buf, ", pubtruncate");
else
 appendPQExpBufferStr(&buf, ", 'f' AS pubtruncate");

if (has_pubgencols)
 appendPQExpBufferStr(&buf, ", pubgencols");
else
 appendPQExpBufferStr(&buf, ", 'f' AS pubgencols");

if (has_pubviaroot)
 appendPQExpBufferStr(&buf, ", pubviaroot");
else
 appendPQExpBufferStr(&buf, ", 'f' AS pubviaroot");

~~

Unless I am mistaken this will simplify the subsequent code a lot because:
1. Now you can put the cols in the same order you want to display them
2. Now the tuple result has a fixed number of cols for all server versions
3. Now hardcoding the indexes (1,2,3,4...) is fine because they are
always the same

Thoughts?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  297. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-19T12:07:27Z

    On Sun, 19 Jan 2025 at 06:39, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> I was having some second thoughts about this patch and my previous suggestion.
>
> Currently the code is current written something like:
>
> printfPQExpBuffer(&buf,
>  "SELECT oid, pubname,\n"
>  "  pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
>  "  puballtables, pubinsert, pubupdate, pubdelete");
>
> if (has_pubtruncate)
>  appendPQExpBufferStr(&buf, ", pubtruncate");
>
> if (has_pubgencols)
>  appendPQExpBufferStr(&buf, ", pubgencols");
>
> if (has_pubviaroot)
>  appendPQExpBufferStr(&buf, ", pubviaroot");
>
> ~~
>
> IIUC the variable number of result columns (for different server
> versions) is what is causing all the subsequent hassles.
>
> So, wouldn't the easiest fix be to change the code by adding the
> appropriate 'else' alias for when the column is not available?
>
> Like this:
>
> printfPQExpBuffer(&buf,
>  "SELECT oid, pubname,\n"
>  "  pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
>  "  puballtables, pubinsert, pubupdate, pubdelete");
>
> if (has_pubtruncate)
>  appendPQExpBufferStr(&buf, ", pubtruncate");
> else
>  appendPQExpBufferStr(&buf, ", 'f' AS pubtruncate");
>
> if (has_pubgencols)
>  appendPQExpBufferStr(&buf, ", pubgencols");
> else
>  appendPQExpBufferStr(&buf, ", 'f' AS pubgencols");
>
> if (has_pubviaroot)
>  appendPQExpBufferStr(&buf, ", pubviaroot");
> else
>  appendPQExpBufferStr(&buf, ", 'f' AS pubviaroot");
>
> ~~
>
> Unless I am mistaken this will simplify the subsequent code a lot because:
> 1. Now you can put the cols in the same order you want to display them
> 2. Now the tuple result has a fixed number of cols for all server versions
> 3. Now hardcoding the indexes (1,2,3,4...) is fine because they are
> always the same
>
> Thoughts?

We typically use this approach when performing a dump, where we
retrieve the default values for older versions and store them in a
structure. The values are then included in the dump only if they
differ from the default. However, this approach cannot be used with
psql because older version servers may not have these columns. As a
result, we must avoid displaying these columns when interacting with
older version servers.

Regards,
Vignesh

  298. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-19T12:16:49Z

    On Fri, 17 Jan 2025 at 11:23, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh.
>
> Some review comments for patch v52-0003
>
> ======
>
> 1. GENERAL - change to use enum.
>
> On Thu, Jan 16, 2025 at 7:47 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 15 Jan 2025 at 11:17, Peter Smith <smithpb2250@gmail.com> wrote:
> > > 2.
> > > As suggested in more detail below, I think it would be better if you
> > > can define a C code enum type for these potential values instead of
> > > just using #define macros and char. I guess that will impact a lot of
> > > the APIs.
> >
> > If we change it to enum, we will not be able to access
> > PUBLISH_GENCOLS_NONE and PUBLISH_GENCOLS_STORED from describe.c files.
> > Maybe that is the reason the macros were used in the case of
> > pg_subscription.h also.
>
> Hm. I am not sure. Can't you just define the enum inside the #ifdef
> EXPOSE_TO_CLIENT_CODE? I saw some examples of this already (see
> src/include/catalog/pg_cast.h)
>
> e.g. I tried following, which compiles for me:
>
> #ifdef EXPOSE_TO_CLIENT_CODE
>
> typedef enum PublishGencolsType
> {
>   /* Generated columns present should not be replicated. */
>   PUBLISH_GENCOLS_NONE = 'n',
>
>   /* Generated columns present should be replicated. */
>   PUBLISH_GENCOLS_STORED = 's',
>
> } PublishGencolsType;
>
> #endif  /* EXPOSE_TO_CLIENT_CODE */
>
> typedef struct Publication
> {
>   Oid    oid;
>   char   *name;
>   bool   alltables;
>   bool   pubviaroot;
>   PublishGencolsType pubgencols_type;
>   PublicationActions pubactions;
> } Publication;

Yes, the compilation seems fine but listPublications which uses
CppAsString2(PUBLISH_GENCOLS_NONE) does not get 'n' but gets it as
publish_gencols_none like below:
postgres=# \dRp
ERROR:  column "publish_gencols_none" does not exist
LINE 9:     WHEN PUBLISH_GENCOLS_NONE THEN 'non

The rest of the comments are handled and also the comments from [1]
are fixed in the v53 version patch attached.

[1] - https://www.postgresql.org/message-id/CAHut%2BPuykmXF57T9AcBnsVqQEddmpBeHZvnGdJt%3DbyYozXAcSg%40mail.gmail.com

Regards,
Vignesh

  299. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-20T00:44:09Z

    Hi Vignesh,

Review comments for patch v53-0001:

I don't doubt that the latest 0001 patch is working OK, but IMO it
didn't need to be this complicated. I still think just using an alias
for all the columns not present as I suggested yesterday [1] would be
far simpler than the current patch.


On Sun, Jan 19, 2025 at 11:07 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Sun, 19 Jan 2025 at 06:39, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh,
> >
> > I was having some second thoughts about this patch and my previous suggestion.
> >
> > Currently the code is current written something like:
> >
> > printfPQExpBuffer(&buf,
> >  "SELECT oid, pubname,\n"
> >  "  pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
> >  "  puballtables, pubinsert, pubupdate, pubdelete");
> >
> > if (has_pubtruncate)
> >  appendPQExpBufferStr(&buf, ", pubtruncate");
> >
> > if (has_pubgencols)
> >  appendPQExpBufferStr(&buf, ", pubgencols");
> >
> > if (has_pubviaroot)
> >  appendPQExpBufferStr(&buf, ", pubviaroot");
> >
> > ~~
> >
> > IIUC the variable number of result columns (for different server
> > versions) is what is causing all the subsequent hassles.
> >
> > So, wouldn't the easiest fix be to change the code by adding the
> > appropriate 'else' alias for when the column is not available?
> >
> > Like this:
> >
> > printfPQExpBuffer(&buf,
> >  "SELECT oid, pubname,\n"
> >  "  pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
> >  "  puballtables, pubinsert, pubupdate, pubdelete");
> >
> > if (has_pubtruncate)
> >  appendPQExpBufferStr(&buf, ", pubtruncate");
> > else
> >  appendPQExpBufferStr(&buf, ", 'f' AS pubtruncate");
> >
> > if (has_pubgencols)
> >  appendPQExpBufferStr(&buf, ", pubgencols");
> > else
> >  appendPQExpBufferStr(&buf, ", 'f' AS pubgencols");
> >
> > if (has_pubviaroot)
> >  appendPQExpBufferStr(&buf, ", pubviaroot");
> > else
> >  appendPQExpBufferStr(&buf, ", 'f' AS pubviaroot");
> >
> > ~~
> >
> > Unless I am mistaken this will simplify the subsequent code a lot because:
> > 1. Now you can put the cols in the same order you want to display them
> > 2. Now the tuple result has a fixed number of cols for all server versions
> > 3. Now hardcoding the indexes (1,2,3,4...) is fine because they are
> > always the same
> >
> > Thoughts?
>
> We typically use this approach when performing a dump, where we
> retrieve the default values for older versions and store them in a
> structure. The values are then included in the dump only if they
> differ from the default. However, this approach cannot be used with
> psql because older version servers may not have these columns. As a
> result, we must avoid displaying these columns when interacting with
> older version servers.
>

Maybe I have some fundamental misunderstanding here, but I don't see
why "this approach cannot be used with psql because older version
servers may not have these columns". Not having the columns is the
whole point of using an alias approach in the first place e.g. the
below table t1 does not have a column called "banana" but it works
just fine to select an alias using that name...

test_pub=# CREATE TABLE t1 (a int, b int);
CREATE TABLE

test_pub=# INSERT INTO t1 VALUES (123,456);
INSERT 0 1

test_pub=# SELECT a, b, 999 AS banana FROM t1;
  a  |  b  | banana
-----+-----+--------
 123 | 456 |    999
(1 row)


And you also said: "we must avoid displaying these columns when
interacting with older version servers".

But, why must we? I don't know why it is a bad thing to display
"Generated columns" as 'f' for a PG17 server. Anyway, you are always
at liberty to not display the missing columns if that is what you
want... just call the 'printTableAddCell' conditionally the same as
now. Except now, the PQgetvalue can have just a simple fixed index
because the SELECT (with aliases) will also have a fixed number of
columns in the result.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPs3ORjQTcce0MyHyT3aEqCh_jyUMNHUjm0XcrxKNEM8-Q%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

  300. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-20T03:29:06Z

    IIUC, patch v53-0004 is primarily a bug fix for a docs omission of the
master implementation.

So,

1. IMO think this patch in its current form must come *before* the
0003 patch where you changed the PUBLICATION option from bool to enum.

2. Then the patch (currently called) 0003 needs to update this doc
fragment to change the type from bool to char; it should also itemise
the possible values 'n', 's' saying what those values mean.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  301. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-20T04:10:41Z

    Hi Vignesh.

Here are my review comments for patch v53-0003.

On Sun, Jan 19, 2025 at 11:17 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 17 Jan 2025 at 11:23, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh.
> >
> > Some review comments for patch v52-0003
> >
> > ======
> >
> > 1. GENERAL - change to use enum.
> >
> > On Thu, Jan 16, 2025 at 7:47 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, 15 Jan 2025 at 11:17, Peter Smith <smithpb2250@gmail.com> wrote:
> > > > 2.
> > > > As suggested in more detail below, I think it would be better if you
> > > > can define a C code enum type for these potential values instead of
> > > > just using #define macros and char. I guess that will impact a lot of
> > > > the APIs.
> > >
> > > If we change it to enum, we will not be able to access
> > > PUBLISH_GENCOLS_NONE and PUBLISH_GENCOLS_STORED from describe.c files.
> > > Maybe that is the reason the macros were used in the case of
> > > pg_subscription.h also.
> >
> > Hm. I am not sure. Can't you just define the enum inside the #ifdef
> > EXPOSE_TO_CLIENT_CODE? I saw some examples of this already (see
> > src/include/catalog/pg_cast.h)
> >
> > e.g. I tried following, which compiles for me:
> >
> > #ifdef EXPOSE_TO_CLIENT_CODE
> >
> > typedef enum PublishGencolsType
> > {
> >   /* Generated columns present should not be replicated. */
> >   PUBLISH_GENCOLS_NONE = 'n',
> >
> >   /* Generated columns present should be replicated. */
> >   PUBLISH_GENCOLS_STORED = 's',
> >
> > } PublishGencolsType;
> >
> > #endif  /* EXPOSE_TO_CLIENT_CODE */
> >
> > typedef struct Publication
> > {
> >   Oid    oid;
> >   char   *name;
> >   bool   alltables;
> >   bool   pubviaroot;
> >   PublishGencolsType pubgencols_type;
> >   PublicationActions pubactions;
> > } Publication;
>
> Yes, the compilation seems fine but listPublications which uses
> CppAsString2(PUBLISH_GENCOLS_NONE) does not get 'n' but gets it as
> publish_gencols_none like below:
> postgres=# \dRp
> ERROR:  column "publish_gencols_none" does not exist
> LINE 9:     WHEN PUBLISH_GENCOLS_NONE THEN 'non
>

1.
Yeah, but that is hardly any obstacle-- we can simply rewrite that
code fragment like shown below:

appendPQExpBuffer(&buf,
  ",\n (CASE pubgencols_type\n"
  "    WHEN '%c' THEN 'none'\n"
  "    WHEN '%c' THEN 'stored'\n"
  "   END) AS \"%s\"",
  PUBLISH_GENCOLS_NONE,
  PUBLISH_GENCOLS_STORED,
  gettext_noop("Generated columns"));

...

I have made an experimental change (see PS_0003_DIFF.txt) which does
exactly this. All your tests pass just fine.

But, now I think you should be able to modify/improve lots of the APIs
to make good use of the enum PublishGencolsType instead of just
passing char.

======
doc/src/sgml/ref/create_publication.sgml

2.
          <para>
           Specifies whether the generated columns present in the tables
-          associated with the publication should be replicated.
-          The default is <literal>false</literal>.
+          associated with the publication should be replicated. Possible values
+          are <literal>none</literal> and <literal>stored</literal>.
+          The default is <literal>none</literal> meaning the generated
+          columns present in the tables associated with publication will not be
+          replicated.
+         </para>

nit - I think it looks better with a blank line above "The default
is..." giving each value its own paragraph, but it is OK as-is if you
prefer.

======
src/backend/commands/publicationcmds.c

3.
  * 2. Ensures that all the generated columns referenced in the REPLICA IDENTITY
- *    are published either by listing them in the column list or by enabling
- *    publish_generated_columns option. If any unpublished generated column is
- *    found, *invalid_gen_col is set to true.
+ *    are published either by listing them in the column list or if
+ *    publish_generated_columns option is 's'(stored). If any unpublished
+ *    generated column is found, *invalid_gen_col is set to true.

I know the "by listing them" part was existing wording but the result
now is tricky to understand.

CURRENT
Ensures that all the generated columns referenced in the REPLICA
IDENTITY are published either by listing them in the column list or if
publish_generated_columns option is 's'(stored).

SUGGESTION
Ensures that all the generated columns referenced in the REPLICA
IDENTITY are published, either by being explicitly named in the column
list or, if no column list is specified, by setting the option
publish_generated_columns = stored.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  302. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-20T05:32:07Z

    Hi Vignesh,

Thanks for including my patch in your patch set, ensuring that it does
not get left behind.

1.
These docs are mostly still OK, but have become slightly stale because:

a) they should mention "stored" in some places

b) the examples now need to be using the new enum form of the
publication option introduced in your patch 0003


~~~

2.
Fixed a typo /explicty/explicitly/

======

I've attached a top-up patch to "fix" all issues I found for the
v53-0005 docs patch. Please have a look and apply them if they seem OK
to you.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  303. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-21T08:27:49Z

    On Mon, 20 Jan 2025 at 06:14, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Review comments for patch v53-0001:
>
>
> Maybe I have some fundamental misunderstanding here, but I don't see
> why "this approach cannot be used with psql because older version
> servers may not have these columns". Not having the columns is the
> whole point of using an alias approach in the first place e.g. the
> below table t1 does not have a column called "banana" but it works
> just fine to select an alias using that name...

This is simpler than maintaining the indexes. I misunderstood your
comment to include displaying the columns for older versions, I did
not want to display a value for the older versions as these columns do
not exist. I have updated the patch based on the alias approach. The
attached patch has the changes for the same.

Regards,
Vignesh

  304. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-21T08:34:06Z

    On Mon, 20 Jan 2025 at 08:59, Peter Smith <smithpb2250@gmail.com> wrote:
>
> IIUC, patch v53-0004 is primarily a bug fix for a docs omission of the
> master implementation.
>
> So,
>
> 1. IMO think this patch in its current form must come *before* the
> 0003 patch where you changed the PUBLICATION option from bool to enum.
>
> 2. Then the patch (currently called) 0003 needs to update this doc
> fragment to change the type from bool to char; it should also itemise
> the possible values 'n', 's' saying what those values mean.

These changes are done in the v54 version patch attached at [1].
Another thought was to remove this patch as it will get handled when
the "Change publish_generated_columns option to use enumChange
publish_generated_columns option to use enum" patch gets committed.
Also the comments form [2] and [3] are handled at [1].

[1] - https://www.postgresql.org/message-id/CALDaNm3zxQfJwYw7PwxtvYFAeCk6WkRt2iWu8HPWih8BubwU9g%40mail.gmail.com
[2] - https://www.postgresql.org/message-id/CAHut%2BPvuNx57RB%3DfUZv95q1Eb_01Lzv-%3DEnWDcDE%2BqFh7_yVag%40mail.gmail.com
[3] - https://www.postgresql.org/message-id/CAHut%2BPv9X9LjTJt0wU%2BySbZU-sXCO_bFmEGxinROkA48f8Ws%2Bw%40mail.gmail.com

Regards,
Vignesh

  305. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-22T06:20:41Z

    On Tue, Jan 21, 2025 at 7:28 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 20 Jan 2025 at 06:14, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh,
> >
> > Review comments for patch v53-0001:
> >
> >
> > Maybe I have some fundamental misunderstanding here, but I don't see
> > why "this approach cannot be used with psql because older version
> > servers may not have these columns". Not having the columns is the
> > whole point of using an alias approach in the first place e.g. the
> > below table t1 does not have a column called "banana" but it works
> > just fine to select an alias using that name...
>
> This is simpler than maintaining the indexes. I misunderstood your
> comment to include displaying the columns for older versions, I did
> not want to display a value for the older versions as these columns do
> not exist. I have updated the patch based on the alias approach. The
> attached patch has the changes for the same.
>

Patch v54-0001 LGTM.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  306. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-22T06:47:06Z

    Patch v54-0002 LGTM, although it is missing an explanatory commit message.

e.g. Should be something like:

------
Adds the documentation for the 'pubgencols' column of catalog
pg_publication. This was accidentally missing from commit 7054186.
------

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  307. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-22T10:51:49Z

    On Tue, Jan 21, 2025 at 1:58 PM vignesh C <vignesh21@gmail.com> wrote:
>

I have pushed the 0001 patch. I don't think 0002 and 0003 need to be
committed separately. So, combine them in 0004 and post a new version.

-- 
With Regards,
Amit Kapila.

  308. Re: Pgoutput not capturing the generated columns

    Peter Eisentraut <peter@eisentraut.org> — 2025-01-22T13:52:19Z

    On 21.01.25 09:27, vignesh C wrote:
>> Maybe I have some fundamental misunderstanding here, but I don't see
>> why "this approach cannot be used with psql because older version
>> servers may not have these columns". Not having the columns is the
>> whole point of using an alias approach in the first place e.g. the
>> below table t1 does not have a column called "banana" but it works
>> just fine to select an alias using that name...
> 
> This is simpler than maintaining the indexes. I misunderstood your
> comment to include displaying the columns for older versions, I did
> not want to display a value for the older versions as these columns do
> not exist. I have updated the patch based on the alias approach. The
> attached patch has the changes for the same.

The v54-0004 patch looks ok to me.  Thanks for working on this.

I will wait until that gets committed before moving forward with the 
virtual generated columns patch.


A few quick comments on the v54-0005 patch (the documentation one):

+<programlisting>
+test_pub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS 
AS (a + 1) STORED);

I don't like when examples include the prompt like this.  Then it's 
harder to copy the commands out of the examples.

+     <row>
+ 
<entry>No</entry><entry>GENERATED</entry><entry>GENERATED</entry><entry>Publisher 
table column is not replicated. Use the subscriber table generated 
column value.</entry>
+     </row>

This should be formatted vertically, one line per <entry>.

+<programlisting>
+test_pub=# CREATE TABLE t1 (a int PRIMARY KEY, b int,
+test_pub(#                  c int GENERATED ALWAYS AS (a + 1) STORED,
+test_pub(#                  d int GENERATED ALWAYS AS (b + 1) STORED);

Also here check that this is formatted more consistently with the rest 
of the documentation.

+  <itemizedlist>
+   <listitem><para>
+    <literal>t1.a</literal> is a regular column. It gets replicated 
normally.
+   </para></listitem>
+   <listitem><para>
+    <literal>t1.b</literal> is a regular column. It gets replicated 
normally.
+   </para></listitem>

Leave more whitespace.  Typically, there is a blank line between block 
elements on the same level.

  309. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-22T14:00:07Z

    On Wed, 22 Jan 2025 at 16:22, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 21, 2025 at 1:58 PM vignesh C <vignesh21@gmail.com> wrote:
> >
>
> I have pushed the 0001 patch. I don't think 0002 and 0003 need to be
> committed separately. So, combine them in 0004 and post a new version.

Thanks for pushing this, here is an updated patch which merges 0002
and 0003 along with 0004 patch.

Regards,
Vignesh

  310. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-23T00:22:07Z

    Some review comments for patch v54-0004.

(since preparing this post, I saw you have already posted v55-0001 but
AFAIK, since 55-0001 is just a merge, the same review comments are
still applicable)

======
doc/src/sgml/catalogs.sgml

1.
       <para>
-       If true, this publication replicates the stored generated columns
-       present in the tables associated with the publication.
+       <literal>n</literal> indicates that the generated columns in the tables
+       associated with the publication should not be replicated.
+       <literal>s</literal> indicates that the stored generated columns in the
+       tables associated with the publication should be replicated.
       </para></entry>

It looks OK, but maybe we should use a wording style similar to that
used already for pg_subscription.substream?

Also, should this mention column lists?

SUGGESTION
Indicates how to handle generated column replication (when there is no
publication column list): <literal>n</literal> = generated columns in
the tables associated with the publication should not be replicated;
<literal>s</literal> = stored generated columns in the tables
associated with the publication should be replicated.

======
src/backend/commands/publicationcmds.c

parse_publication_options:

2.
    bool *publish_generated_columns_given,
-   bool *publish_generated_columns)
+   char *publish_generated_columns)

Why not use the PublishGencolsType enum here?

~~~

CreatePublication:

3.
- bool publish_generated_columns;
+ char publish_generated_columns;
  AclResult aclresult;

Why not use the PublishGencolsType enum here?

~~~

AlterPublicationOptions:

4.
  bool publish_generated_columns_given;
- bool publish_generated_columns;
+ char publish_generated_columns;

Why not use the PublishGencolsType enum here?

~~~

defGetGeneratedColsOption::

5.
+/*
+ * Extract the publish_generated_columns option value from a DefElem. "stored"
+ * and "none" values are accepted.
+ */
+static char
+defGetGeneratedColsOption(DefElem *def)
+{

The return type should be PublishGencolsType.

======
src/include/catalog/pg_publication.h

6.
- /* true if generated columns data should be published */
- bool pubgencols;
+ /*
+ * none if generated column data should not be published. stored if stored
+ * generated column data should be published.
+ */
+ char pubgencols_type;
 } FormData_pg_publication;


Maybe this was accidentally changed by some global replacement you
did. IMO the previous (v53) version comment was better here.

- bool pubgencols;
+ /*
+ * 'n'(none) if generated column data should not be published.
+ * 's'(stored) if stored generated column data should be published.
+ */

======

FYI, please see the attached diffs where I have already made some of
these changes while experimenting with the suggestions.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  311. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-23T00:50:59Z

    On Thu, Jan 23, 2025 at 1:00 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 22 Jan 2025 at 16:22, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jan 21, 2025 at 1:58 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> >
> > I have pushed the 0001 patch. I don't think 0002 and 0003 need to be
> > committed separately. So, combine them in 0004 and post a new version.
>
> Thanks for pushing this, here is an updated patch which merges 0002
> and 0003 along with 0004 patch.
>

FYI, the v55-0001 merge seems broken.

git apply ../patches_misc/v55-0001-Change-publish_generated_columns-option-to-use-e.patch
error: patch failed: doc/src/sgml/catalogs.sgml:6396
error: doc/src/sgml/catalogs.sgml: patch does not apply

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  312. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-23T03:48:41Z

    Hi Vignesh,

Patch v55-0002 (the docs chapter one) is a re-badged v54-0005.

I've addressed the PeterE review comments [1] as detailed below.:

FYI, because v55-0001 was broken I have posted this DOCS patch
separately. Please add it back into the patch set as vXX-0002 when you
next post it.


On Thu, Jan 23, 2025 at 12:52 AM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> A few quick comments on the v54-0005 patch (the documentation one):
>
> +<programlisting>
> +test_pub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS
> AS (a + 1) STORED);
>
> I don't like when examples include the prompt like this.  Then it's
> harder to copy the commands out of the examples.
>

I didn't remove all of these prompts because for logical replication
they add value by differentiating which pub/sub node the command is
run on. Also, these prompts are consistent with all other sections of
Chapter 29 -- e.g. see "Row filters", "Column Lists", and more.

OTOH, I did remove all the "continuation" prompts, so now command
cut/copy is easy.

> +     <row>
> +
> <entry>No</entry><entry>GENERATED</entry><entry>GENERATED</entry><entry>Publisher
> table column is not replicated. Use the subscriber table generated
> column value.</entry>
> +     </row>
>
> This should be formatted vertically, one line per <entry>.
>

Fixed.

> +<programlisting>
> +test_pub=# CREATE TABLE t1 (a int PRIMARY KEY, b int,
> +test_pub(#                  c int GENERATED ALWAYS AS (a + 1) STORED,
> +test_pub(#                  d int GENERATED ALWAYS AS (b + 1) STORED);
>
> Also here check that this is formatted more consistently with the rest
> of the documentation.
>

I modified it to put every column on a separate line (e.g. same
formatting as examples on the CREATE TABLE page). As mentioned above,
the continuation prompts are also removed now.

> +  <itemizedlist>
> +   <listitem><para>
> +    <literal>t1.a</literal> is a regular column. It gets replicated
> normally.
> +   </para></listitem>
> +   <listitem><para>
> +    <literal>t1.b</literal> is a regular column. It gets replicated
> normally.
> +   </para></listitem>
>
> Leave more whitespace.  Typically, there is a blank line between block
> elements on the same level.
>

Fixed.

======
[1] PeterE v54-0005 review -
https://www.postgresql.org/message-id/18f56f71-ea01-41aa-811e-367b692e9ca4%40eisentraut.org

Kind Regards,
Peter Smith.
Fujitsu Australia

  313. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-23T03:53:29Z

    On Wed, Jan 22, 2025 at 7:22 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 21.01.25 09:27, vignesh C wrote:
> >> Maybe I have some fundamental misunderstanding here, but I don't see
> >> why "this approach cannot be used with psql because older version
> >> servers may not have these columns". Not having the columns is the
> >> whole point of using an alias approach in the first place e.g. the
> >> below table t1 does not have a column called "banana" but it works
> >> just fine to select an alias using that name...
> >
> > This is simpler than maintaining the indexes. I misunderstood your
> > comment to include displaying the columns for older versions, I did
> > not want to display a value for the older versions as these columns do
> > not exist. I have updated the patch based on the alias approach. The
> > attached patch has the changes for the same.
>
> The v54-0004 patch looks ok to me.
>

Thanks for the review.

> I will wait until that gets committed before moving forward with the
> virtual generated columns patch.
>

Noted.

>
> A few quick comments on the v54-0005 patch (the documentation one):
>
> +<programlisting>
> +test_pub=# CREATE TABLE tab_gen_to_gen (a int, b int GENERATED ALWAYS
> AS (a + 1) STORED);
>
> I don't like when examples include the prompt like this.  Then it's
> harder to copy the commands out of the examples.
>

One point to consider in this context is that examples need to perform
the same or similar commands on pub/sub, so the above prompt can help
in distinguishing the same though it is not necessary. Another thing
is that we have already used prompt style in other examples on the
logical replication page. For example, see "29.2.2. Examples: Set Up
Logical Replication" [1]. We may want to change those places
(separately) if we don't want to use prompt style here.

[1] - https://www.postgresql.org/docs/devel/logical-replication-subscription.html

-- 
With Regards,
Amit Kapila.

  314. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-23T04:28:09Z

    On Thu, 23 Jan 2025 at 05:52, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Some review comments for patch v54-0004.
>
> (since preparing this post, I saw you have already posted v55-0001 but
> AFAIK, since 55-0001 is just a merge, the same review comments are
> still applicable)
>
> ======
> doc/src/sgml/catalogs.sgml
>
> 1.
>        <para>
> -       If true, this publication replicates the stored generated columns
> -       present in the tables associated with the publication.
> +       <literal>n</literal> indicates that the generated columns in the tables
> +       associated with the publication should not be replicated.
> +       <literal>s</literal> indicates that the stored generated columns in the
> +       tables associated with the publication should be replicated.
>        </para></entry>
>
> It looks OK, but maybe we should use a wording style similar to that
> used already for pg_subscription.substream?
>
> Also, should this mention column lists?
>
> SUGGESTION
> Indicates how to handle generated column replication (when there is no
> publication column list): <literal>n</literal> = generated columns in
> the tables associated with the publication should not be replicated;
> <literal>s</literal> = stored generated columns in the tables
> associated with the publication should be replicated.
>
> ======
> src/backend/commands/publicationcmds.c
>
> parse_publication_options:
>
> 2.
>     bool *publish_generated_columns_given,
> -   bool *publish_generated_columns)
> +   char *publish_generated_columns)
>
> Why not use the PublishGencolsType enum here?
>
> ~~~
>
> CreatePublication:
>
> 3.
> - bool publish_generated_columns;
> + char publish_generated_columns;
>   AclResult aclresult;
>
> Why not use the PublishGencolsType enum here?
>
> ~~~
>
> AlterPublicationOptions:
>
> 4.
>   bool publish_generated_columns_given;
> - bool publish_generated_columns;
> + char publish_generated_columns;
>
> Why not use the PublishGencolsType enum here?
>
> ~~~
>
> defGetGeneratedColsOption::
>
> 5.
> +/*
> + * Extract the publish_generated_columns option value from a DefElem. "stored"
> + * and "none" values are accepted.
> + */
> +static char
> +defGetGeneratedColsOption(DefElem *def)
> +{
>
> The return type should be PublishGencolsType.
>
> ======
> src/include/catalog/pg_publication.h
>
> 6.
> - /* true if generated columns data should be published */
> - bool pubgencols;
> + /*
> + * none if generated column data should not be published. stored if stored
> + * generated column data should be published.
> + */
> + char pubgencols_type;
>  } FormData_pg_publication;
>
>
> Maybe this was accidentally changed by some global replacement you
> did. IMO the previous (v53) version comment was better here.
>
> - bool pubgencols;
> + /*
> + * 'n'(none) if generated column data should not be published.
> + * 's'(stored) if stored generated column data should be published.
> + */

Thanks for the comments, the attached v56 version patch has the fixes
for the above comments. v56-0002 is the same as the patch posted at
[1].
[1] - https://www.postgresql.org/message-id/CAHut%2BPuD7_yh54z-sGyK9xBq9cwUeJUG0BRQaHOhCLdw8msnjQ%40mail.gmail.com

Regards,
Vignesh

  315. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-23T05:47:38Z

    Patch v56-0001 LGTM.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  316. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-23T11:33:39Z

    On Thu, Jan 23, 2025 at 11:18 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> Patch v56-0001 LGTM.
>

I have pushed this patch with minor modifications (especially I didn't
took Peter Smith's last suggestion to convert some functions to return
enum instead of char as the proposed one was consistent with
subscription side code and avoided casting in the patch). On of the BF
failed
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2025-01-23%2010%3A17%3A03

pg_dump: error: query failed: ERROR:  column "publish_gencols_none"
does not exist
LINE 1: ...elete, false AS pubtruncate, false AS pubviaroot, PUBLISH_GE...
                                                             ^
pg_dump: detail: Query was: SELECT p.tableoid, p.oid, p.pubname,
p.pubowner, p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
false AS pubtruncate, false AS pubviaroot, PUBLISH_GENCOLS_NONE AS
pubgencols_type FROM pg_publication p
pg_dumpall: error: pg_dump failed on database "template1", exiting

It is probably the case with dump/restore of previous versions. I'll
look into this.


-- 
With Regards,
Amit Kapila.

  317. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-23T12:19:12Z

    On Thu, 23 Jan 2025 at 17:03, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 23, 2025 at 11:18 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Patch v56-0001 LGTM.
> >
>
> I have pushed this patch with minor modifications (especially I didn't
> took Peter Smith's last suggestion to convert some functions to return
> enum instead of char as the proposed one was consistent with
> subscription side code and avoided casting in the patch). On of the BF
> failed
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2025-01-23%2010%3A17%3A03
>
> pg_dump: error: query failed: ERROR:  column "publish_gencols_none"
> does not exist
> LINE 1: ...elete, false AS pubtruncate, false AS pubviaroot, PUBLISH_GE...
>                                                              ^
> pg_dump: detail: Query was: SELECT p.tableoid, p.oid, p.pubname,
> p.pubowner, p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete,
> false AS pubtruncate, false AS pubviaroot, PUBLISH_GENCOLS_NONE AS
> pubgencols_type FROM pg_publication p
> pg_dumpall: error: pg_dump failed on database "template1", exiting
>
> It is probably the case with dump/restore of previous versions. I'll
> look into this.

When dumping from Postgres <=PG17 servers, the query generated for
pubgencols_type incorrectly included the macro name instead of the
macro value. This resulted in dump failures. This commit fixes the
issue by correctly specifying the macro value in the query. The
attached patch has the changes for the same.

Regards,
Vignesh

  318. Re: Pgoutput not capturing the generated columns

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-23T12:44:45Z

    > On 23 Jan 2025, at 13:19, vignesh C <vignesh21@gmail.com> wrote:

> When dumping from Postgres <=PG17 servers, the query generated for
> pubgencols_type incorrectly included the macro name instead of the
> macro value. This resulted in dump failures. This commit fixes the
> issue by correctly specifying the macro value in the query. The
> attached patch has the changes for the same.

I was just looking at the Xversion test failure on crake (which has the log
entry below) when I saw your email.

pg_dump: error: query failed: ERROR: column "publish_gencols_none" does not exist
LINE 1: ...elete, false AS pubtruncate, false AS pubviaroot, PUBLISH_GE...
^
pg_dump: detail: Query was: SELECT p.tableoid, p.oid, p.pubname, p.pubowner, p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS pubtruncate, false AS pubviaroot, PUBLISH_GENCOLS_NONE AS pubgencols_type FROM pg_publication p
pg_dumpall: error: pg_dump failed on database "template1", exiting

Your patch seems like a reasonable fix.

--
Daniel Gustafsson

  319. RE: Pgoutput not capturing the generated columns

    Shinoda, Noriyoshi <noriyoshi.shinoda@hpe.com> — 2025-01-23T13:02:33Z

    Hi, hackers.

Thanks for developing this great feature.
The documentation for the pg_publication catalog shows a 'pubgencols' column, but the actual column name is the 'pubgencols_type' column.
Also, the order of the columns in the documentation differs from the order of the columns in the actual pg_publication catalog.
The attached patch changes the column names and order in the documentation.

Regards,
Noriyoshi Shinoda
-----Original Message-----
From: Daniel Gustafsson <daniel@yesql.se> 
Sent: Thursday, January 23, 2025 9:45 PM
To: vignesh C <vignesh21@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Smith <smithpb2250@gmail.com>; Michael Paquier <michael@paquier.xyz>; Shinoda, Noriyoshi (SXD Japan FSI) <noriyoshi.shinoda@hpe.com>; Shubham Khanna <khannashubham1197@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Rajendra Kumar Dangwal <dangwalrajendra888@gmail.com>; pgsql-hackers@lists.postgresql.org; euler@eulerto.com
Subject: Re: Pgoutput not capturing the generated columns

> On 23 Jan 2025, at 13:19, vignesh C <vignesh21@gmail.com> wrote:

> When dumping from Postgres <=PG17 servers, the query generated for 
> pubgencols_type incorrectly included the macro name instead of the 
> macro value. This resulted in dump failures. This commit fixes the 
> issue by correctly specifying the macro value in the query. The 
> attached patch has the changes for the same.

I was just looking at the Xversion test failure on crake (which has the log entry below) when I saw your email.

pg_dump: error: query failed: ERROR: column "publish_gencols_none" does not exist LINE 1: ...elete, false AS pubtruncate, false AS pubviaroot, PUBLISH_GE...
^
pg_dump: detail: Query was: SELECT p.tableoid, p.oid, p.pubname, p.pubowner, p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS pubtruncate, false AS pubviaroot, PUBLISH_GENCOLS_NONE AS pubgencols_type FROM pg_publication p
pg_dumpall: error: pg_dump failed on database "template1", exiting

Your patch seems like a reasonable fix.

--
Daniel Gustafsson

  320. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-23T23:10:56Z

    On Fri, Jan 24, 2025 at 12:03 AM Shinoda, Noriyoshi (SXD Japan FSI)
<noriyoshi.shinoda@hpe.com> wrote:
>
> Hi, hackers.
>
> Thanks for developing this great feature.
> The documentation for the pg_publication catalog shows a 'pubgencols' column, but the actual column name is the 'pubgencols_type' column.
> Also, the order of the columns in the documentation differs from the order of the columns in the actual pg_publication catalog.
> The attached patch changes the column names and order in the documentation.
>

Your catalog pg_publication docs fix patch does what it says and
looked ok to me.

However, in hindsight, I am not sure that the column should have been
renamed 'pubgencols_type' in the first place because I cannot find any
other catalogs with an underscore in their column names.

Maybe 'pubgencolstype' or simply leaving it as 'pubgencols' would have
been better?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  321. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-24T02:24:09Z

    On Fri, Jan 24, 2025 at 4:41 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> However, in hindsight, I am not sure that the column should have been
> renamed 'pubgencols_type' in the first place because I cannot find any
> other catalogs with an underscore in their column names.
>

See pg_rewrite.ev_type for a similar case.

-- 
With Regards,
Amit Kapila.

  322. Re: Pgoutput not capturing the generated columns

    Tom Lane <tgl@sss.pgh.pa.us> — 2025-01-24T03:09:07Z

    Amit Kapila <amit.kapila16@gmail.com> writes:
> On Fri, Jan 24, 2025 at 4:41 AM Peter Smith <smithpb2250@gmail.com> wrote:
>> However, in hindsight, I am not sure that the column should have been
>> renamed 'pubgencols_type' in the first place because I cannot find any
>> other catalogs with an underscore in their column names.

> See pg_rewrite.ev_type for a similar case.

I think the problem is not so much the underscore as the
inconsistency.  You've got "pub", "gen", and "cols" run together,
but then you feel a need to separate "type"?  It just looks wrong,
the more so because no other column in that catalog has an
underscore in its name.

I see that this was carried over from a related C typedef name,
but users aren't going to see that.  They'll just see that
somebody couldn't be bothered to maintain a consistent style.

			regards, tom lane

  323. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-24T04:15:29Z

    On Fri, Jan 24, 2025 at 8:39 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com> writes:
> > On Fri, Jan 24, 2025 at 4:41 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >> However, in hindsight, I am not sure that the column should have been
> >> renamed 'pubgencols_type' in the first place because I cannot find any
> >> other catalogs with an underscore in their column names.
>
> > See pg_rewrite.ev_type for a similar case.
>
> I think the problem is not so much the underscore as the
> inconsistency.  You've got "pub", "gen", and "cols" run together,
> but then you feel a need to separate "type"?
>

It was easy to read and to avoid getting a single word too long.
However, I do understand your concern. so will change it to
pubgencolstype unless you or someone prefers pubgencols?

-- 
With Regards,
Amit Kapila.

  324. Re: Pgoutput not capturing the generated columns

    Tom Lane <tgl@sss.pgh.pa.us> — 2025-01-24T04:21:46Z

    Amit Kapila <amit.kapila16@gmail.com> writes:
> On Fri, Jan 24, 2025 at 8:39 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I think the problem is not so much the underscore as the
>> inconsistency.  You've got "pub", "gen", and "cols" run together,
>> but then you feel a need to separate "type"?

> It was easy to read and to avoid getting a single word too long.
> However, I do understand your concern. so will change it to
> pubgencolstype unless you or someone prefers pubgencols?

I think I'd vote for "pubgencols".  I don't see what the "_type"
suffix is supposed to convey --- there is nothing very type-y about
this.

			regards, tom lane

  325. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-24T05:06:38Z

    On Fri, 24 Jan 2025 at 09:51, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com> writes:
> > On Fri, Jan 24, 2025 at 8:39 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> I think the problem is not so much the underscore as the
> >> inconsistency.  You've got "pub", "gen", and "cols" run together,
> >> but then you feel a need to separate "type"?
>
> > It was easy to read and to avoid getting a single word too long.
> > However, I do understand your concern. so will change it to
> > pubgencolstype unless you or someone prefers pubgencols?
>
> I think I'd vote for "pubgencols".  I don't see what the "_type"
> suffix is supposed to convey --- there is nothing very type-y about
> this.

I believe simply renaming the catalog column to 'pubgencols' should
suffice. We can keep the internal structure name as 'pubgencols_type'
as it is not exposed, unless you prefer to update it to 'pubgencols'
as well.

Regards,
Vignesh

  326. Re: Pgoutput not capturing the generated columns

    vignesh C <vignesh21@gmail.com> — 2025-01-24T07:38:18Z

    On Fri, 24 Jan 2025 at 10:36, vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 24 Jan 2025 at 09:51, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > Amit Kapila <amit.kapila16@gmail.com> writes:
> > > On Fri, Jan 24, 2025 at 8:39 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >> I think the problem is not so much the underscore as the
> > >> inconsistency.  You've got "pub", "gen", and "cols" run together,
> > >> but then you feel a need to separate "type"?
> >
> > > It was easy to read and to avoid getting a single word too long.
> > > However, I do understand your concern. so will change it to
> > > pubgencolstype unless you or someone prefers pubgencols?
> >
> > I think I'd vote for "pubgencols".  I don't see what the "_type"
> > suffix is supposed to convey --- there is nothing very type-y about
> > this.
>
> I believe simply renaming the catalog column to 'pubgencols' should
> suffice. We can keep the internal structure name as 'pubgencols_type'
> as it is not exposed, unless you prefer to update it to 'pubgencols'
> as well.

The attached patch has the changes for the same.

Regards,
Vignesh

  327. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-27T05:22:53Z

    On Fri, Jan 24, 2025 at 1:08 PM vignesh C <vignesh21@gmail.com> wrote:
>
> The attached patch has the changes for the same.
>

LGTM. Unless there are more comments, I'll push this in a day or so.

-- 
With Regards,
Amit Kapila.

  328. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-28T00:26:09Z

    On Fri, Jan 24, 2025 at 6:38 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 24 Jan 2025 at 10:36, vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 24 Jan 2025 at 09:51, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >
> > > Amit Kapila <amit.kapila16@gmail.com> writes:
> > > > On Fri, Jan 24, 2025 at 8:39 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > >> I think the problem is not so much the underscore as the
> > > >> inconsistency.  You've got "pub", "gen", and "cols" run together,
> > > >> but then you feel a need to separate "type"?
> > >
> > > > It was easy to read and to avoid getting a single word too long.
> > > > However, I do understand your concern. so will change it to
> > > > pubgencolstype unless you or someone prefers pubgencols?
> > >
> > > I think I'd vote for "pubgencols".  I don't see what the "_type"
> > > suffix is supposed to convey --- there is nothing very type-y about
> > > this.
> >
> > I believe simply renaming the catalog column to 'pubgencols' should
> > suffice. We can keep the internal structure name as 'pubgencols_type'
> > as it is not exposed, unless you prefer to update it to 'pubgencols'
> > as well.
>
> The attached patch has the changes for the same.
>

Hi Vignesh

The changes LGTM.

I was surprised that there was no need to modify any expected test
output. I guess that means there are no tests anywhere directly
looking at the pg_publication catalog column names, but instead, all
tests for that catalog must be going via a publication view or using
psql describe output.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  329. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-28T08:58:51Z

    On Thu, Jan 23, 2025 at 9:58 AM vignesh C <vignesh21@gmail.com> wrote:
>

I have pushed the remaining part of this patch. Now, we can review the
proposed documentation part.

I feel we don't need the Examples sub-section for this part of the
docs. The behavior is quite clear from the "Behavior Summary"
sub-section table. Also, I don't find the sub-sections 29.6.1 and
29.6.2 are worth mentioning as separate sub-chapters.

*
+     non-PostgreSQL database via plugin output,

In the above sentence, shouldn't it be 'output plugin' instead of
'plugin output'? We use that way at other places in the docs.

-- 
With Regards,
Amit Kapila.

  330. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-29T00:32:34Z

    On Tue, Jan 28, 2025 at 7:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 23, 2025 at 9:58 AM vignesh C <vignesh21@gmail.com> wrote:
> >
>
> I have pushed the remaining part of this patch. Now, we can review the
> proposed documentation part.
>
> I feel we don't need the Examples sub-section for this part of the
> docs. The behavior is quite clear from the "Behavior Summary"
> sub-section table.

It is good to hear that the "Behavior Summary" matrix is clear, but it
is never the purpose of examples to show behaviour that is not already
clearly documented. The examples are simply to demonstrate some real
usage. Personally, I find it far easier to understand this (or any
other) feature by working through a few examples in conjunction with
the behaviour matrix, instead of just having the matrix and nothing
else.

Before removing the examples section I'd like to know if other people
also think it has no value.

> Also, I don't find the sub-sections 29.6.1 and
> 29.6.2 are worth mentioning as separate sub-chapters.
>

OK. Removed these as requested.

> *
> +     non-PostgreSQL database via plugin output,
>
> In the above sentence, shouldn't it be 'output plugin' instead of
> 'plugin output'? We use that way at other places in the docs.
>

Fixed.

~~~

I also modified some whitespace indentations in the SGML file.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  331. Re: Pgoutput not capturing the generated columns

    Amit Kapila <amit.kapila16@gmail.com> — 2025-01-29T03:48:27Z

    On Wed, Jan 29, 2025 at 6:03 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jan 28, 2025 at 7:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jan 23, 2025 at 9:58 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> >
> > I have pushed the remaining part of this patch. Now, we can review the
> > proposed documentation part.
> >
> > I feel we don't need the Examples sub-section for this part of the
> > docs. The behavior is quite clear from the "Behavior Summary"
> > sub-section table.
>
> It is good to hear that the "Behavior Summary" matrix is clear, but it
> is never the purpose of examples to show behaviour that is not already
> clearly documented. The examples are simply to demonstrate some real
> usage. Personally, I find it far easier to understand this (or any
> other) feature by working through a few examples in conjunction with
> the behaviour matrix, instead of just having the matrix and nothing
> else.
>

I am not against giving examples in the docs to make the topic easy to
understand but in this particular case, I am not sure if additional
examples are useful. You already gave one example in the beginning:
"For example, note below that subscriber table generated column value
comes from the subscriber column's calculation." the remaining text is
clear enough to understand the feature.

If you still want to make a case for additional examples, divide this
patch into two parts. The first part without examples could be part of
this thread and I can commit that. Then you can start a separate
thread just for the examples and then we can see what others think and
make a decision based on that.

-- 
With Regards,
Amit Kapila.

  332. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-01-29T23:15:47Z

    On Wed, Jan 29, 2025 at 2:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 29, 2025 at 6:03 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2025 at 7:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Jan 23, 2025 at 9:58 AM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > >
> > > I have pushed the remaining part of this patch. Now, we can review the
> > > proposed documentation part.
> > >
> > > I feel we don't need the Examples sub-section for this part of the
> > > docs. The behavior is quite clear from the "Behavior Summary"
> > > sub-section table.
> >
> > It is good to hear that the "Behavior Summary" matrix is clear, but it
> > is never the purpose of examples to show behaviour that is not already
> > clearly documented. The examples are simply to demonstrate some real
> > usage. Personally, I find it far easier to understand this (or any
> > other) feature by working through a few examples in conjunction with
> > the behaviour matrix, instead of just having the matrix and nothing
> > else.
> >
>
> I am not against giving examples in the docs to make the topic easy to
> understand but in this particular case, I am not sure if additional
> examples are useful. You already gave one example in the beginning:
> "For example, note below that subscriber table generated column value
> comes from the subscriber column's calculation." the remaining text is
> clear enough to understand the feature.
>
> If you still want to make a case for additional examples, divide this
> patch into two parts. The first part without examples could be part of
> this thread and I can commit that. Then you can start a separate
> thread just for the examples and then we can see what others think and
> make a decision based on that.
>

The v57-0001 DOCS patch has been split into 2 as requested.

v58-0001 Same as v57-0001 but with the "Examples" subsection removed
v58-0002 Adds the "Examples" subsection back on top of v58-0001

If you want any ongoing discussion about Examples to be moved into a
separate thread, I can start that thread after the 0001 patch is
committed.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  333. Re: Pgoutput not capturing the generated columns

    Peter Smith <smithpb2250@gmail.com> — 2025-02-02T22:58:04Z

    On Wed, Jan 29, 2025 at 2:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 29, 2025 at 6:03 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2025 at 7:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Jan 23, 2025 at 9:58 AM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > >
> > > I have pushed the remaining part of this patch. Now, we can review the
> > > proposed documentation part.
> > >
> > > I feel we don't need the Examples sub-section for this part of the
> > > docs. The behavior is quite clear from the "Behavior Summary"
> > > sub-section table.
> >
> > It is good to hear that the "Behavior Summary" matrix is clear, but it
> > is never the purpose of examples to show behaviour that is not already
> > clearly documented. The examples are simply to demonstrate some real
> > usage. Personally, I find it far easier to understand this (or any
> > other) feature by working through a few examples in conjunction with
> > the behaviour matrix, instead of just having the matrix and nothing
> > else.
> >
>
> I am not against giving examples in the docs to make the topic easy to
> understand but in this particular case, I am not sure if additional
> examples are useful. You already gave one example in the beginning:
> "For example, note below that subscriber table generated column value
> comes from the subscriber column's calculation." the remaining text is
> clear enough to understand the feature.
>
> If you still want to make a case for additional examples, divide this
> patch into two parts. The first part without examples could be part of
> this thread and I can commit that. Then you can start a separate
> thread just for the examples and then we can see what others think and
> make a decision based on that.
>

I have created a new thread [1] to propose adding the "Examples" subsection.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPt_7GV8eHSW4XQsC6rF13TWrz-SrGeeiV71%3DSE14DC4Jg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia